Rasbperry pi 4 performance issues

[StarfoxHUN]

Hi all! I have my Jellyfin server installed on a Raspberry pi 4. It works great, but it can buffer a lot when certain videos start or even during playback. I assume its because of transcoding, as playing the exact same video trough UPNP or SMB they work fine. So i know that currently hardware transcoding is not meant to work on Raspberry devices, so i wonder if there is anything i can do to make things work smoother. I tired to set every setting on that meant to help avoid transcoding, but it seems i still cannot avoid it in some cases. Is there anything i can do to make thinks work smoother without causing instability? (The server is used by family members too, so i cannot resort to unstable solutions sadly.) Or i have to accept it as a limit of the raspberry devices and have to endure it till i get something beefier?
Thanks for the help!