SOLVED: Poor transcoding performance

[ottomatik]

Hi everyone,

I'm currently running jellyfin in Docker using an OpenMediaVault OS on a server carrying an Intel i3-12100 (UHD 730 iGPU) + 16GB of RAM. I'm using linuxserver image.

When I try to watch a 4K HEVC HDR DV file on a device that needs transcoding (mainly iPhone 13 or laptop web browser), I get poor transcoding performance: the file can take up to a full minute to load (especially on phone) and I get stutters every 5-10 seconds once it plays.
I can't navigate inside the file either, each time I try to reach a different timing it takes forever to load and sometimes won't at all.
It's slightly better when using other iOS apps like Streamyfin. Also better if I restart the jellyfin container but then worsens over time.
Once I stop the file, the CPU usage stays very high for a few minutes before going back to normal, but during that time the whole server gets really slow.

My jellyfin is behind nginx-proxy-manager but even when I reach it in LAN I have poor performance.
HWA seems to be working properly: I'm getting almost 100% usage on my iGPU when monitoring intel_gpu_top. I went through the tutorial on enabling Intel LP modes and they're working.
Files are on a HDD but the transcoding folder is on a SSD.

On transcoding settings here's what I set :
- decoding all except VP8 (according to Intel's website)
- OS decoders (VA-API/DXVA) enabled, tried disabling with no significant difference
- HWA and LP H264 HWA enabled, disabling H265 encoding made a slight increase in performance
- tonal mapping enabled (tried disabling VPP but no difference)
- "auto" encoding profile (tried ultrafast but no difference)
- deleting segments and lowering transcoding speed both enabled.

Here is a ffmepg log when trying to watch on iPhone with poor performance (long loading time + some stutters): https://pastebin.com/cLwvEerE. What about those NAL units?
Here's my docker-compose: https://pastebin.com/gBEmya2j
Picture of intel_gpu_top when running a transcode: https://ibb.co/Q7jmGFJC

So is it just my iGPU that doesn't have enough horsepower to handle heavy transcodes? Or is there something else I should try in the settings?

[ArneR]

I'd say the performance is in line with what I'd expect out of a 12th gen iGPU.

frame= 4586 fps= 80 q=24.0 size=N/A time=00:03:10.52 bitrate=N/A speed=3.34x

For comparison, this is Dune (2021) transcoding on an intel i5-11600k (UHD 750) and a Xeon 6244 with an intel ARC A310:

         

But burning in subtitles also have a cost to performance, so I'd suggest you try enabling the experimental PGS rendering under client settings.

   

And I couldn't see that you had altered the bitrate, so I'd assume it's on auto, you could try lowering it since you're watching on a phone, there's really no benefit to having the stream in 4K at ~50000kbps

Audio is being transcoded as well in your case, and that is done on the CPU. There's no clear hint to it being a problem, but the 12100 is only a four core if I remember correctly, and depending on how many threads are being passed through could be worth looking into. I'm running my jellyfin on my truenas scale server, where it's easy to set resource limits in regards to threads and RAM. I also transcode to RAM since I have 256GB of it, less wear on the SSDs.

[ottomatik]

Thanks for your answer. Looks like both rendering PGS subtitles and lowering bitrates improved transcoding. I also stuck with the fastest encoding profile and I get reasonable performance now.