Slow (?) HW-Transcoding with I5-11400 on Proxmox + Jellyfin via Docker

[bentrades]

Hi all,
I'm facing an issue with Jellyfin on my spouse's Android phone (S23) running Jellyfin app 2.6.2. Any content she watches gets transcoded, unlike when viewing via:

• Chromecast with Google TV (official Jellyfin App)
  
• Apple TV 4K (Infuse)
  
• iPhone (Streamyfin/Infuse)
  

I have 2160P HEVC content that gets transcoded to MP4 H264 AAC at 26.5 Mbps due to unsupported container and audio codec. My setup includes an i5-11400/UHD 730, showing 35 FPS during transcoding. I'm unsure if this is expected or a misconfiguration.
Here's my current setup:

• GPU Passthrough via Proxmox with /dev/dri in docker-compose
  
• intel_gpu_top in VM shows 70-80% utilization
  
• CPU (12 threads assigned via CPU Type: host to the VM) shows roughly 50% utilization
  
• Hardware Transcoding enabled for Intel QuickSync Video
  

I'm considering upgrading to an ARC 380 for more headroom. Below is the FFMPEG log of the transcode. Is the speed to be expected or did I just mess it up when setting it up.
Thanks for any insights!

ffmpeg version 7.0.2-Jellyfin Copyright © 2000-2024 the FFmpeg developers
  built with gcc 13 (Ubuntu 13.3.0-6ubuntu2~24.04)
  configuration: --prefix=/usr/lib/jellyfin-ffmpeg --target-os=linux --extra-version=Jellyfin --disable-doc --disable-ffplay --disable-ptx-compression --disable-static --disable-libxcb --disable-sdl2 --disable-xlib --enable-lto=auto --enable-gpl --enable-version3 --enable-shared --enable-gmp --enable-gnutls --enable-chromaprint --enable-opencl --enable-libdrm --enable-libxml2 --enable-libass --enable-libfreetype --enable-libfribidi --enable-libfontconfig --enable-libharfbuzz --enable-libbluray --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libopenmpt --enable-libdav1d --enable-libsvtav1 --enable-libwebp --enable-libvpx --enable-libx264 --enable-libx265 --enable-libzvbi --enable-libzimg --enable-libfdk-aac --arch=amd64 --enable-libshaderc --enable-libplacebo --enable-vulkan --enable-vaapi --enable-amf --enable-libvpl --enable-ffnvcodec --enable-cuda --enable-cuda-llvm --enable-cuvid --enable-nvdec --enable-nvenc
  libavutil      59.  8.100 / 59.  8.100
  libavcodec     61.  3.100 / 61.  3.100
  libavformat    61.  1.100 / 61.  1.100
  libavdevice    61.  1.100 / 61.  1.100
  libavfilter    10.  1.100 / 10.  1.100
  libswscale      8.  1.100 /  8.  1.100
  libswresample   5.  1.100 /  5.  1.100
  libpostproc    58.  1.100 / 58.  1.100
libva info: VA-API version 1.22.0
libva info: Trying to open /usr/lib/jellyfin-ffmpeg/lib/dri/iHD_drv_video.so
libva info: Found init function __vaDriverInit_1_22
libva info: va_openDriver() returns 0
libva info: VA-API version 1.22.0
libva info: Trying to open /usr/lib/jellyfin-ffmpeg/lib/dri/iHD_drv_video.so
libva info: Found init function __vaDriverInit_1_22
libva info: va_openDriver() returns 0
Input #0, matroska,webm, from 'filedata/Shows/Something/Season 3/Something- S03E03 - Title 2160p.mkv':
  Metadata:
    encoder         : libebml v1.4.5 + libmatroska v1.7.1
    creation_time   : 2024-08-11T22:37:47.000000Z
  Duration: 00:29:30.19, start: 0.000000, bitrate: 15276 kb/s
  Stream #0:0(eng): Video: hevc (Main 10), yuv420p10le(tv, bt2020nc/bt2020/smpte2084), 3840x2160 [SAR 1:1 DAR 16:9], 23.98 fps, 23.98 tbr, 1k tbn (default)
      Metadata:
        title           : Web
        BPS             : 14761947
        DURATION        : 00:29:30.185000000
        NUMBER_OF_FRAMES: 42442
        NUMBER_OF_BYTES : 3266422336
        _STATISTICS_WRITING_APP: mkvmerge v84.0 ('Sleeper') 64-bit
        _STATISTICS_WRITING_DATE_UTC: 2024-08-11 22:37:47
        _STATISTICS_TAGS: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
      Side data:
        DOVI configuration record: version: 1.0, profile: 8, level: 6, rpu flag: 1, el flag: 0, bl flag: 1, compatibility id: 1
  Stream #0:1(ger): Audio: eac3, 48000 Hz, 5.1(side), fltp, 256 kb/s (default)
      Metadata:
        title           : Web
        BPS             : 256000
        DURATION        : 00:29:30.144000000
        NUMBER_OF_FRAMES: 55317
        NUMBER_OF_BYTES : 56644608
        _STATISTICS_WRITING_APP: mkvmerge v84.0 ('Sleeper') 64-bit
        _STATISTICS_WRITING_DATE_UTC: 2024-08-11 22:37:47
        _STATISTICS_TAGS: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
  Stream #0:2(eng): Audio: eac3, 48000 Hz, 5.1(side), fltp, 256 kb/s (default)
      Metadata:
        title           : Web
        BPS             : 256000
        DURATION        : 00:29:30.144000000
        NUMBER_OF_FRAMES: 55317
        NUMBER_OF_BYTES : 56644608
        _STATISTICS_WRITING_APP: mkvmerge v84.0 ('Sleeper') 64-bit
        _STATISTICS_WRITING_DATE_UTC: 2024-08-11 22:37:47
        _STATISTICS_TAGS: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
  Stream #0:3(ger): Subtitle: subrip (srt) (forced)
      Metadata:
        title           : Forced
        BPS             : 0
        DURATION        : 00:21:03.554000000
        NUMBER_OF_FRAMES: 4
        NUMBER_OF_BYTES : 36
        _STATISTICS_WRITING_APP: mkvmerge v84.0 ('Sleeper') 64-bit
        _STATISTICS_WRITING_DATE_UTC: 2024-08-11 22:37:47
        _STATISTICS_TAGS: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
  Stream #0:4(ger): Subtitle: subrip (srt) (default)
      Metadata:
        title           : Full
        BPS             : 63
        DURATION        : 00:27:11.046000000
        NUMBER_OF_FRAMES: 434
        NUMBER_OF_BYTES : 13036
        _STATISTICS_WRITING_APP: mkvmerge v84.0 ('Sleeper') 64-bit
        _STATISTICS_WRITING_DATE_UTC: 2024-08-11 22:37:47
        _STATISTICS_TAGS: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
  Stream #0:5(eng): Subtitle: subrip (srt) (default)
      Metadata:
        title           : Full
        BPS             : 79
        DURATION        : 00:25:04.962000000
        NUMBER_OF_FRAMES: 442
        NUMBER_OF_BYTES : 14975
        _STATISTICS_WRITING_APP: mkvmerge v84.0 ('Sleeper') 64-bit
        _STATISTICS_WRITING_DATE_UTC: 2024-08-11 22:37:47
        _STATISTICS_TAGS: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
  Stream #0:6(eng): Subtitle: subrip (srt) (hearing impaired)
      Metadata:
        title           : SDH
        BPS             : 79
        DURATION        : 00:27:06.083000000
        NUMBER_OF_FRAMES: 456
        NUMBER_OF_BYTES : 16128
        _STATISTICS_WRITING_APP: mkvmerge v84.0 ('Sleeper') 64-bit
        _STATISTICS_WRITING_DATE_UTC: 2024-08-11 22:37:47
        _STATISTICS_TAGS: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
Stream mapping:
  Stream #0:0 -> #0:0 (hevc (hevc_qsv) -> h264 (libx264))
  Stream #0:1 -> #0:1 (eac3 (native) -> aac (libfdk_aac))
Press [q] to stop, [?] for help
[libx264 @ 0x615c7e54c240] using SAR=1/1
[libx264 @ 0x615c7e54c240] frame MB size (240x135) > level limit (8192)
[libx264 @ 0x615c7e54c240] DPB size (4 frames, 129600 mbs) > level limit (1 frames, 32768 mbs)
[libx264 @ 0x615c7e54c240] MB rate (776823) > level limit (245760)
[libx264 @ 0x615c7e54c240] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2 AVX512
[libx264 @ 0x615c7e54c240] profile High, level 4.1, 4:2:0, 8-bit
[libx264 @ 0x615c7e54c240] 264 - core 164 r3108 31e19f9 - H.264/MPEG-4 AVC codec - Copyleft 2003-2023 - http://www.videolan.org/x264.html - options: cabac=1 ref=1 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=0 psy=1 psy_rd=1.00:0.00 mixed_ref=0 me_range=16 chroma_me=1 trellis=0 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=0 threads=18 lookahead_threads=3 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=1 keyint=250 keyint_min=23 scenecut=0 intra_refresh=0 rc_lookahead=10 rc=crf mbtree=1 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 vbv_maxrate=25461 vbv_bufsize=50922 crf_max=0.0 nal_hrd=none filler=0 ip_ratio=1.40 aq=1:1.00
[hls @ 0x615c7eb7dc00] Opening '/config/cache/transcodes/2b65f8ac529c5a1ab511c2002bbf3fbd-1.mp4' for writing
Output #0, hls, to '/config/cache/transcodes/2b65f8ac529c5a1ab511c2002bbf3fbd.m3u8':
  Metadata:
    encoder         : Lavf61.1.100
  Stream #0:0: Video: h264, nv12(tv, bt709, progressive), 3840x2160 [SAR 1:1 DAR 16:9], q=2-31, 23.98 fps, 24k tbn (default)
      Metadata:
        encoder         : Lavc61.3.100 libx264
      Side data:
        cpb: bitrate max/min/avg: 25461000/0/0 buffer size: 50922000 vbv_delay: N/A
        DOVI configuration record: version: 1.0, profile: 8, level: 6, rpu flag: 1, el flag: 0, bl flag: 1, compatibility id: 1
  Stream #0:1: Audio: aac, 48000 Hz, stereo, s16, 256 kb/s (default)
      Metadata:
        encoder         : Lavc61.3.100 libfdk_aac
frame=    0 fps=0.0 q=0.0 size=N/A time=N/A bitrate=N/A speed=N/A    
frame=    0 fps=0.0 q=0.0 size=N/A time=N/A bitrate=N/A speed=N/A    
frame=   10 fps=6.7 q=28.0 size=N/A time=00:00:00.00 bitrate=N/A speed=   0x    
frame=   28 fps= 14 q=28.0 size=N/A time=00:00:00.75 bitrate=N/A speed=0.375x    
frame=   47 fps= 19 q=28.0 size=N/A time=00:00:01.54 bitrate=N/A speed=0.617x    
frame=   65 fps= 22 q=28.0 size=N/A time=00:00:02.29 bitrate=N/A speed=0.764x    
[hls @ 0x615c7eb7dc00] Opening '/config/cache/transcodes/2b65f8ac529c5a1ab511c2002bbf3fbd0.mp4' for writing
frame=   83 fps= 24 q=28.0 size=N/A time=00:00:03.04 bitrate=N/A speed=0.87x    
frame=  101 fps= 25 q=28.0 size=N/A time=00:00:03.79 bitrate=N/A speed=0.949x    
frame=  119 fps= 26 q=28.0 size=N/A time=00:00:04.54 bitrate=N/A speed=1.01x    
frame=  139 fps= 28 q=28.0 size=N/A time=00:00:05.38 bitrate=N/A speed=1.08x    
[hls @ 0x615c7eb7dc00] Opening '/config/cache/transcodes/2b65f8ac529c5a1ab511c2002bbf3fbd1.mp4' for writing
frame=  158 fps= 29 q=28.0 size=N/A time=00:00:06.17 bitrate=N/A speed=1.12x    
frame=  177 fps= 29 q=28.0 size=N/A time=00:00:06.96 bitrate=N/A speed=1.16x    
frame=  196 fps= 30 q=28.0 size=N/A time=00:00:07.75 bitrate=N/A speed=1.19x    
frame=  215 fps= 31 q=28.0 size=N/A time=00:00:08.55 bitrate=N/A speed=1.22x    
[hls @ 0x615c7eb7dc00] Opening '/config/cache/transcodes/2b65f8ac529c5a1ab511c2002bbf3fbd2.mp4' for writing

[TheDreadPirate]

You didn't enable hardware transcoding. It is still software transcoding. Share screenshots of Dashboard > Playback > Transcoding

Code:

Stream #0:0 -> #0:0 (hevc (hevc_qsv) -> h264 (libx264))

Also, is the Android phone using the web player or integrated player? Settings > Client settings. The integrated player will have better codec support.

[bentrades]

You didn't enable hardware transcoding.  It is still software transcoding.  Share screenshots of Dashboard > Playback > Transcoding

Code:

Stream #0:0 -> #0:0 (hevc (hevc_qsv) -> h264 (libx264))

Also, is the Android phone using the web player or integrated player?  Settings > Client settings.  The integrated player will have better codec support.

Thanks for the quick reply, from what I think I have Hardware Transcoding activated. Find attached the Settings Screenshot.
Okay will take a look in the app, thanks for pointing me towards it!
edit: Changed it to integrated player like you said. Transcoding stopped <3

[TheDreadPirate]

You did not check the box "enable hardware encoding". You can also check "allow encoding in HEVC format" and both tone mapping boxes.

[bentrades]

You did not check the box "enable hardware encoding".  You can also check "allow encoding in HEVC format" and both tone mapping boxes.

If I do this my transcoding from a different content fails, below is the ffmpeg log which kinda gives me the feeling i still have misconfigured it yet.

 _STATISTICS_TAGS-eng: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
Stream mapping:
  Stream #0:0 -> #0:0 (hevc (hevc_qsv) -> hevc (hevc_qsv))
  Stream #0:1 -> #0:1 (eac3 (native) -> aac (libfdk_aac))
Press [q] to stop, [?] for help
[hevc_qsv @ 0x61c550461800] Selected ratecontrol mode is unsupported
[hevc_qsv @ 0x61c550461800] some encoding parameters are not supported by the QSV runtime. Please double check the input parameters.
[vost#0:0/hevc_qsv @ 0x61c550035b40] Error while opening encoder - maybe incorrect parameters such as bit_rate, rate, width or height.
[vf#0:0 @ 0x61c5505bc900] Error sending frames to consumers: Function not implemented
[vf#0:0 @ 0x61c5505bc900] Task finished with error code: -38 (Function not implemented)
[vf#0:0 @ 0x61c5505bc900] Terminating thread with return code -38 (Function not implemented)
[vost#0:0/hevc_qsv @ 0x61c550035b40] Could not open encoder before EOF
[vost#0:0/hevc_qsv @ 0x61c550035b40] Task finished with error code: -22 (Invalid argument)
[vost#0:0/hevc_qsv @ 0x61c550035b40] Terminating thread with return code -22 (Invalid argument)
[out#0/hls @ 0x61c5505bca80] Nothing was written into output file, because at least one of its streams received no packets.
frame=    0 fps=0.0 q=0.0 Lsize=      0KiB time=N/A bitrate=N/A speed=N/A   
Conversion failed!

[TheDreadPirate]

I see that you enabled low power encoding. Did you install and setup the GuC and HuC firmware?

https://jellyfin.org/docs/general/admini...r-encoding

Those two settings are optional, btw. You can disable them if you don't want to bother setting up the firmware.

[bentrades]

I did the driver activation but are unable to get it working with both low power encoders checked.
However I was able I think to set up Hardware Based Transcoding and are now at 41 FPS when transcoding a HEVC 10bit SDR with 16 mbits to 15. And now really the CPU is just idling around how I expected it to be. Assuming i have reached the right configuration.

Stream mapping:
Stream #0:0 -> #0:0 (hevc (hevc_qsv) -> hevc (hevc_qsv))
Stream #0:1 -> #0:1 (eac3 (native) -> aac (libfdk_aac))
Press [q] to stop, [?] for help
[hls @ 0x5eac22f6ef40] Opening '/config/cache/transcodes/44523164c44ca1a35593492e9099b7d6-1.mp4' for writing
Output #0, hls, to '/config/cache/transcodes/44523164c44ca1a35593492e9099b7d6.m3u8':
Metadata:
encoder : Lavf61.1.100
Stream #0:0: Video: hevc (hvc1 / 0x3163766, qsv(tv, bt709, progressive), 3840x2160 [SAR 1:1 DAR 16:9], q=2-31, 14616 kb/s, 23.98 fps, 24k tbn (default)
Metadata:
encoder : Lavc61.3.100 hevc_qsv
Side data:
cpb: bitrate max/min/avg: 14616000/0/14616000 buffer size: 29232000 vbv_delay: N/A
Stream #0:1: Audio: aac, 48000 Hz, stereo, s16, 256 kb/s (default)
Metadata:
encoder : Lavc61.3.100 libfdk_aac
frame= 12 fps=0.0 q=-0.0 size=N/A time=00:00:00.00 bitrate=N/A speed= 0x
frame= 33 fps= 33 q=-0.0 size=N/A time=00:00:00.87 bitrate=N/A speed=0.876x
frame= 54 fps= 36 q=-0.0 size=N/A time=00:00:01.75 bitrate=N/A speed=1.17x
[hls @ 0x5eac22f6ef40] Opening '/config/cache/transcodes/44523164c44ca1a35593492e9099b7d641.mp4' for writing
frame= 77 fps= 38 q=-0.0 size=N/A time=00:00:02.71 bitrate=N/A speed=1.36x
frame= 97 fps= 39 q=-0.0 size=N/A time=00:00:03.58 bitrate=N/A speed=1.43x
frame= 119 fps= 40 q=-0.0 size=N/A time=00:00:04.46 bitrate=N/A speed=1.49x
frame= 145 fps= 41 q=-0.0 size=N/A time=00:00:05.54 bitrate=N/A speed=1.58x
[hls @ 0x5eac22f6ef40] Opening '/config/cache/transcodes/44523164c44ca1a35593492e9099b7d642.mp4' for writing