10.9 broke QSV VPP encoding

[TheMaFa]

My server contains the i5-13500 with the UHD 770 iGPU. I followed the guide on hardware acceleration thoroughly, three times. Double checked every single thing.
On the previous version 10.8.x everything worked like a charm. That means hardware encoding and low power encoding worked.
But as I upgraded to 10.9.1 QSV hardware encoding broke.

What I mean with that is, that when I disable the "Hardware encoding" setting, the video transcodes just fine (very slow but that doesn't matter).
When I enable it though I get the default transcoding error message "fatal error...".

Here are the crucial ffmpeg log lines:
libva info: VA-API version 1.21.0
libva info: Trying to open /usr/lib/jellyfin-ffmpeg/lib/dri/iHD_drv_video.so
libva info: Found init function __vaDriverInit_1_21
libva info: va_openDriver() returns 0
libva info: VA-API version 1.21.0
libva info: Trying to open /usr/lib/jellyfin-ffmpeg/lib/dri/iHD_drv_video.so
libva info: Found init function __vaDriverInit_1_21
libva info: va_openDriver() returns 0
[h264_qsv @ 0x6284ec691f80] Error querying (IOSurf) the encoding parameters: invalid video parameters (-15)
[vost#0:0/h264_qsv @ 0x6284ec676880] Error initializing output stream: Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height
[libfdk_aac @ 0x6284ec69b380] 2 frames left in the queue on closing
Conversion failed!


Now it's crucial to say that this is not a problem with my setup, because when I switch over to VA-API it just works, even with hardware encoding enabled.
The only thing that bothers me is that transcoding with VA-API is relatively slow. I mean it worked with 10.8, I upgraded, then it broke.
I followed the new hardware acceleration guide where ffmpeg6 is mentioned, deleting deprecated jellyfin package, etc., if that matters.

Jellyfin is running in an LXC and this appears on the host when doing dmesg | grep i915:
[    5.697380] i915 0000:00:02.0: vgaarb: deactivate vga console
[    5.697426] i915 0000:00:02.0: [drm] Using Transparent Hugepages
[    5.698073] i915 0000:00:02.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=io+mem"colon"owns=io+mem
[    5.699113] i915 0000:00:02.0: [drm] Finished loading DMC firmware i915/adls_dmc_ver2_01.bin (v2.1)
[    5.699154] mei_hdcp 0000:00:16.0-b638ab7e-94e2-4ea2-a552-d1c54b627f04: bound 0000:00:02.0 (ops i915_hdcp_ops [i915])
[    5.703765] i915 0000:00:02.0: [drm] GT0: GuC firmware i915/tgl_guc_70.bin version 70.20.0
[    5.703770] i915 0000:00:02.0: [drm] GT0: HuC firmware i915/tgl_huc.bin version 7.9.3
[    5.717093] i915 0000:00:02.0: [drm] GT0: HuC: authenticated for all workloads
[    5.717100] i915 0000:00:02.0: [drm] GT0: GUC: submission disabled
[    5.717102] i915 0000:00:02.0: [drm] GT0: GUC: SLPC disabled
[    5.717958] mei_pxp 0000:00:16.0-fbf6fcf1-96cf-4e2e-a6a6-1bab8cbe36b1: bound 0000:00:02.0 (ops i915_pxp_tee_component_ops [i915])
[    5.718059] i915 0000:00:02.0: [drm] Protected Xe Path (PXP) protected content support initialized
[    5.718596] [drm] Initialized i915 1.6.0 20230929 for 0000:00:02.0 on minor 0
[    5.719785] i915 display info: display version: 12
[    5.719786] i915 display info: cursor_needs_physical: no
[    5.719787] i915 display info: has_cdclk_crawl: no
[    5.719788] i915 display info: has_cdclk_squash: no
[    5.719788] i915 display info: has_ddi: yes
[    5.719789] i915 display info: has_dp_mst: yes
[    5.719790] i915 display info: has_dsb: yes
[    5.719791] i915 display info: has_fpga_dbg: yes
[    5.719791] i915 display info: has_gmch: no
[    5.719792] i915 display info: has_hotplug: yes
[    5.719793] i915 display info: has_hti: yes
[    5.719793] i915 display info: has_ipc: yes
[    5.719794] i915 display info: has_overlay: no
[    5.719795] i915 display info: has_psr: yes
[    5.719796] i915 display info: has_psr_hw_tracking: no
[    5.719796] i915 display info: overlay_needs_physical: no
[    5.719797] i915 display info: supports_tv: no
[    5.719798] i915 display info: has_hdcp: yes
[    5.719798] i915 display info: has_dmc: yes
[    5.719799] i915 display info: has_dsc: yes

seems fine.

The OpenCL runtime status (in LXC):
[AVHWDeviceContext @ 0x5f36a6c756c0] libva: VA-API version 1.21.0
[AVHWDeviceContext @ 0x5f36a6c756c0] libva: Trying to open /usr/lib/jellyfin-ffmpeg/lib/dri/iHD_drv_video.so
[AVHWDeviceContext @ 0x5f36a6c756c0] libva: Found init function __vaDriverInit_1_21
[AVHWDeviceContext @ 0x5f36a6c756c0] libva: va_openDriver() returns 0
[AVHWDeviceContext @ 0x5f36a6c756c0] Initialised VAAPI connection: version 1.21
[AVHWDeviceContext @ 0x5f36a6c756c0] VAAPI driver: Intel iHD driver for Intel® Gen Graphics - 24.2.1 (0593864).
[AVHWDeviceContext @ 0x5f36a6c756c0] Driver not found in known nonstandard list, using standard behaviour.
[AVHWDeviceContext @ 0x5f36a6ca7680] 0.0: Intel® OpenCL Graphics / Intel® UHD Graphics 770
[AVHWDeviceContext @ 0x5f36a6ca7680] Intel QSV to OpenCL mapping function found (clCreateFromVA_APIMediaSurfaceINTEL).
[AVHWDeviceContext @ 0x5f36a6ca7680] Intel QSV in OpenCL acquire function found (clEnqueueAcquireVA_APIMediaSurfacesINTEL).
[AVHWDeviceContext @ 0x5f36a6ca7680] Intel QSV in OpenCL release function found (clEnqueueReleaseVA_APIMediaSurfacesINTEL).
Hyper fast Audio and Video encoder

seems fine too.

QSV and VA-API codecs (first few lines):
Trying display: drm
libva info: VA-API version 1.21.0
libva info: Trying to open /usr/lib/jellyfin-ffmpeg/lib/dri/iHD_drv_video.so
libva info: Found init function __vaDriverInit_1_21
libva info: va_openDriver() returns 0
vainfo: VA-API version: 1.21 (libva 2.21.0)
vainfo: Driver version: Intel iHD driver for Intel® Gen Graphics - 24.2.1 (0593864)

seems fine too!
I installed the intel compute runtime packages on both the host and the LXC because I wasn't sure:
intel-opencl-icd:
  Installed: 24.13.29138.7
  Candidate: 24.13.29138.7
  Version table:
*** 24.13.29138.7 100
        100 /var/lib/dpkg/status

it shows this in both instances.


Any help would be greatly appreciated, thanks in advance!

[TheMaFa]

What I might add is, that I have NOT done this step, as advised:
sudo mkdir -p /etc/modprobe.d
sudo sh -c "echo 'options i915 enable_guc=2' >> /etc/modprobe.d/i915.conf"

And no mather what number I choose, this directory doesn't exist:
/sys/kernel/debug/dri/1/gt/uc

[TheDreadPirate]

Check this directory.

/sys/kernel/debug/dri/1/gt0/uc

Notice the 0 at the end of gt.

Also, I don't think the guc/huc firmware is required for 13th gen iGPUs. But you can enable them for better tone mapping performance.

What is your underlying distro and what kernel are you running?

[TheMaFa]

Alright, so this seems to be the correct directory:
/sys/kernel/debug/dri/0000:00:02.0/gt0/uc
Which makes sense since this is the PCIe ID.
Both the GuC and HuC firmware is running and no errors are shown.

I run Proxmox VE 8.2.2 as the underlying host system.
uname -mrs: Linux 6.8.4-3-pve x86_64
It runs Debian 12 (Bookworm). Jellyfin sits in a privileged LXC.
But as I said, the passthrough doesn't seem to be the problem.

[TheDreadPirate]

@gnattu @nyanmisaka any ideas?

[TheMaFa]

I checked again. The transcoding error occurs regardless of the source file resolution.
I downgraded to Jellyfin 10.8.x to see if something fundamental broke, but it worked again (fully).
Upgraded to 10.9.1 and again, there is the error.

Rebooted the host system, still persists.

[TheDreadPirate]

A thought occurred. When you upgraded to Jellyfin 10.9, did you also install jellyfin-ffmpeg6?

[TheMaFa]

It does that automatically. At least it did for me. The official hardware acceleration guide mentions ffmpeg6 too, and I noticed a couple changed overall which means it was updated for 10.9.

Edit: It is set to conflict with jellyfin-ffmpeg5 hence there can be no wrong version installed.

[TheDreadPirate]

Just making sure. Its possible to remove the meta package and update server and web separate from jellyfin-ffmpeg. At least one person here in the last week still had jellyfin-ffmpeg5 with server/web 10.9 due to not having the meta package installed.

[TheMaFa]

That's confusing because in the hardware acceleration guide I read that the jellyfin meta package is deprecated and can break dependencies.
Whatever, I have removed it and I will try upgrading to 10.9.2 to see if that somehow resolves the issue.

Update: It did not. Same error message. VA-API still works.