Intel HW transcoding not working

[jvanstensel]

This mixes proxmox and jellyfin troubleshooting, but right now my issue appears to be with jellyfin, so I'm posting this here.

I cannot for the life of me get hardware transcoding to run on my Intel 6700K (Skylake).  I followed the instructions on the Jellyfin website for an Intel GPU and everything seems to be correct, but my GPU utilization never increases above zero.  I tried both QSV and VAAPI, neither worked.

I'm using a privileged container in proxmox and passing the /dev/dri/card1 and /dev/dri/renderD128 resources through to the privileged LXC container.  It seems like the resource gets passed through to the container.  However, whenever I try and transode a video, it is only using the processor.  I'm transcoding a wtv file which is just a container around mpeg2.  I can see in the logs that it is transcoding, but it never uses hardware transcoding as indicated by the log and intel-gpu-tools. I'm using Jellyfin v10.11.0 RC4.

I attached the jellyfin logs since I'm sure there are some useful nuggets there.  There are 2 logs because I'm running on 2 CPUs.

I verified the GPU pass through with the following commands.
root@jellyfin:/var/log/jellyfin# /usr/lib/jellyfin-ffmpeg/vainfo --display drm --device /dev/dri/renderD128
Trying display: drm
libva info: VA-API version 1.22.0
libva info: Trying to open /usr/lib/jellyfin-ffmpeg/lib/dri/iHD_drv_video.so
libva info: Found init function __vaDriverInit_1_22
libva info: va_openDriver() returns 0
vainfo: VA-API version: 1.22 (libva 2.22.0)
vainfo: Driver version: Intel iHD driver for Intel® Gen Graphics - 25.2.6 (c6ab0c9)
vainfo: Supported profile and entrypoints
      VAProfileNone                  : VAEntrypointVideoProc
      VAProfileNone                  : VAEntrypointStats
      VAProfileMPEG2Simple            : VAEntrypointVLD
      VAProfileMPEG2Simple            : VAEntrypointEncSlice
      VAProfileMPEG2Main              : VAEntrypointVLD
      VAProfileMPEG2Main              : VAEntrypointEncSlice
      VAProfileH264Main              : VAEntrypointVLD
      VAProfileH264Main              : VAEntrypointEncSlice
      VAProfileH264Main              : VAEntrypointFEI
      VAProfileH264Main              : VAEntrypointEncSliceLP
      VAProfileH264High              : VAEntrypointVLD
      VAProfileH264High              : VAEntrypointEncSlice
      VAProfileH264High              : VAEntrypointFEI
      VAProfileH264High              : VAEntrypointEncSliceLP
      VAProfileVC1Simple              : VAEntrypointVLD
      VAProfileVC1Main                : VAEntrypointVLD
      VAProfileVC1Advanced            : VAEntrypointVLD
      VAProfileJPEGBaseline          : VAEntrypointVLD
      VAProfileJPEGBaseline          : VAEntrypointEncPicture
      VAProfileH264ConstrainedBaseline: VAEntrypointVLD
      VAProfileH264ConstrainedBaseline: VAEntrypointEncSlice
      VAProfileH264ConstrainedBaseline: VAEntrypointFEI
      VAProfileH264ConstrainedBaseline: VAEntrypointEncSliceLP
      VAProfileVP8Version0_3          : VAEntrypointVLD
      VAProfileVP8Version0_3          : VAEntrypointEncSlice
      VAProfileHEVCMain              : VAEntrypointVLD
      VAProfileHEVCMain              : VAEntrypointEncSlice
      VAProfileHEVCMain              : VAEntrypointFEI

root@jellyfin:/var/log/jellyfin# /usr/lib/jellyfin-ffmpeg/ffmpeg -v verbose -init_hw_device vaapi=va:/dev/dri/renderD128 -init_hw_device opencl@va
ffmpeg version 7.1.1-Jellyfin Copyright © 2000-2025 the FFmpeg developers
  built with gcc 12 (Debian 12.2.0-14+deb12u1)
  configuration: --prefix=/usr/lib/jellyfin-ffmpeg --target-os=linux --extra-version=Jellyfin --disable-doc --disable-ffplay --disable-static --disable-libxcb --disable-sdl2 --disable-xlib --enable-lto=auto --enable-gpl --enable-version3 --enable-shared --enable-gmp --enable-gnutls --enable-chromaprint --enable-opencl --enable-libdrm --enable-libxml2 --enable-libass --enable-libfreetype --enable-libfribidi --enable-libfontconfig --enable-libharfbuzz --enable-libbluray --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libopenmpt --enable-libdav1d --enable-libsvtav1 --enable-libwebp --enable-libvpx --enable-libx264 --enable-libx265 --enable-libzvbi --enable-libzimg --enable-libfdk-aac --arch=amd64 --enable-libshaderc --enable-libplacebo --enable-vulkan --enable-vaapi --enable-amf --enable-libvpl --enable-ffnvcodec --enable-cuda --enable-cuda-llvm --enable-cuvid --enable-nvdec --enable-nvenc
  libavutil      59. 39.100 / 59. 39.100
  libavcodec    61. 19.101 / 61. 19.101
  libavformat    61.  7.100 / 61.  7.100
  libavdevice    61.  3.100 / 61.  3.100
  libavfilter    10.  4.100 / 10.  4.100
  libswscale      8.  3.100 /  8.  3.100
  libswresample  5.  3.100 /  5.  3.100
  libpostproc    58.  3.100 / 58.  3.100
[AVHWDeviceContext @ 0x5d1849755b80] libva: VA-API version 1.22.0
[AVHWDeviceContext @ 0x5d1849755b80] libva: Trying to open /usr/lib/jellyfin-ffmpeg/lib/dri/iHD_drv_video.so
[AVHWDeviceContext @ 0x5d1849755b80] libva: Found init function __vaDriverInit_1_22
[AVHWDeviceContext @ 0x5d1849755b80] libva: va_openDriver() returns 0
[AVHWDeviceContext @ 0x5d1849755b80] Initialised VAAPI connection: version 1.22
[AVHWDeviceContext @ 0x5d1849755b80] VAAPI driver: Intel iHD driver for Intel® Gen Graphics - 25.2.6 (c6ab0c9).
[AVHWDeviceContext @ 0x5d1849755b80] Driver not found in known nonstandard list, using standard behaviour.
[AVHWDeviceContext @ 0x5d184978dc80] 0.0: Intel® OpenCL HD Graphics / Intel® HD Graphics 530 [0x1912]
[AVHWDeviceContext @ 0x5d184978dc80] Intel QSV to OpenCL mapping function found (clCreateFromVA_APIMediaSurfaceINTEL).
[AVHWDeviceContext @ 0x5d184978dc80] Intel QSV in OpenCL acquire function found (clEnqueueAcquireVA_APIMediaSurfacesINTEL).
[AVHWDeviceContext @ 0x5d184978dc80] Intel QSV in OpenCL release function found (clEnqueueReleaseVA_APIMediaSurfacesINTEL).
Universal media converter
usage: ffmpeg [options] [[infile options] -i infile]... {[outfile options] outfile}...

Use -h to get full help or, even better, run 'man ffmpeg'