Intel HW transcoding not working

[jvanstensel]

This mixes proxmox and jellyfin troubleshooting, but right now my issue appears to be with jellyfin, so I'm posting this here.

I cannot for the life of me get hardware transcoding to run on my Intel 6700K (Skylake).  I followed the instructions on the Jellyfin website for an Intel GPU and everything seems to be correct, but my GPU utilization never increases above zero.  I tried both QSV and VAAPI, neither worked.

I'm using a privileged container in proxmox and passing the /dev/dri/card1 and /dev/dri/renderD128 resources through to the privileged LXC container.  It seems like the resource gets passed through to the container.  However, whenever I try and transode a video, it is only using the processor.  I'm transcoding a wtv file which is just a container around mpeg2.  I can see in the logs that it is transcoding, but it never uses hardware transcoding as indicated by the log and intel-gpu-tools. I'm using Jellyfin v10.11.0 RC4.

I attached the jellyfin logs since I'm sure there are some useful nuggets there.  There are 2 logs because I'm running on 2 CPUs.

I verified the GPU pass through with the following commands.
root@jellyfin:/var/log/jellyfin# /usr/lib/jellyfin-ffmpeg/vainfo --display drm --device /dev/dri/renderD128
Trying display: drm
libva info: VA-API version 1.22.0
libva info: Trying to open /usr/lib/jellyfin-ffmpeg/lib/dri/iHD_drv_video.so
libva info: Found init function __vaDriverInit_1_22
libva info: va_openDriver() returns 0
vainfo: VA-API version: 1.22 (libva 2.22.0)
vainfo: Driver version: Intel iHD driver for Intel® Gen Graphics - 25.2.6 (c6ab0c9)
vainfo: Supported profile and entrypoints
      VAProfileNone                  : VAEntrypointVideoProc
      VAProfileNone                  : VAEntrypointStats
      VAProfileMPEG2Simple            : VAEntrypointVLD
      VAProfileMPEG2Simple            : VAEntrypointEncSlice
      VAProfileMPEG2Main              : VAEntrypointVLD
      VAProfileMPEG2Main              : VAEntrypointEncSlice
      VAProfileH264Main              : VAEntrypointVLD
      VAProfileH264Main              : VAEntrypointEncSlice
      VAProfileH264Main              : VAEntrypointFEI
      VAProfileH264Main              : VAEntrypointEncSliceLP
      VAProfileH264High              : VAEntrypointVLD
      VAProfileH264High              : VAEntrypointEncSlice
      VAProfileH264High              : VAEntrypointFEI
      VAProfileH264High              : VAEntrypointEncSliceLP
      VAProfileVC1Simple              : VAEntrypointVLD
      VAProfileVC1Main                : VAEntrypointVLD
      VAProfileVC1Advanced            : VAEntrypointVLD
      VAProfileJPEGBaseline          : VAEntrypointVLD
      VAProfileJPEGBaseline          : VAEntrypointEncPicture
      VAProfileH264ConstrainedBaseline: VAEntrypointVLD
      VAProfileH264ConstrainedBaseline: VAEntrypointEncSlice
      VAProfileH264ConstrainedBaseline: VAEntrypointFEI
      VAProfileH264ConstrainedBaseline: VAEntrypointEncSliceLP
      VAProfileVP8Version0_3          : VAEntrypointVLD
      VAProfileVP8Version0_3          : VAEntrypointEncSlice
      VAProfileHEVCMain              : VAEntrypointVLD
      VAProfileHEVCMain              : VAEntrypointEncSlice
      VAProfileHEVCMain              : VAEntrypointFEI

root@jellyfin:/var/log/jellyfin# /usr/lib/jellyfin-ffmpeg/ffmpeg -v verbose -init_hw_device vaapi=va:/dev/dri/renderD128 -init_hw_device opencl@va
ffmpeg version 7.1.1-Jellyfin Copyright © 2000-2025 the FFmpeg developers
  built with gcc 12 (Debian 12.2.0-14+deb12u1)
  configuration: --prefix=/usr/lib/jellyfin-ffmpeg --target-os=linux --extra-version=Jellyfin --disable-doc --disable-ffplay --disable-static --disable-libxcb --disable-sdl2 --disable-xlib --enable-lto=auto --enable-gpl --enable-version3 --enable-shared --enable-gmp --enable-gnutls --enable-chromaprint --enable-opencl --enable-libdrm --enable-libxml2 --enable-libass --enable-libfreetype --enable-libfribidi --enable-libfontconfig --enable-libharfbuzz --enable-libbluray --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libopenmpt --enable-libdav1d --enable-libsvtav1 --enable-libwebp --enable-libvpx --enable-libx264 --enable-libx265 --enable-libzvbi --enable-libzimg --enable-libfdk-aac --arch=amd64 --enable-libshaderc --enable-libplacebo --enable-vulkan --enable-vaapi --enable-amf --enable-libvpl --enable-ffnvcodec --enable-cuda --enable-cuda-llvm --enable-cuvid --enable-nvdec --enable-nvenc
  libavutil      59. 39.100 / 59. 39.100
  libavcodec    61. 19.101 / 61. 19.101
  libavformat    61.  7.100 / 61.  7.100
  libavdevice    61.  3.100 / 61.  3.100
  libavfilter    10.  4.100 / 10.  4.100
  libswscale      8.  3.100 /  8.  3.100
  libswresample  5.  3.100 /  5.  3.100
  libpostproc    58.  3.100 / 58.  3.100
[AVHWDeviceContext @ 0x5d1849755b80] libva: VA-API version 1.22.0
[AVHWDeviceContext @ 0x5d1849755b80] libva: Trying to open /usr/lib/jellyfin-ffmpeg/lib/dri/iHD_drv_video.so
[AVHWDeviceContext @ 0x5d1849755b80] libva: Found init function __vaDriverInit_1_22
[AVHWDeviceContext @ 0x5d1849755b80] libva: va_openDriver() returns 0
[AVHWDeviceContext @ 0x5d1849755b80] Initialised VAAPI connection: version 1.22
[AVHWDeviceContext @ 0x5d1849755b80] VAAPI driver: Intel iHD driver for Intel® Gen Graphics - 25.2.6 (c6ab0c9).
[AVHWDeviceContext @ 0x5d1849755b80] Driver not found in known nonstandard list, using standard behaviour.
[AVHWDeviceContext @ 0x5d184978dc80] 0.0: Intel® OpenCL HD Graphics / Intel® HD Graphics 530 [0x1912]
[AVHWDeviceContext @ 0x5d184978dc80] Intel QSV to OpenCL mapping function found (clCreateFromVA_APIMediaSurfaceINTEL).
[AVHWDeviceContext @ 0x5d184978dc80] Intel QSV in OpenCL acquire function found (clEnqueueAcquireVA_APIMediaSurfacesINTEL).
[AVHWDeviceContext @ 0x5d184978dc80] Intel QSV in OpenCL release function found (clEnqueueReleaseVA_APIMediaSurfacesINTEL).
Universal media converter
usage: ffmpeg [options] [[infile options] -i infile]... {[outfile options] outfile}...

Use -h to get full help or, even better, run 'man ffmpeg'

[jvanstensel]

I think I figured it out. A different wtv file worked with hardware transcoding. I decided to try and stream copy the container from wtv to mp4 and the previously not working hardware transcoding started working. It seems like ffmpeg support for wtv transcoding is limited. That is actually why I'm on v10.11.0 RC4 in the first place because there was a bug with wtv transcoding and the -fps_mode switch.

I wrote a script to stream copy my wtv files into mp4 files. I'll test with that a bit to see if that has any impact and then I'll convert all my wtv files.

[jvanstensel]

There is an impact. I lose the meta-data.