Couldn't Enable Hardware Acceleration with VAAPI on WSL2

[SessionID]

Hi,

I discovered Jellyfin recently and loved it. But I've encountered with a problem that makes the software unusable for me so I would really appreciate if we can find a way to solve it.

I use Windows 11 Pro 23H2 with WSL2 and I have an AMD Ryzen 9 CPU with integrated graphics. If I just use the installer for Windows and enable the hardware acceleration, it will work without any problem. But I want to install Jellyfin as a Docker container inside WSL2. It really seems doable since WSL2 supports D3D12 GPU video acceleration. But I couldn't manage to work it.

I followed this guide from Microsoft to enable hardware acceleration: https://devblogs.microsoft.com/commandli...available/

Here is the vainfo output at WSL2 terminal, everything seems perfect until now:

Code:

$ vainfo --display drm --device /dev/dri/card0
libva info: VA-API version 1.14.0
libva info: User environment variable requested driver 'd3d12'
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/d3d12_drv_video.so
libva info: Found init function __vaDriverInit_1_14
libva info: va_openDriver() returns 0
vainfo: VA-API version: 1.14 (libva 2.12.0)
vainfo: Driver version: Mesa Gallium driver 23.0.4-0ubuntu1~22.04.1 for D3D12 (AMD Radeon(TM) Graphics)
vainfo: Supported profile and entrypoints
      VAProfileH264ConstrainedBaseline: VAEntrypointVLD
      VAProfileH264ConstrainedBaseline: VAEntrypointEncSlice
      VAProfileH264Main              : VAEntrypointVLD
      VAProfileH264Main              : VAEntrypointEncSlice
      VAProfileH264High              : VAEntrypointVLD
      VAProfileH264High              : VAEntrypointEncSlice
      VAProfileHEVCMain              : VAEntrypointVLD
      VAProfileHEVCMain              : VAEntrypointEncSlice
      VAProfileHEVCMain10            : VAEntrypointVLD
      VAProfileHEVCMain10            : VAEntrypointEncSlice
      VAProfileVP9Profile0            : VAEntrypointVLD
      VAProfileVP9Profile2            : VAEntrypointVLD
      VAProfileAV1Profile0            : VAEntrypointVLD
      VAProfileNone                  : VAEntrypointVideoProc

So I thought I could just map the device to Docker container and make it work. Here is my Docker Compose file:

Code:

version: '3.5'
services:
  jellyfin:
    image: linuxserver/jellyfin:latest
    container_name: jellyfin
    devices:
      - /dev/dri:/dev/dri
    environment:
      - LIBVA_DRIVER_NAME=d3d12
      - PUID=1000
      - PGID=1000
    network_mode: 'host'
    volumes:
      - /path/to/config:/config
      - /path/to/media:/media
    restart: unless-stopped

With this configuration, Jellyfish runs perfect. I can use it with software encoding, it functions great. But things get really confusing when I enable VAAPI. My playback settings after I enable VAAPI:

   

When I try to play a video with these settings, I get "This client isn't compatible with the media and the server isn't sending a compatible media format" error. I can see some details when I check ffmpeg logs. It is not so useful but I guess better than nothing:

Code:

/usr/lib/jellyfin-ffmpeg/ffmpeg -analyzeduration 200M -init_hw_device vaapi=va:/dev/dri/card0 -filter_hw_device va -hwaccel vaapi -hwaccel_output_format vaapi -autorotate 0 -i file:"sample.mkv" -autoscale 0 -map_metadata -1 -map_chapters -1 -threads 0 -map 0:0 -map 0:1 -map -0:s -codec:v:0 h264_vaapi -rc_mode VBR -b:v 3722355 -maxrate 3722355 -bufsize 7444710 -force_key_frames:0 "expr:gte(t,0+n_forced*3)" -vf "setparams=color_primaries=bt709:color_trc=bt709:colorspace=bt709,scale_vaapi=format=nv12:extra_hw_frames=24" -codec:a:0 libfdk_aac -ac 2 -ab 71958 -ar 48000 -copyts -avoid_negative_ts disabled -max_muxing_queue_size 2048 -f hls -max_delay 5000000 -hls_time 3 -hls_segment_type mpegts -start_number 0 -hls_segment_filename "/config/data/transcodes/c22dfb1580de3e13a974967922d10a39%d.ts" -hls_playlist_type vod -hls_list_size 0 -y "/config/data/transcodes/c22dfb1580de3e13a974967922d10a39.m3u8"


ffmpeg version 5.1.4-Jellyfin Copyright (c) 2000-2023 the FFmpeg developers
  built with gcc 11 (Ubuntu 11.4.0-1ubuntu1~22.04)
  configuration: --prefix=/usr/lib/jellyfin-ffmpeg --target-os=linux --extra-libs=-lfftw3f --extra-version=Jellyfin --disable-doc --disable-ffplay --disable-ptx-compression --disable-static --disable-libxcb --disable-sdl2 --disable-xlib --enable-lto --enable-gpl --enable-version3 --enable-shared --enable-gmp --enable-gnutls --enable-chromaprint --enable-libdrm --enable-libass --enable-libfreetype --enable-libfribidi --enable-libfontconfig --enable-libbluray --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libopenmpt --enable-libdav1d --enable-libwebp --enable-libvpx --enable-libx264 --enable-libx265 --enable-libzvbi --enable-libzimg --enable-libfdk-aac --arch=amd64 --enable-libsvtav1 --enable-libshaderc --enable-libplacebo --enable-vulkan --enable-opencl --enable-vaapi --enable-amf --enable-libmfx --enable-ffnvcodec --enable-cuda --enable-cuda-llvm --enable-cuvid --enable-nvdec --enable-nvenc
  libavutil      57. 28.100 / 57. 28.100
  libavcodec    59. 37.100 / 59. 37.100
  libavformat    59. 27.100 / 59. 27.100
  libavdevice    59.  7.100 / 59.  7.100
  libavfilter    8. 44.100 /  8. 44.100
  libswscale      6.  7.100 /  6.  7.100
  libswresample  4.  7.100 /  4.  7.100
  libpostproc    56.  6.100 / 56.  6.100
[AVHWDeviceContext @ 0x561ea12e6980] libva: /usr/lib/x86_64-linux-gnu/dri/d3d12_drv_video.so init failed
[AVHWDeviceContext @ 0x561ea12e6980] Failed to initialise VAAPI connection: 2 (resource allocation failed).
Device creation failed: -5.
Failed to set value 'vaapi=va:/dev/dri/card0' for option 'init_hw_device': Input/output error
Error parsing global options: Input/output error

My vainfo output in the container is as follows:

Code:

$ docker exec -it -e PUID=1000 -e PGID=1000 jellyfin /bin/sh
# ./usr/share/jellyfin-ffmpeg/vainfo --display drm --device /dev/dri/card0
Trying display: drm
libva info: VA-API version 1.20.0
libva info: Trying to open /usr/lib/jellyfin-ffmpeg/lib/dri/d3d12_drv_video.so
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/d3d12_drv_video.so
libva info: Found init function __vaDriverInit_1_14
libva error: /usr/lib/x86_64-linux-gnu/dri/d3d12_drv_video.so init failed
libva info: va_openDriver() returns 2
vaInitialize failed with error code 2 (resource allocation failed),exit

At this point, I am not quite sure what I am doing wrong. I have a feeling like maybe d3d12 driver is not supported with Jellyfin, since the driver is not available under /usr/lib/jellyfin-ffmpeg/lib/dri and it falls back to /usr/lib/x86_64-linux-gnu/dri.

How can we solve this issue? My real intention is using Jellyfin with CasaOS under WSL2. I guess it would help lots of people who is in the same situation with me. Thanks.