Couldn't Enable Hardware Acceleration with VAAPI on WSL2

[SessionID]

Hi,

I discovered Jellyfin recently and loved it. But I've encountered with a problem that makes the software unusable for me so I would really appreciate if we can find a way to solve it.

I use Windows 11 Pro 23H2 with WSL2 and I have an AMD Ryzen 9 CPU with integrated graphics. If I just use the installer for Windows and enable the hardware acceleration, it will work without any problem. But I want to install Jellyfin as a Docker container inside WSL2. It really seems doable since WSL2 supports D3D12 GPU video acceleration. But I couldn't manage to work it.

I followed this guide from Microsoft to enable hardware acceleration: https://devblogs.microsoft.com/commandli...available/

Here is the vainfo output at WSL2 terminal, everything seems perfect until now:

Code:

$ vainfo --display drm --device /dev/dri/card0
libva info: VA-API version 1.14.0
libva info: User environment variable requested driver 'd3d12'
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/d3d12_drv_video.so
libva info: Found init function __vaDriverInit_1_14
libva info: va_openDriver() returns 0
vainfo: VA-API version: 1.14 (libva 2.12.0)
vainfo: Driver version: Mesa Gallium driver 23.0.4-0ubuntu1~22.04.1 for D3D12 (AMD Radeon(TM) Graphics)
vainfo: Supported profile and entrypoints
      VAProfileH264ConstrainedBaseline: VAEntrypointVLD
      VAProfileH264ConstrainedBaseline: VAEntrypointEncSlice
      VAProfileH264Main              : VAEntrypointVLD
      VAProfileH264Main              : VAEntrypointEncSlice
      VAProfileH264High              : VAEntrypointVLD
      VAProfileH264High              : VAEntrypointEncSlice
      VAProfileHEVCMain              : VAEntrypointVLD
      VAProfileHEVCMain              : VAEntrypointEncSlice
      VAProfileHEVCMain10            : VAEntrypointVLD
      VAProfileHEVCMain10            : VAEntrypointEncSlice
      VAProfileVP9Profile0            : VAEntrypointVLD
      VAProfileVP9Profile2            : VAEntrypointVLD
      VAProfileAV1Profile0            : VAEntrypointVLD
      VAProfileNone                  : VAEntrypointVideoProc

So I thought I could just map the device to Docker container and make it work. Here is my Docker Compose file:

Code:

version: '3.5'
services:
  jellyfin:
    image: linuxserver/jellyfin:latest
    container_name: jellyfin
    devices:
      - /dev/dri:/dev/dri
    environment:
      - LIBVA_DRIVER_NAME=d3d12
      - PUID=1000
      - PGID=1000
    network_mode: 'host'
    volumes:
      - /path/to/config:/config
      - /path/to/media:/media
    restart: unless-stopped

With this configuration, Jellyfish runs perfect. I can use it with software encoding, it functions great. But things get really confusing when I enable VAAPI. My playback settings after I enable VAAPI:

   

When I try to play a video with these settings, I get "This client isn't compatible with the media and the server isn't sending a compatible media format" error. I can see some details when I check ffmpeg logs. It is not so useful but I guess better than nothing:

Code:

/usr/lib/jellyfin-ffmpeg/ffmpeg -analyzeduration 200M -init_hw_device vaapi=va:/dev/dri/card0 -filter_hw_device va -hwaccel vaapi -hwaccel_output_format vaapi -autorotate 0 -i file:"sample.mkv" -autoscale 0 -map_metadata -1 -map_chapters -1 -threads 0 -map 0:0 -map 0:1 -map -0:s -codec:v:0 h264_vaapi -rc_mode VBR -b:v 3722355 -maxrate 3722355 -bufsize 7444710 -force_key_frames:0 "expr:gte(t,0+n_forced*3)" -vf "setparams=color_primaries=bt709:color_trc=bt709:colorspace=bt709,scale_vaapi=format=nv12:extra_hw_frames=24" -codec:a:0 libfdk_aac -ac 2 -ab 71958 -ar 48000 -copyts -avoid_negative_ts disabled -max_muxing_queue_size 2048 -f hls -max_delay 5000000 -hls_time 3 -hls_segment_type mpegts -start_number 0 -hls_segment_filename "/config/data/transcodes/c22dfb1580de3e13a974967922d10a39%d.ts" -hls_playlist_type vod -hls_list_size 0 -y "/config/data/transcodes/c22dfb1580de3e13a974967922d10a39.m3u8"


ffmpeg version 5.1.4-Jellyfin Copyright (c) 2000-2023 the FFmpeg developers
  built with gcc 11 (Ubuntu 11.4.0-1ubuntu1~22.04)
  configuration: --prefix=/usr/lib/jellyfin-ffmpeg --target-os=linux --extra-libs=-lfftw3f --extra-version=Jellyfin --disable-doc --disable-ffplay --disable-ptx-compression --disable-static --disable-libxcb --disable-sdl2 --disable-xlib --enable-lto --enable-gpl --enable-version3 --enable-shared --enable-gmp --enable-gnutls --enable-chromaprint --enable-libdrm --enable-libass --enable-libfreetype --enable-libfribidi --enable-libfontconfig --enable-libbluray --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libopenmpt --enable-libdav1d --enable-libwebp --enable-libvpx --enable-libx264 --enable-libx265 --enable-libzvbi --enable-libzimg --enable-libfdk-aac --arch=amd64 --enable-libsvtav1 --enable-libshaderc --enable-libplacebo --enable-vulkan --enable-opencl --enable-vaapi --enable-amf --enable-libmfx --enable-ffnvcodec --enable-cuda --enable-cuda-llvm --enable-cuvid --enable-nvdec --enable-nvenc
  libavutil      57. 28.100 / 57. 28.100
  libavcodec    59. 37.100 / 59. 37.100
  libavformat    59. 27.100 / 59. 27.100
  libavdevice    59.  7.100 / 59.  7.100
  libavfilter    8. 44.100 /  8. 44.100
  libswscale      6.  7.100 /  6.  7.100
  libswresample  4.  7.100 /  4.  7.100
  libpostproc    56.  6.100 / 56.  6.100
[AVHWDeviceContext @ 0x561ea12e6980] libva: /usr/lib/x86_64-linux-gnu/dri/d3d12_drv_video.so init failed
[AVHWDeviceContext @ 0x561ea12e6980] Failed to initialise VAAPI connection: 2 (resource allocation failed).
Device creation failed: -5.
Failed to set value 'vaapi=va:/dev/dri/card0' for option 'init_hw_device': Input/output error
Error parsing global options: Input/output error

My vainfo output in the container is as follows:

Code:

$ docker exec -it -e PUID=1000 -e PGID=1000 jellyfin /bin/sh
# ./usr/share/jellyfin-ffmpeg/vainfo --display drm --device /dev/dri/card0
Trying display: drm
libva info: VA-API version 1.20.0
libva info: Trying to open /usr/lib/jellyfin-ffmpeg/lib/dri/d3d12_drv_video.so
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/d3d12_drv_video.so
libva info: Found init function __vaDriverInit_1_14
libva error: /usr/lib/x86_64-linux-gnu/dri/d3d12_drv_video.so init failed
libva info: va_openDriver() returns 2
vaInitialize failed with error code 2 (resource allocation failed),exit

At this point, I am not quite sure what I am doing wrong. I have a feeling like maybe d3d12 driver is not supported with Jellyfin, since the driver is not available under /usr/lib/jellyfin-ffmpeg/lib/dri and it falls back to /usr/lib/x86_64-linux-gnu/dri.

How can we solve this issue? My real intention is using Jellyfin with CasaOS under WSL2. I guess it would help lots of people who is in the same situation with me. Thanks.

[Serge K]

try non docker Jellyfin server
For me, I don't like the Docker version because it has very little room for experimentation. The feeling of hands being tied))))

[SessionID]

try non docker Jellyfin server
For me, I don't like the Docker version because it has very little room for experimentation. The feeling of hands being tied))))

It is definitely a great solution for vanilla Jellyfin users, and I would go this way if only I don't use CasaOS. But CasaOS seems kinda dependent to Docker instances, which leaves me with no option other than making it work

[Serge K]

read this may be help

https://github.com/pdulvp/jellyfin-qnap

[chlorophycae]

services:
jellyfin:
privileged: true

[dzmbbs]

Hi, I turned on Hardware Acceleration with VAAPI on WSL2 Docker and can convert hevc to h264.

Device: Windows11 + Docker in WSL2 + Jellyfin.

---

First modify docker-compose.yml.

1. Add /usr/lib/wsl to volumes.
2. Add /dev/dxg to devices.
3. Add environment.


services:
  jellyfin:
    container_name: jellyfin
    image: jellyfin/jellyfin
    restart: always
    network_mode: host
    volumes:
      - ./config:/config
      - ./media:/media
      - /usr/lib/wsl:/usr/lib/wsl
    devices:
      - /dev/dxg:/dev/dxg
      - /dev/dri/card0:/dev/dri/card0
      - /dev/dri/renderD128:/dev/dri/renderD128
    environment:
      - LIBVA_DRIVER_NAME=d3d12
      - LD_LIBRARY_PATH=/usr/lib/wsl/lib




Then install dependencies in the jellyfin container.


apt update && apt install -y vainfo mesa-va-drivers mesa-vulkan-drivers vulkan-tools



Finally restart the container.


docker compose restart


Refer to: https://github.com/microsoft/wslg/blob/m...ated-video and https://jellyfin.org/docs/general/admini...ualization

[nyanmisaka]

Run hardware acceleration natively on Windows, instead of using WSL2.

HWA on WSL2 is not supported by us, nor is it officially supported by Intel (QuickSync) or AMD (AMF).

NVIDIA does claim full CUDA + NV{ENC/DEC} support for WSL2, but you're on your own.