SOLVED: Massive artifacts in transcoded videos with VA-API hardware acceleration

[nurunet]

[Multiple messages because of long logs...]

System:
Asrock Deskmini X300, BIOS 1.80E
Ryzen Pro 4650G
32 GB DDR4 RAM
Running OMV 6

Hi, am running Jellyfin 10.8.13 on Docker. The host system is an Openmediavault 6 install on a Deskmini X300 w/ Ryzen 5 APU.

I follwed the install instructions here: https://jellyfin.org/docs/general/admini...ualization. Output looks good, I think?

Code:

root@omv1:/dev/dri# docker exec -it jellyfin /usr/lib/jellyfin-ffmpeg/vainfo --display drm --device /dev/dri/renderD128
Trying display: drm
libva info: VA-API version 1.20.0
libva info: Trying to open /usr/lib/jellyfin-ffmpeg/lib/dri/radeonsi_drv_video.so
libva info: Found init function __vaDriverInit_1_20
amdgpu: os_same_file_description couldn't determine if two DRM fds reference the same file description.
If they do, bad things may happen!
libva info: va_openDriver() returns 0
vainfo: VA-API version: 1.20 (libva 2.20.0)
vainfo: Driver version: Mesa Gallium driver 23.2.1 for AMD Radeon Graphics (renoir, LLVM 13.0.1, DRM 3.49, 6.1.0-0.deb11.11-amd64)
vainfo: Supported profile and entrypoints
      VAProfileMPEG2Simple            : VAEntrypointVLD
      VAProfileMPEG2Main              : VAEntrypointVLD
      VAProfileVC1Simple              : VAEntrypointVLD
      VAProfileVC1Main                : VAEntrypointVLD
      VAProfileVC1Advanced            : VAEntrypointVLD
      VAProfileH264ConstrainedBaseline: VAEntrypointVLD
      VAProfileH264ConstrainedBaseline: VAEntrypointEncSlice
      VAProfileH264Main              : VAEntrypointVLD
      VAProfileH264Main              : VAEntrypointEncSlice
      VAProfileH264High              : VAEntrypointVLD
      VAProfileH264High              : VAEntrypointEncSlice
      VAProfileHEVCMain              : VAEntrypointVLD
      VAProfileHEVCMain              : VAEntrypointEncSlice
      VAProfileHEVCMain10            : VAEntrypointVLD
      VAProfileHEVCMain10            : VAEntrypointEncSlice
      VAProfileJPEGBaseline          : VAEntrypointVLD
      VAProfileVP9Profile0            : VAEntrypointVLD
      VAProfileVP9Profile2            : VAEntrypointVLD
      VAProfileNone                  : VAEntrypointVideoProc

Code:

root@omv1:/dev/dri# docker exec -it jellyfin /usr/lib/jellyfin-ffmpeg/ffmpeg -v debug -init_hw_device opencl
ffmpeg version 5.1.4-Jellyfin Copyright (c) 2000-2023 the FFmpeg developers
  built with gcc 10 (Debian 10.2.1-6)
  configuration: --prefix=/usr/lib/jellyfin-ffmpeg --target-os=linux --extra-libs=-lfftw3f --extra-version=Jellyfin --disable-doc --disable-ffplay --disable-ptx-compression --disable-static --disable-libxcb --disable-sdl2 --disable-xlib --enable-lto --enable-gpl --enable-version3 --enable-shared --enable-gmp --enable-gnutls --enable-chromaprint --enable-libdrm --enable-libass --enable-libfreetype --enable-libfribidi --enable-libfontconfig --enable-libbluray --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libopenmpt --enable-libdav1d --enable-libwebp --enable-libvpx --enable-libx264 --enable-libx265 --enable-libzvbi --enable-libzimg --enable-libfdk-aac --arch=amd64 --enable-libsvtav1 --enable-libshaderc --enable-libplacebo --enable-vulkan --enable-opencl --enable-vaapi --enable-amf --enable-libmfx --enable-ffnvcodec --enable-cuda --enable-cuda-llvm --enable-cuvid --enable-nvdec --enable-nvenc
  libavutil      57. 28.100 / 57. 28.100
  libavcodec    59. 37.100 / 59. 37.100
  libavformat    59. 27.100 / 59. 27.100
  libavdevice    59.  7.100 / 59.  7.100
  libavfilter    8. 44.100 /  8. 44.100
  libswscale      6.  7.100 /  6.  7.100
  libswresample  4.  7.100 /  4.  7.100
  libpostproc    56.  6.100 / 56.  6.100
Splitting the commandline.
Reading option '-v' ... matched as option 'v' (set logging level) with argument 'debug'.
Reading option '-init_hw_device' ... matched as option 'init_hw_device' (initialise hardware device) with argument 'opencl'.
Finished splitting the commandline.
Parsing a group of options: global .
Applying option v (set logging level) with argument debug.
Applying option init_hw_device (initialise hardware device) with argument opencl.
[AVHWDeviceContext @ 0x55ebeecd3180] 1 OpenCL platforms found.
[AVHWDeviceContext @ 0x55ebeecd3180] 1 OpenCL devices found on platform "AMD Accelerated Parallel Processing".
[AVHWDeviceContext @ 0x55ebeecd3180] 0.0: AMD Accelerated Parallel Processing / gfx90c:xnack-
[AVHWDeviceContext @ 0x55ebeecd3180] The cl_intel_va_api_media_sharing extension is required for QSV to OpenCL mapping.
[AVHWDeviceContext @ 0x55ebeecd3180] QSV to OpenCL mapping not usable.
Successfully parsed a group of options.
Hyper fast Audio and Video encoder
usage: ffmpeg [options] [[infile options] -i infile]... {[outfile options] outfile}...

Use -h to get full help or, even better, run 'man ffmpeg'

Code:

root@omv1:/dev/dri# docker exec -it jellyfin /usr/lib/jellyfin-ffmpeg/ffmpeg -v debug -init_hw_device vulkan
ffmpeg version 5.1.4-Jellyfin Copyright (c) 2000-2023 the FFmpeg developers
  built with gcc 10 (Debian 10.2.1-6)
  configuration: --prefix=/usr/lib/jellyfin-ffmpeg --target-os=linux --extra-libs=-lfftw3f --extra-version=Jellyfin --disable-doc --disable-ffplay --disable-ptx-compression --disable-static --disable-libxcb --disable-sdl2 --disable-xlib --enable-lto --enable-gpl --enable-version3 --enable-shared --enable-gmp --enable-gnutls --enable-chromaprint --enable-libdrm --enable-libass --enable-libfreetype --enable-libfribidi --enable-libfontconfig --enable-libbluray --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libopenmpt --enable-libdav1d --enable-libwebp --enable-libvpx --enable-libx264 --enable-libx265 --enable-libzvbi --enable-libzimg --enable-libfdk-aac --arch=amd64 --enable-libsvtav1 --enable-libshaderc --enable-libplacebo --enable-vulkan --enable-opencl --enable-vaapi --enable-amf --enable-libmfx --enable-ffnvcodec --enable-cuda --enable-cuda-llvm --enable-cuvid --enable-nvdec --enable-nvenc
  libavutil      57. 28.100 / 57. 28.100
  libavcodec    59. 37.100 / 59. 37.100
  libavformat    59. 27.100 / 59. 27.100
  libavdevice    59.  7.100 / 59.  7.100
  libavfilter    8. 44.100 /  8. 44.100
  libswscale      6.  7.100 /  6.  7.100
  libswresample  4.  7.100 /  4.  7.100
  libpostproc    56.  6.100 / 56.  6.100
Splitting the commandline.
Reading option '-v' ... matched as option 'v' (set logging level) with argument 'debug'.
Reading option '-init_hw_device' ... matched as option 'init_hw_device' (initialise hardware device) with argument 'vulkan'.
Finished splitting the commandline.
Parsing a group of options: global .
Applying option v (set logging level) with argument debug.
Applying option init_hw_device (initialise hardware device) with argument vulkan.
[AVHWDeviceContext @ 0x562aeab5e180] Supported validation layers:
[AVHWDeviceContext @ 0x562aeab5e180]    VK_LAYER_MESA_device_select
[AVHWDeviceContext @ 0x562aeab5e180]    VK_LAYER_MESA_overlay
[AVHWDeviceContext @ 0x562aeab5e180] GPU listing:
[AVHWDeviceContext @ 0x562aeab5e180]    0: AMD Radeon Graphics (RADV RENOIR) (integrated) (0x1636)
[AVHWDeviceContext @ 0x562aeab5e180] Device 0 selected: AMD Radeon Graphics (RADV RENOIR) (integrated) (0x1636)
[AVHWDeviceContext @ 0x562aeab5e180] Queue families:
[AVHWDeviceContext @ 0x562aeab5e180]    0: graphics compute transfer sparse (queues: 1)
[AVHWDeviceContext @ 0x562aeab5e180]    1: compute transfer sparse (queues: 4)
[AVHWDeviceContext @ 0x562aeab5e180] Using device extension VK_KHR_push_descriptor
[AVHWDeviceContext @ 0x562aeab5e180] Using device extension VK_KHR_sampler_ycbcr_conversion
[AVHWDeviceContext @ 0x562aeab5e180] Using device extension VK_KHR_synchronization2
[AVHWDeviceContext @ 0x562aeab5e180] Using device extension VK_KHR_external_memory_fd
[AVHWDeviceContext @ 0x562aeab5e180] Using device extension VK_EXT_external_memory_dma_buf
[AVHWDeviceContext @ 0x562aeab5e180] Using device extension VK_EXT_image_drm_format_modifier
[AVHWDeviceContext @ 0x562aeab5e180] Using device extension VK_KHR_external_semaphore_fd
[AVHWDeviceContext @ 0x562aeab5e180] Using device extension VK_EXT_external_memory_host
[AVHWDeviceContext @ 0x562aeab5e180] Using device: AMD Radeon Graphics (RADV RENOIR)
[AVHWDeviceContext @ 0x562aeab5e180] Alignments:
[AVHWDeviceContext @ 0x562aeab5e180]    optimalBufferCopyRowPitchAlignment: 1
[AVHWDeviceContext @ 0x562aeab5e180]    minMemoryMapAlignment:              4096
[AVHWDeviceContext @ 0x562aeab5e180]    minImportedHostPointerAlignment:    4096
[AVHWDeviceContext @ 0x562aeab5e180] Using queue family 0 (queues: 1) for graphics
[AVHWDeviceContext @ 0x562aeab5e180] Using queue family 1 (queues: 4) for compute transfers
Successfully parsed a group of options.
Hyper fast Audio and Video encoder
usage: ffmpeg [options] [[infile options] -i infile]... {[outfile options] outfile}...

Use -h to get full help or, even better, run 'man ffmpeg'

Now I set up playback on Jellyfin's root account for VAAPI hardware transcode (all codecs except AV1). Unfortunately, when playing back (tested on an LG OLED w/ latest WebOS app and a Pixel 7a w/ the Android app), I encounter massive issues:

1. The first transcode was done in H265, I think, and briefly froze every second or so
   
2. I fiddled with some settings, e.g. "Allow encoding in HEVC format", and further encodes were in H264 (even after enabling the HEVC setting again?). Now, in addition to more irregular freezes, the image showed varying degrees of blocky artifacts, which sometimes took over most of the image.
   
3. When disabling hardware transcoding, the image looks fine but CPU load increases, so it is not an ideal solution
   
4. Switching to AMF instead of VAAPI just yields an error message when trying to play video (Playback Error - This client isn't compatible with the media and the server isn't sending a compatible media format)