SOLVED: Massive artifacts in transcoded videos with VA-API hardware acceleration

[MisterMcDuck]

I'm not sure how helpful this reply will be, but I've been testing Jellyfin 10.9 unstable and have found VAAPI transcoding and tone-mapping to be working very well. I've played H264/H265 with and without HDR/DV in 1080/4K. I too had issues with 10.8, but so far 10.9 looks good. Here are all my settings:



and my docker-compose:

Code:

services:
  jellyfin:
    image: jellyfin:AnimeUnstable
    container_name: jellyfinTesting
    # render/video group
    group_add:
      - "110"
      - "44"
    volumes:
      - ./config:/config
      - /fast/transcodes/:/config/transcodes
#snipped

    devices:
      - /dev/dri/renderD128:/dev/dri/renderD128
      - /dev/dri/card0:/dev/dri/card0
    ports:
      - "0.0.0.0:8097:8096"
    restart: 'unless-stopped'
    deploy:
      resources:
        limits:
          cpus: '24'
          memory: 16384M

And here's an example truncated ffmpeg log for comparison with a 4K remux w/ HDR:

Code:

/usr/lib/jellyfin-ffmpeg/ffmpeg -analyzeduration 200M -probesize 1G -ss 00:07:51.000 -noaccurate_seek -init_hw_device drm=dr:/dev/dri/renderD128 -init_hw_device vaapi=va@dr -init_hw_device vulkan=vk@dr -filter_hw_device vk -hwaccel vaapi -hwaccel_output_format vaapi -noautorotate -i file:"/RippedMovies/Dune Part Two (2024)/Dune Part Two (2024) Remux-2160p.mkv" -noautoscale -map_metadata -1 -map_chapters -1 -threads 0 -map 0:0 -map 0:1 -map -0:s -codec:v:0 h264_vaapi -rc_mode VBR -b:v 92507430 -maxrate 92507430 -bufsize 185014860 -sei -a53_cc -force_key_frames:0 "expr:gte(t,n_forced*3)" -vf "setparams=color_primaries=bt2020:color_trc=smpte2084:colorspace=bt2020nc,hwmap=derive_device=vulkan,format=vulkan,libplacebo=upscaler=none:downscaler=none:format=bgra:tonemapping=bt.2390:peak_detect=0:color_primaries=bt709:color_trc=bt709:colorspace=bt709,hwmap=derive_device=vaapi,format=vaapi,scale_vaapi=format=nv12" -codec:a:0 libfdk_aac -ac 2 -ab 256000 -copyts -avoid_negative_ts disabled -max_muxing_queue_size 2048 -f hls -max_delay 5000000 -hls_time 3 -hls_segment_type fmp4 -hls_fmp4_init_filename "6b3a0017437be6b9bbd88ac0b1a92cea-1.mp4" -start_number 157 -hls_segment_filename "/config/transcodes/6b3a0017437be6b9bbd88ac0b1a92cea%d.mp4" -hls_playlist_type vod -hls_list_size 0 -y "/config/transcodes/6b3a0017437be6b9bbd88ac0b1a92cea.m3u8"


ffmpeg version 6.0.1-Jellyfin Copyright (c) 2000-2023 the FFmpeg developers
  built with gcc 12 (Debian 12.2.0-14)
  configuration: --prefix=/usr/lib/jellyfin-ffmpeg --target-os=linux --extra-version=Jellyfin --disable-doc --disable-ffplay --disable-ptx-compression --disable-static --disable-libxcb --disable-sdl2 --disable-xlib --enable-lto --enable-gpl --enable-version3 --enable-shared --enable-gmp --enable-gnutls --enable-chromaprint --enable-opencl --enable-libdrm --enable-libass --enable-libfreetype --enable-libfribidi --enable-libfontconfig --enable-libbluray --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libopenmpt --enable-libdav1d --enable-libsvtav1 --enable-libwebp --enable-libvpx --enable-libx264 --enable-libx265 --enable-libzvbi --enable-libzimg --enable-libfdk-aac --arch=amd64 --enable-libshaderc --enable-libplacebo --enable-vulkan --enable-vaapi --enable-amf --enable-libvpl --enable-ffnvcodec --enable-cuda --enable-cuda-llvm --enable-cuvid --enable-nvdec --enable-nvenc
  libavutil      58.  2.100 / 58.  2.100
  libavcodec    60.  3.100 / 60.  3.100
  libavformat    60.  3.100 / 60.  3.100
  libavdevice    60.  1.100 / 60.  1.100
  libavfilter    9.  3.100 /  9.  3.100
  libswscale      7.  1.100 /  7.  1.100
  libswresample  4. 10.100 /  4. 10.100
  libpostproc    57.  1.100 / 57.  1.100
amdgpu: os_same_file_description couldn't determine if two DRM fds reference the same file description.
If they do, bad things may happen!
Input #0, matroska,webm, from 'file:/NewMovies/Dune Part Two (2024)/Dune Part Two (2024) Remux-2160p.mkv':
  Metadata:
    title          : Dune: Part Two (2024) 4K
    encoder        : libebml v1.4.5 + libmatroska v1.7.1
    creation_time  : 2024-05-07T03:27:12.000000Z
    IMDB            : tt15239678
    TMDB            : movie/693134
  Duration: 02:45:49.02, start: 0.000000, bitrate: 55504 kb/s
  Chapters:
    Chapter #0:0: start 0.000000, end 554.220000
      Metadata:
        title          : Chapter 1
    Chapter #0:1: start 554.220000, end 1288.913000
      Metadata:
        title          : Chapter 2
    Chapter #0:2: start 1288.913000, end 1770.018000
      Metadata:
        title          : Chapter 3
    Chapter #0:3: start 1770.018000, end 2407.905000
      Metadata:
        title          : Chapter 4
    Chapter #0:4: start 2407.905000, end 3039.912000
      Metadata:
        title          : Chapter 5
    Chapter #0:5: start 3039.912000, end 3633.129000
      Metadata:
        title          : Chapter 6
    Chapter #0:6: start 3633.129000, end 4193.523000
      Metadata:
        title          : Chapter 7
    Chapter #0:7: start 4193.523000, end 4779.483000
      Metadata:
        title          : Chapter 8
    Chapter #0:8: start 4779.483000, end 5438.933000
      Metadata:
        title          : Chapter 9
    Chapter #0:9: start 5438.933000, end 5945.064000
      Metadata:
        title          : Chapter 10
    Chapter #0:10: start 5945.064000, end 6574.860000
      Metadata:
        title          : Chapter 11
    Chapter #0:11: start 6574.860000, end 7179.422000
      Metadata:
        title          : Chapter 12
    Chapter #0:12: start 7179.422000, end 7823.607000
      Metadata:
        title          : Chapter 13
    Chapter #0:13: start 7823.607000, end 8329.279000
      Metadata:
        title          : Chapter 14
    Chapter #0:14: start 8329.279000, end 8658.233000
      Metadata:
        title          : Chapter 15
    Chapter #0:15: start 8658.233000, end 9464.789000
      Metadata:
        title          : Chapter 16
    Chapter #0:16: start 9464.789000, end 9949.023000
      Metadata:
        title          : Chapter 17
  Stream #0:0: Video: hevc (Main 10), yuv420p10le(tv, bt2020nc/bt2020/smpte2084), 3840x2160 [SAR 1:1 DAR 16:9], 23.98 fps, 23.98 tbr, 1k tbn (default)
    Metadata:
      BPS            : 51278574
      DURATION        : 02:45:49.023000000
      NUMBER_OF_FRAMES: 238538
      NUMBER_OF_BYTES : 63771464649
      SOURCE_ID      : 001011
      _STATISTICS_WRITING_APP: mkvmerge v84.0 ('Sleeper') 64-bit
      _STATISTICS_WRITING_DATE_UTC: 2024-05-07 03:27:12
      _STATISTICS_TAGS: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES SOURCE_ID
    Side data:
      DOVI configuration record: version: 1.0, profile: 7, level: 6, rpu flag: 1, el flag: 1, bl flag: 1, compatibility id: 6
  Stream #0:1(eng): Audio: truehd, 48000 Hz, 7.1, s32 (24 bit) (default)
    Metadata:
      title          : TrueHD Atmos 7.1
      BPS            : 3257429
      DURATION        : 02:45:48.982000000
      NUMBER_OF_FRAMES: 11938778
      NUMBER_OF_BYTES : 4051013168
      _STATISTICS_WRITING_APP: mkvmerge v84.0 ('Sleeper') 64-bit
      _STATISTICS_WRITING_DATE_UTC: 2024-05-07 03:27:12
      _STATISTICS_TAGS: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
  Stream #0:2(eng): Audio: eac3, 48000 Hz, 5.1(side), fltp, 768 kb/s
    Metadata:
      title          : E-AC-3 Atmos 5.1
      BPS            : 768000
      DURATION        : 02:45:47.008000000
      NUMBER_OF_FRAMES: 310844
      NUMBER_OF_BYTES : 954912768
      _STATISTICS_WRITING_APP: mkvmerge v84.0 ('Sleeper') 64-bit
      _STATISTICS_WRITING_DATE_UTC: 2024-05-07 03:27:12
      _STATISTICS_TAGS: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
  Stream #0:3(eng): Subtitle: subrip
    Metadata:
      title          : English
      BPS            : 27
      DURATION        : 02:35:52.850000000
      NUMBER_OF_FRAMES: 1250
      NUMBER_OF_BYTES : 32717
      _STATISTICS_WRITING_APP: mkvmerge v84.0 ('Sleeper') 64-bit
      _STATISTICS_WRITING_DATE_UTC: 2024-05-07 03:27:12
      _STATISTICS_TAGS: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
*SNIP*
Stream mapping:
  Stream #0:0 -> #0:0 (hevc (native) -> h264 (h264_vaapi))
  Stream #0:1 -> #0:1 (truehd (native) -> aac (libfdk_aac))
Press [q] to stop, [?] for help
[hevc @ 0x59838ad566c0] Skipping NAL unit 63
    Last message repeated 17 times

[h264_vaapi @ 0x59838b019980] Driver does not support some wanted packed headers (wanted 0xd, found 0x1).
[hls @ 0x59838ad43d40] Opening '/config/transcodes/6b3a0017437be6b9bbd88ac0b1a92cea-1.mp4' for writing
Output #0, hls, to '/config/transcodes/6b3a0017437be6b9bbd88ac0b1a92cea.m3u8':
  Metadata:
    encoder        : Lavf60.3.100
  Stream #0:0: Video: h264 (High), vaapi(tv, bt709, progressive), 3840x2160 [SAR 1:1 DAR 16:9], q=2-31, 92507 kb/s, 23.98 fps, 24k tbn (default)
    Metadata:
      encoder        : Lavc60.3.100 h264_vaapi
    Side data:
      DOVI configuration record: version: 1.0, profile: 7, level: 6, rpu flag: 1, el flag: 1, bl flag: 1, compatibility id: 6
  Stream #0:1: Audio: aac, 48000 Hz, stereo, s16, 256 kb/s (default)
    Metadata:
      encoder        : Lavc60.3.100 libfdk_aac
frame=    0 fps=0.0 q=0.0 size=N/A time=00:00:00.00 bitrate=N/A speed=  0x   
[hevc @ 0x59838b052e80] Skipping NAL unit 63
    Last message repeated 10 times

frame=  13 fps=0.0 q=-0.0 size=N/A time=00:00:00.74 bitrate=N/A speed=1.04x   

frame=  33 fps= 27 q=-0.0 size=N/A time=00:00:01.62 bitrate=N/A speed= 1.3x   

frame=  53 fps= 30 q=-0.0 size=N/A time=00:00:02.38 bitrate=N/A speed=1.36x   

[hls @ 0x59838ad43d40] Opening '/config/transcodes/6b3a0017437be6b9bbd88ac0b1a92cea157.mp4' for writing
...

frame= 3746 fps= 37 q=-0.0 Lsize=N/A time=00:02:36.28 bitrate=N/A speed=1.55x   
video:1698430kB audio:4903kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown

I have an AMD 5700XT, which might be part of the reason it's working better for me, since it's a newer card.