SOLVED: Massive artifacts in transcoded videos with VA-API hardware acceleration

[TheDreadPirate]

No idea. The 4650U and G are both Renoir. Supposedly the same CPU and GPU architecture. Supposedly the same codec capabilities.

Even the 5600G, Cezanne, is grouped together with Renoir in terms of codec capabilities. Has the same Vega 7 GPU as the 4650G.

I don't know much more about AMD transcoding beyond what we've already talked about.

[nurunet]

I swapped the 4650G for my PC's 5600G, and I haven't been able to reproduce the issue since. Very weird. I still wonder if the 4650G doesn't have a defect in its transcoding unit.

[MisterMcDuck]

I'm not sure how helpful this reply will be, but I've been testing Jellyfin 10.9 unstable and have found VAAPI transcoding and tone-mapping to be working very well. I've played H264/H265 with and without HDR/DV in 1080/4K. I too had issues with 10.8, but so far 10.9 looks good. Here are all my settings:



and my docker-compose:

Code:

services:
  jellyfin:
    image: jellyfin:AnimeUnstable
    container_name: jellyfinTesting
    # render/video group
    group_add:
      - "110"
      - "44"
    volumes:
      - ./config:/config
      - /fast/transcodes/:/config/transcodes
#snipped

    devices:
      - /dev/dri/renderD128:/dev/dri/renderD128
      - /dev/dri/card0:/dev/dri/card0
    ports:
      - "0.0.0.0:8097:8096"
    restart: 'unless-stopped'
    deploy:
      resources:
        limits:
          cpus: '24'
          memory: 16384M

And here's an example truncated ffmpeg log for comparison with a 4K remux w/ HDR:

Code:

/usr/lib/jellyfin-ffmpeg/ffmpeg -analyzeduration 200M -probesize 1G -ss 00:07:51.000 -noaccurate_seek -init_hw_device drm=dr:/dev/dri/renderD128 -init_hw_device vaapi=va@dr -init_hw_device vulkan=vk@dr -filter_hw_device vk -hwaccel vaapi -hwaccel_output_format vaapi -noautorotate -i file:"/RippedMovies/Dune Part Two (2024)/Dune Part Two (2024) Remux-2160p.mkv" -noautoscale -map_metadata -1 -map_chapters -1 -threads 0 -map 0:0 -map 0:1 -map -0:s -codec:v:0 h264_vaapi -rc_mode VBR -b:v 92507430 -maxrate 92507430 -bufsize 185014860 -sei -a53_cc -force_key_frames:0 "expr:gte(t,n_forced*3)" -vf "setparams=color_primaries=bt2020:color_trc=smpte2084:colorspace=bt2020nc,hwmap=derive_device=vulkan,format=vulkan,libplacebo=upscaler=none:downscaler=none:format=bgra:tonemapping=bt.2390:peak_detect=0:color_primaries=bt709:color_trc=bt709:colorspace=bt709,hwmap=derive_device=vaapi,format=vaapi,scale_vaapi=format=nv12" -codec:a:0 libfdk_aac -ac 2 -ab 256000 -copyts -avoid_negative_ts disabled -max_muxing_queue_size 2048 -f hls -max_delay 5000000 -hls_time 3 -hls_segment_type fmp4 -hls_fmp4_init_filename "6b3a0017437be6b9bbd88ac0b1a92cea-1.mp4" -start_number 157 -hls_segment_filename "/config/transcodes/6b3a0017437be6b9bbd88ac0b1a92cea%d.mp4" -hls_playlist_type vod -hls_list_size 0 -y "/config/transcodes/6b3a0017437be6b9bbd88ac0b1a92cea.m3u8"


ffmpeg version 6.0.1-Jellyfin Copyright (c) 2000-2023 the FFmpeg developers
  built with gcc 12 (Debian 12.2.0-14)
  configuration: --prefix=/usr/lib/jellyfin-ffmpeg --target-os=linux --extra-version=Jellyfin --disable-doc --disable-ffplay --disable-ptx-compression --disable-static --disable-libxcb --disable-sdl2 --disable-xlib --enable-lto --enable-gpl --enable-version3 --enable-shared --enable-gmp --enable-gnutls --enable-chromaprint --enable-opencl --enable-libdrm --enable-libass --enable-libfreetype --enable-libfribidi --enable-libfontconfig --enable-libbluray --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libopenmpt --enable-libdav1d --enable-libsvtav1 --enable-libwebp --enable-libvpx --enable-libx264 --enable-libx265 --enable-libzvbi --enable-libzimg --enable-libfdk-aac --arch=amd64 --enable-libshaderc --enable-libplacebo --enable-vulkan --enable-vaapi --enable-amf --enable-libvpl --enable-ffnvcodec --enable-cuda --enable-cuda-llvm --enable-cuvid --enable-nvdec --enable-nvenc
  libavutil      58.  2.100 / 58.  2.100
  libavcodec    60.  3.100 / 60.  3.100
  libavformat    60.  3.100 / 60.  3.100
  libavdevice    60.  1.100 / 60.  1.100
  libavfilter    9.  3.100 /  9.  3.100
  libswscale      7.  1.100 /  7.  1.100
  libswresample  4. 10.100 /  4. 10.100
  libpostproc    57.  1.100 / 57.  1.100
amdgpu: os_same_file_description couldn't determine if two DRM fds reference the same file description.
If they do, bad things may happen!
Input #0, matroska,webm, from 'file:/NewMovies/Dune Part Two (2024)/Dune Part Two (2024) Remux-2160p.mkv':
  Metadata:
    title          : Dune: Part Two (2024) 4K
    encoder        : libebml v1.4.5 + libmatroska v1.7.1
    creation_time  : 2024-05-07T03:27:12.000000Z
    IMDB            : tt15239678
    TMDB            : movie/693134
  Duration: 02:45:49.02, start: 0.000000, bitrate: 55504 kb/s
  Chapters:
    Chapter #0:0: start 0.000000, end 554.220000
      Metadata:
        title          : Chapter 1
    Chapter #0:1: start 554.220000, end 1288.913000
      Metadata:
        title          : Chapter 2
    Chapter #0:2: start 1288.913000, end 1770.018000
      Metadata:
        title          : Chapter 3
    Chapter #0:3: start 1770.018000, end 2407.905000
      Metadata:
        title          : Chapter 4
    Chapter #0:4: start 2407.905000, end 3039.912000
      Metadata:
        title          : Chapter 5
    Chapter #0:5: start 3039.912000, end 3633.129000
      Metadata:
        title          : Chapter 6
    Chapter #0:6: start 3633.129000, end 4193.523000
      Metadata:
        title          : Chapter 7
    Chapter #0:7: start 4193.523000, end 4779.483000
      Metadata:
        title          : Chapter 8
    Chapter #0:8: start 4779.483000, end 5438.933000
      Metadata:
        title          : Chapter 9
    Chapter #0:9: start 5438.933000, end 5945.064000
      Metadata:
        title          : Chapter 10
    Chapter #0:10: start 5945.064000, end 6574.860000
      Metadata:
        title          : Chapter 11
    Chapter #0:11: start 6574.860000, end 7179.422000
      Metadata:
        title          : Chapter 12
    Chapter #0:12: start 7179.422000, end 7823.607000
      Metadata:
        title          : Chapter 13
    Chapter #0:13: start 7823.607000, end 8329.279000
      Metadata:
        title          : Chapter 14
    Chapter #0:14: start 8329.279000, end 8658.233000
      Metadata:
        title          : Chapter 15
    Chapter #0:15: start 8658.233000, end 9464.789000
      Metadata:
        title          : Chapter 16
    Chapter #0:16: start 9464.789000, end 9949.023000
      Metadata:
        title          : Chapter 17
  Stream #0:0: Video: hevc (Main 10), yuv420p10le(tv, bt2020nc/bt2020/smpte2084), 3840x2160 [SAR 1:1 DAR 16:9], 23.98 fps, 23.98 tbr, 1k tbn (default)
    Metadata:
      BPS            : 51278574
      DURATION        : 02:45:49.023000000
      NUMBER_OF_FRAMES: 238538
      NUMBER_OF_BYTES : 63771464649
      SOURCE_ID      : 001011
      _STATISTICS_WRITING_APP: mkvmerge v84.0 ('Sleeper') 64-bit
      _STATISTICS_WRITING_DATE_UTC: 2024-05-07 03:27:12
      _STATISTICS_TAGS: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES SOURCE_ID
    Side data:
      DOVI configuration record: version: 1.0, profile: 7, level: 6, rpu flag: 1, el flag: 1, bl flag: 1, compatibility id: 6
  Stream #0:1(eng): Audio: truehd, 48000 Hz, 7.1, s32 (24 bit) (default)
    Metadata:
      title          : TrueHD Atmos 7.1
      BPS            : 3257429
      DURATION        : 02:45:48.982000000
      NUMBER_OF_FRAMES: 11938778
      NUMBER_OF_BYTES : 4051013168
      _STATISTICS_WRITING_APP: mkvmerge v84.0 ('Sleeper') 64-bit
      _STATISTICS_WRITING_DATE_UTC: 2024-05-07 03:27:12
      _STATISTICS_TAGS: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
  Stream #0:2(eng): Audio: eac3, 48000 Hz, 5.1(side), fltp, 768 kb/s
    Metadata:
      title          : E-AC-3 Atmos 5.1
      BPS            : 768000
      DURATION        : 02:45:47.008000000
      NUMBER_OF_FRAMES: 310844
      NUMBER_OF_BYTES : 954912768
      _STATISTICS_WRITING_APP: mkvmerge v84.0 ('Sleeper') 64-bit
      _STATISTICS_WRITING_DATE_UTC: 2024-05-07 03:27:12
      _STATISTICS_TAGS: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
  Stream #0:3(eng): Subtitle: subrip
    Metadata:
      title          : English
      BPS            : 27
      DURATION        : 02:35:52.850000000
      NUMBER_OF_FRAMES: 1250
      NUMBER_OF_BYTES : 32717
      _STATISTICS_WRITING_APP: mkvmerge v84.0 ('Sleeper') 64-bit
      _STATISTICS_WRITING_DATE_UTC: 2024-05-07 03:27:12
      _STATISTICS_TAGS: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
*SNIP*
Stream mapping:
  Stream #0:0 -> #0:0 (hevc (native) -> h264 (h264_vaapi))
  Stream #0:1 -> #0:1 (truehd (native) -> aac (libfdk_aac))
Press [q] to stop, [?] for help
[hevc @ 0x59838ad566c0] Skipping NAL unit 63
    Last message repeated 17 times

[h264_vaapi @ 0x59838b019980] Driver does not support some wanted packed headers (wanted 0xd, found 0x1).
[hls @ 0x59838ad43d40] Opening '/config/transcodes/6b3a0017437be6b9bbd88ac0b1a92cea-1.mp4' for writing
Output #0, hls, to '/config/transcodes/6b3a0017437be6b9bbd88ac0b1a92cea.m3u8':
  Metadata:
    encoder        : Lavf60.3.100
  Stream #0:0: Video: h264 (High), vaapi(tv, bt709, progressive), 3840x2160 [SAR 1:1 DAR 16:9], q=2-31, 92507 kb/s, 23.98 fps, 24k tbn (default)
    Metadata:
      encoder        : Lavc60.3.100 h264_vaapi
    Side data:
      DOVI configuration record: version: 1.0, profile: 7, level: 6, rpu flag: 1, el flag: 1, bl flag: 1, compatibility id: 6
  Stream #0:1: Audio: aac, 48000 Hz, stereo, s16, 256 kb/s (default)
    Metadata:
      encoder        : Lavc60.3.100 libfdk_aac
frame=    0 fps=0.0 q=0.0 size=N/A time=00:00:00.00 bitrate=N/A speed=  0x   
[hevc @ 0x59838b052e80] Skipping NAL unit 63
    Last message repeated 10 times

frame=  13 fps=0.0 q=-0.0 size=N/A time=00:00:00.74 bitrate=N/A speed=1.04x   

frame=  33 fps= 27 q=-0.0 size=N/A time=00:00:01.62 bitrate=N/A speed= 1.3x   

frame=  53 fps= 30 q=-0.0 size=N/A time=00:00:02.38 bitrate=N/A speed=1.36x   

[hls @ 0x59838ad43d40] Opening '/config/transcodes/6b3a0017437be6b9bbd88ac0b1a92cea157.mp4' for writing
...

frame= 3746 fps= 37 q=-0.0 Lsize=N/A time=00:02:36.28 bitrate=N/A speed=1.55x   
video:1698430kB audio:4903kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown

I have an AMD 5700XT, which might be part of the reason it's working better for me, since it's a newer card.

[nurunet]

I swapped the 4650G for my PC's 5600G, and I haven't been able to reproduce the issue since. Very weird. I still wonder if the 4650G doesn't have a defect in its transcoding unit.

Well... I very briefly tested it on my desktop under Windows, using the 4650G, and wasn't able to reproduce it there either. What gives?

---

I'm not sure how helpful this reply will be, but I've been testing Jellyfin 10.9 unstable and have found VAAPI transcoding and tone-mapping to be working very well. I've played H264/H265 with and without HDR/DV in 1080/4K. I too had issues with 10.8, but so far 10.9 looks good. Here are all my settings: (...)

Awesome, thanks. Two questions:

1. Why did you disable HW decoding for anything except AV1? I would have thought modern GPUs should be able to handle all those codecs.
   
2. Can VPP work without Intel hardware?
   

[Efficient_Good_5784]

1. Why did you disable HW decoding for anything except AV1? I would have thought modern GPUs should be able to handle all those codecs.
   

Are you getting confused? I'm looking at the picture that @MisterMcDuck shared of his HWA decode settings and he has different settings.

The options he has enabled:

1. H264
   
2. HEVC
   
3. HEVC 10-bit
   
4. VC1
   
5. VP9 10-bit
   

The options he has disabled:

1. MPEG2
   
2. VP8
   
3. VP9
   
4. AV1
   

[MisterMcDuck]

---

I'm not sure how helpful this reply will be, but I've been testing Jellyfin 10.9 unstable and have found VAAPI transcoding and tone-mapping to be working very well. I've played H264/H265 with and without HDR/DV in 1080/4K. I too had issues with 10.8, but so far 10.9 looks good. Here are all my settings: (...)

Awesome, thanks. Two questions:

1. Why did you disable HW decoding for anything except AV1? I would have thought modern GPUs should be able to handle all those codecs.
   
2. Can VPP work without Intel hardware?
   

1. As Efficient_Good_5784 said,

The options he has enabled:

1. H264
   
   
2. HEVC
   
   
3. HEVC 10-bit
   
   
4. VC1
   
   
5. VP9 10-bit
   
   

The options he has disabled:

1. MPEG2
   
   
2. VP8
   
   
3. VP9
   
   
4. AV1
   

The disabled and enabled in this section I think were mostly the defaults. I don't recall explicitly disabling anything.

2.
I do not believe Jellyfin uses VPP on my system. I presume it checks and falls back on VAAPI for my AMD GPU.

[nurunet]

1. Why did you disable HW decoding for anything except AV1? I would have thought modern GPUs should be able to handle all those codecs.
   

Are you getting confused? I'm looking at the picture that @MisterMcDuck shared of his HWA decode settings and he has different settings.

The options he has enabled:

1. H264
   
2. HEVC
   
3. HEVC 10-bit
   
4. VC1
   
5. VP9 10-bit
   

The options he has disabled:

1. MPEG2
   
2. VP8
   
3. VP9
   
4. AV1
   

Sorry, not a native speaker. I meant to ask why AV1 was not the sole "deselected" codec - as I believed the hardware should support all others?

[Efficient_Good_5784]

@MisterMcDuck states that he mainly stuck with the default settings, so he didn't mess with the settings that were given to him.

Just in case you don't know, just because certain hardware is more modern (or recent) doesn't mean that it supports a lot more things. Newer hardware may drop support for older things. In this case with HWA support on a GPU, it sometimes is impossible for a company to build a GPU with physical support for all the new and old video codecs, so they make decisions on when to add or drop support for them. It's not a software thing, it's a hardware thing. There's literal traces on the GPU boards that provide the HWA ability for certain codecs. I'm not sure of any exact reasons (and there could be multiple), but I would imagine space constraints would be one reason why GPU companies will have to give up on certain codecs and add others.

In this case, MisterMcDuck has an AMD 5700XT, which uses the Video Core Next (VCN) 2.0 HWA encoding implementation.

If you look at the Wiki page for VCN, you'll see a paragraph detailing what VCN supports as a baseline for HWA encoding and decoding along with a table that further shows what 2.0 specifically supports apart from the baseline.

Doing this work for MisterMcDuck, the decode settings that he should enable and disable are as follows.

Enable:

1. H264
   
2. HEVC
   
3. MPEG2
   
4. VC1
   
5. VP9
   
6. HEVC 10bit
   
7. VP9 10bit
   

Disable:

1. VP8
   
2. AV1
   

If you're curious about this and have an Intel GPU/iGPU, you can figure this out too by looking at Intel's Quick Sync Wiki page for HWA encode & decode support.