HEVC decoding issue with AMD GPU on Ubuntu

[Aragorn]

I've been having this issue for a while, I think i'll start with a screen capture showing the problem (this is a HEVC HDR clip i grabbed from the Kodi samples repository):

https://www.youtube.com/watch?v=wRt5hfxu5Ek

As you can see, theres a weird green artifact in the top left corner, which frequently explodes out across the screen.

This issue occurs on multiple clients, and appears to be a GPU transcoding issue. It only seems to occur when dealing with HEVC content (infact possibly just HEVC 10bit/HDR), and ofcourse only appears when transcoding is involved.

Some system specs:

Ubuntu 22.04 VM (upgraded to 6.5 Kernel)
Jellyfin 10.9 unstable build (for Vulkan Tonemapping support)
AMD RX6600 GPU VFIO-passthru'd into the VM
Intel i5-12400 CPU

This issue has plagued my setup for a long time. I'm sure it existed on 10.8 too. In trying to get rid of it, i've recently upgraded to Kernel 6.5 (from 5.15), installed the latest Mesa build from the kisak repo, updated Jellyfin multiple times.

The only thing i've found which fixes it, is unticking "HEVC 10bit" support in the Hardware accellerated decoding options in the Transcode menu. Switching off Tonemapping does not change the effect, only means the colors are washed out.

I've tried to attach a transcode log, but it seems the forum doesnt accept text files, i'll stick it in a paste site and link it across.

Any idea on how we can figure out whats going on here and how we can fix it?

Thanks

[nyanmisaka]

What is the jellyfin-ffmpeg version? If it's not the latest, please update to jellyfin-ffmpeg6 and try again.

Can you use MPV (hwdec=vaapi) to play this video on RX 6600 to reproduce the issue?

[Aragorn]

Jellyfin-ffmpeg is at 5.1.2:

ii jellyfin-ffmpeg5 5.1.2-5-jammy amd64

Is there a reason its not pulling a newer version with a dist-upgrade?

I can ofcourse manually install a newer one and test it out, will give that a go this evening, thanks.

Not sure on the MPV part, its a headless server, but i might be able to put something together.

[Aragorn]

Okay upgraded to ffmpeg 6.0.1, symptoms are pretty much the same.

It seemed part of the issue (ffmpeg not updating) was i only had the "unstable" repo active, rather than both "main" and "unstable" which is what the documentation actually suggests for running unstable/10.9 builds.

Seems the jellyfin repository is a little bit broken now though, installing 6.0.1 forced me to remove the jellyfin metapackage, as the metapackage wants ffmpeg5 as a dependancy.

I think I'm going to swap a GTX1660 into the server to get things working for now, and setup the RX6600 on a test bench to investigate a bit more. I need to borrow the RX6600 for a week or so anyway for a different system.

[cassios]

Last windows drivers (24.1.1) was wrong on windows 11 for Jellyfin.
I had rollback to 23.12.1.
Perhaps a driver problem from AMD on Linux too?