HEVC decoding issue with AMD GPU on Ubuntu

[Aragorn]

I've been having this issue for a while, I think i'll start with a screen capture showing the problem (this is a HEVC HDR clip i grabbed from the Kodi samples repository):

https://www.youtube.com/watch?v=wRt5hfxu5Ek

As you can see, theres a weird green artifact in the top left corner, which frequently explodes out across the screen.

This issue occurs on multiple clients, and appears to be a GPU transcoding issue. It only seems to occur when dealing with HEVC content (infact possibly just HEVC 10bit/HDR), and ofcourse only appears when transcoding is involved.

Some system specs:

Ubuntu 22.04 VM (upgraded to 6.5 Kernel)
Jellyfin 10.9 unstable build (for Vulkan Tonemapping support)
AMD RX6600 GPU VFIO-passthru'd into the VM
Intel i5-12400 CPU

This issue has plagued my setup for a long time. I'm sure it existed on 10.8 too. In trying to get rid of it, i've recently upgraded to Kernel 6.5 (from 5.15), installed the latest Mesa build from the kisak repo, updated Jellyfin multiple times.

The only thing i've found which fixes it, is unticking "HEVC 10bit" support in the Hardware accellerated decoding options in the Transcode menu. Switching off Tonemapping does not change the effect, only means the colors are washed out.

I've tried to attach a transcode log, but it seems the forum doesnt accept text files, i'll stick it in a paste site and link it across.

Any idea on how we can figure out whats going on here and how we can fix it?

Thanks