SOLVED: AMD GPU Tone mapping issue

[arrguy]

Hi,
I've a problem with tone mapping on AMD GPU. Recently I've got my hands on used Radeon RX 5700, that I thought could do just fine for transcoding.
After initial configuration, drivers, permissions etc. I've got it working, all is fine, except tone mapping - which I believe should work, since this card should be able to handle HEVC 10 bit - according to https://www.x.org/wiki/RadeonFeature/#ra...xthardware (NAVI 10)
I couldn't find any info about my ffmpeg transcoding error, that's why I wanted to create a topic here.

Environment: Jellyfin 10.10.7 with jellyfin-ffmpeg on Ubuntu, installed by install-debuntu script - running in VM on Proxmox with GPU passthrough

For decoding I've enabled: H264, HEVC, MPEG2, VC1, VP9, HEVC 10bit, VP9 10bit
Enabled encoding for HEVC

ls -l /dev/dri

Code:

drwxr-xr-x 2 root root        80 Jul 25 16:58 by-path
crw-rw---- 1 root video  226,  1 Jul 27 21:20 card1
crw-rw---- 1 root render 226, 128 Jul 25 16:58 renderD128

id jellyfin

Code:

uid=114(jellyfin) gid=119(jellyfin) groups=119(jellyfin),44(video),109(render)

drivers installed:

Code:

amdgpu-install --opencl=rocr --vulkan=amdvlk --opengl=mesa

vainfo:

Code:

error: XDG_RUNTIME_DIR is invalid or not set in the environment.
error: can't connect to X server!
libva info: VA-API version 1.20.0
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/radeonsi_drv_video.so
libva info: Found init function __vaDriverInit_1_16
libva info: va_openDriver() returns 0
vainfo: VA-API version: 1.20 (libva 2.12.0)
vainfo: Driver version: Mesa Gallium driver 25.0.0-devel for AMD Radeon RX 5700 (radeonsi, navi10, LLVM 19.1.5, DRM 3.63, 6.8.0-64-generic)
vainfo: Supported profile and entrypoints
      VAProfileMPEG2Simple            : VAEntrypointVLD
      VAProfileMPEG2Main              : VAEntrypointVLD
      VAProfileVC1Simple              : VAEntrypointVLD
      VAProfileVC1Main                : VAEntrypointVLD
      VAProfileVC1Advanced            : VAEntrypointVLD
      VAProfileH264ConstrainedBaseline: VAEntrypointVLD
      VAProfileH264ConstrainedBaseline: VAEntrypointEncSlice
      VAProfileH264Main              : VAEntrypointVLD
      VAProfileH264Main              : VAEntrypointEncSlice
      VAProfileH264High              : VAEntrypointVLD
      VAProfileH264High              : VAEntrypointEncSlice
      VAProfileHEVCMain              : VAEntrypointVLD
      VAProfileHEVCMain              : VAEntrypointEncSlice
      VAProfileHEVCMain10            : VAEntrypointVLD
      VAProfileHEVCMain10            : VAEntrypointEncSlice
      VAProfileJPEGBaseline          : VAEntrypointVLD
      VAProfileVP9Profile0            : VAEntrypointVLD
      VAProfileVP9Profile2            : VAEntrypointVLD
      VAProfileNone                  : VAEntrypointVideoProc

ffmpeg error:

Code:

[libplacebo @ 0x7bcc48084d80] Requested image size 3840x1608x0 exceeds the maximum allowed dimensions 2560x2560x1 for vulkan image format VK_FORMAT_R16_UNORM
[libplacebo @ 0x7bcc48084d80] Failed mapping frame id 0 with PTS 0.000000
[fc#0 @ 0x58987e0f5d00] Error while filtering: Generic error in an external library
[fc#0 @ 0x58987e0f5d00] Task finished with error code: -542398533 (Generic error in an external library)
[fc#0 @ 0x58987e0f5d00] Terminating thread with return code -542398533 (Generic error in an external library)
[vost#0:0/h264_vaapi @ 0x58987e22cec0] Could not open encoder before EOF
[vost#0:0/h264_vaapi @ 0x58987e22cec0] Task finished with error code: -22 (Invalid argument)
[vost#0:0/h264_vaapi @ 0x58987e22cec0] Terminating thread with return code -22 (Invalid argument)
[out#0/hls @ 0x58987e23c600] Nothing was written into output file, because at least one of its streams received no packets.
frame=    0 fps=0.0 q=0.0 Lsize=      0KiB time=N/A bitrate=N/A speed=N/A   
Conversion failed!

I suspect this card could be used in the past as crypto miner, could it be related (non-standard bios or damaged card) or am I missing some configuration step?

Thank you for your time

[nyanmisaka]

There is no need to install proprietary drivers from AMD repo. Jellyfin package already includes open source Mesa drivers.

Please uninstall AMD proprietary drivers and try again (especially AMDVLK).

[arrguy]

You were right - removing drivers by amdgpu-install --uninstall, removing amdgpu-install completely, cleaning up with apt autoremove and restarting helped (had to restart whole server due to AMD bug where it cannot wake up card again, but it's not Jellyfin related)

Now transcoding with tone mapping works as intended

So simple fix, and I thought those drivers are necessary

Thank you