SOLVED: AMD GPU Tone mapping issue

SOLVED: AMD GPU Tone mapping issue - Printable Version

+- Jellyfin Forum (https://forum.jellyfin.org)
+-- Forum: Support (https://forum.jellyfin.org/f-support)
+--- Forum: Troubleshooting (https://forum.jellyfin.org/f-troubleshooting)
+--- Thread: SOLVED: AMD GPU Tone mapping issue (/t-solved-amd-gpu-tone-mapping-issue)

AMD GPU Tone mapping issue - arrguy - 2025-07-27

Hi,
I've a problem with tone mapping on AMD GPU. Recently I've got my hands on used Radeon RX 5700, that I thought could do just fine for transcoding.
After initial configuration, drivers, permissions etc. I've got it working, all is fine, except tone mapping - which I believe should work, since this card should be able to handle HEVC 10 bit - according to https://www.x.org/wiki/RadeonFeature/#radeonvcnvideocorenexthardware (NAVI 10)
I couldn't find any info about my ffmpeg transcoding error, that's why I wanted to create a topic here.

Environment: Jellyfin 10.10.7 with jellyfin-ffmpeg on Ubuntu, installed by install-debuntu script - running in VM on Proxmox with GPU passthrough

For decoding I've enabled: H264, HEVC, MPEG2, VC1, VP9, HEVC 10bit, VP9 10bit
Enabled encoding for HEVC

ls -l /dev/dri

Code:
drwxr-xr-x 2 root root        80 Jul 25 16:58 by-path

crw-rw---- 1 root video  226,  1 Jul 27 21:20 card1

crw-rw---- 1 root render 226, 128 Jul 25 16:58 renderD128

id jellyfin

Code:
uid=114(jellyfin) gid=119(jellyfin) groups=119(jellyfin),44(video),109(render)

drivers installed:

Code:
amdgpu-install --opencl=rocr --vulkan=amdvlk --opengl=mesa

vainfo:

Code:
error: XDG_RUNTIME_DIR is invalid or not set in the environment.

error: can't connect to X server!

libva info: VA-API version 1.20.0

libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/radeonsi_drv_video.so

libva info: Found init function __vaDriverInit_1_16

libva info: va_openDriver() returns 0

vainfo: VA-API version: 1.20 (libva 2.12.0)

vainfo: Driver version: Mesa Gallium driver 25.0.0-devel for AMD Radeon RX 5700 (radeonsi, navi10, LLVM 19.1.5, DRM 3.63, 6.8.0-64-generic)

vainfo: Supported profile and entrypoints

      VAProfileMPEG2Simple            : VAEntrypointVLD

      VAProfileMPEG2Main              : VAEntrypointVLD

      VAProfileVC1Simple              : VAEntrypointVLD

      VAProfileVC1Main                : VAEntrypointVLD

      VAProfileVC1Advanced            : VAEntrypointVLD

      VAProfileH264ConstrainedBaseline: VAEntrypointVLD

      VAProfileH264ConstrainedBaseline: VAEntrypointEncSlice

      VAProfileH264Main              : VAEntrypointVLD

      VAProfileH264Main              : VAEntrypointEncSlice

      VAProfileH264High              : VAEntrypointVLD

      VAProfileH264High              : VAEntrypointEncSlice

      VAProfileHEVCMain              : VAEntrypointVLD

      VAProfileHEVCMain              : VAEntrypointEncSlice

      VAProfileHEVCMain10            : VAEntrypointVLD

      VAProfileHEVCMain10            : VAEntrypointEncSlice

      VAProfileJPEGBaseline          : VAEntrypointVLD

      VAProfileVP9Profile0            : VAEntrypointVLD

      VAProfileVP9Profile2            : VAEntrypointVLD

      VAProfileNone                  : VAEntrypointVideoProc

ffmpeg error:

Code:
[libplacebo @ 0x7bcc48084d80] Requested image size 3840x1608x0 exceeds the maximum allowed dimensions 2560x2560x1 for vulkan image format VK_FORMAT_R16_UNORM

[libplacebo @ 0x7bcc48084d80] Failed mapping frame id 0 with PTS 0.000000

[fc#0 @ 0x58987e0f5d00] Error while filtering: Generic error in an external library

[fc#0 @ 0x58987e0f5d00] Task finished with error code: -542398533 (Generic error in an external library)

[fc#0 @ 0x58987e0f5d00] Terminating thread with return code -542398533 (Generic error in an external library)

[vost#0:0/h264_vaapi @ 0x58987e22cec0] Could not open encoder before EOF

[vost#0:0/h264_vaapi @ 0x58987e22cec0] Task finished with error code: -22 (Invalid argument)

[vost#0:0/h264_vaapi @ 0x58987e22cec0] Terminating thread with return code -22 (Invalid argument)

[out#0/hls @ 0x58987e23c600] Nothing was written into output file, because at least one of its streams received no packets.

frame=    0 fps=0.0 q=0.0 Lsize=      0KiB time=N/A bitrate=N/A speed=N/A    

Conversion failed!

I suspect this card could be used in the past as crypto miner, could it be related (non-standard bios or damaged card) or am I missing some configuration step?

Thank you for your time

RE: AMD GPU Tone mapping issue - nyanmisaka - 2025-07-28

There is no need to install proprietary drivers from AMD repo. Jellyfin package already includes open source Mesa drivers.

Please uninstall AMD proprietary drivers and try again (especially AMDVLK).

RE: AMD GPU Tone mapping issue - arrguy - 2025-07-28

You were right - removing drivers by amdgpu-install --uninstall, removing amdgpu-install completely, cleaning up with apt autoremove and restarting helped (had to restart whole server due to AMD bug where it cannot wake up card again, but it's not Jellyfin related)

Now transcoding with tone mapping works as intended Smiling-face

So simple fix, and I thought those drivers are necessary

Thank you