Washed out Colors of HDR

[speedhaxu]

Hi, I have jellyfin running 10.11.2 in Docker in unraid. My goal is to watch content with good colors regardless if its in HDR or what client I'm watching on.
I have an intel 12500 which is passed through and transcoding works overall. However, depending on the client I can get washed out colors on HDR content even when trying to transcode and tonemap. I've attached my transcoding options as well. 

I'm testing with a few clients:
1. windows with jellyfin desktop client with direct play works
   

2. windows with jellyfin desktop client with transcode has washed out colors
These are the ffmpeg logs

[13:22:06] [INF] [16] MediaBrowser.MediaEncoding.Transcoding.TranscodeManager: /usr/lib/jellyfin-ffmpeg/ffmpeg -analyzeduration 200M -probesize 1G -ss 00:50:48.000 -f matroska -init_hw_device vaapi=va:,vendor_id=0x8086,driver=iHD -init_hw_device qsv=qs@va -filter_hw_device qs -hwaccel vaapi -hwaccel_output_format vaapi -noautorotate -i file:"/data/Movies/Ratatouille.2007.2160p.UHD.BluRay.x265-TERMiNAL/Ratatouille (2007) [imdbid-tt0382932] - [Bluray-2160p][HDR10][TrueHD Atmos 7.1][x265]-TERMiNAL.mkv" -noautoscale -map_metadata -1 -map_chapters -1 -threads 4 -map 0:0 -map 0:2 -map -0:s -codec:v:0 hevc_qsv -tag:v:0 hvc1 -preset veryfast -mbbrc 1 -b:v 7360000 -maxrate 7360001 -rc_init_occupancy 14720000 -bufsize 29440000 -profile:v:0 main -level 50 -g:v:0 72 -keyint_min:v:0 72 -vf "setparams=color_primaries=bt709:color_trc=bt709:colorspace=bt709,scale_vaapi=w=2560:h=1070:format=nv12:extra_hw_frames=24,hwmap=derive_device=qsv,format=qsv" -codec:a:0 libfdk_aac -ac 2 -ab 256000 -af "volume=2" -copyts -avoid_negative_ts disabled -max_muxing_queue_size 2048 -f hls -max_delay 5000000 -hls_time 3 -hls_segment_type mpegts -start_number 1016 -hls_segment_filename "/config/data/transcodes/af95738135863280ff3e6a4ae7c1b424%d.ts" -hls_playlist_type vod -hls_list_size 0 -y "/config/data/transcodes/af95738135863280ff3e6a4ae7c1b424.m3u8"

   

3. MacOS with jellyfin in firefox direct streams works
   

4. MacOS with jellyfin in firefox transcode has washed out colors
The ffmpeg log for this

endor_id=0x8086,driver=iHD -init_hw_device qsv=qs@va -filter_hw_device qs -hwaccel vaapi -hwaccel_output_format vaapi -noautorotate -i file:"/data/Movies/Ratatouille.2007.2160p.UHD.BluRay.x265-TERMiNAL/Ratatouille (2007) [imdbid-tt0382932] - [Bluray-2160p][HDR10][TrueHD Atmos 7.1][x265]-TERMiNAL.mkv" -noautoscale -map_metadata -1 -map_chapters -1 -threads 4 -map 0:0 -map 0:2 -map -0:s -codec:v:0 hevc_qsv -tag:v:0 hvc1 -preset veryfast -mbbrc 1 -b:v 5616000 -maxrate 5616001 -rc_init_occupancy 11232000 -bufsize 22464000 -profile:v:0 main -level 50 -g:v:0 72 -keyint_min:v:0 72 -vf "setparams=color_primaries=bt709:color_trc=bt709:colorspace=bt709,scale_vaapi=w=1920:h=802:format=nv12:extra_hw_frames=24,hwmap=derive_device=qsv,format=qsv" -codec:a:0 libfdk_aac -ac 2 -ab 256000 -af "volume=2" -copyts -avoid_negative_ts disabled -max_muxing_queue_size 2048 -f hls -max_delay 5000000 -hls_time 3 -hls_segment_type fmp4 -hls_fmp4_init_filename "cff013d082d05c2ca4a7bd1ea83fd65f-1.mp4" -start_number 1082 -hls_segment_filename "/config/data/transcodes/cff013d082d05c2ca4a7bd1ea83fd65f%d.mp4" -hls_playlist_type vod -hls_list_size 0 -hls_segment_options movflags=+frag_discont -y "/config/data/transcodes/cff013d082d05c2ca4a7bd1ea83fd65f.m3u8"

   


I'm not sure how to consistently get good colors, either through HDR playing or tonemapping to SDR. Appreciate all the help!

[ArneR]

Tone mapping is not enabled in your last screenshot.
Side note: I'm not too thrilled about the performance of encoding to HEVC, even though I have an Arc A310, so I just let it encode to H.264 and tone map HDR->SDR, and it works fine for me. Maybe worth a try.