H.265 (HEVC) transcoding not working for 4K HDR videos with QSV/VAAPI on Jellyfin

[NoisyBoy]

Hi,

Jellyfin is installed via the package from https://pdulvp.fr/jellyfin-qnap.html, on a QNAP TS-664 NAS with 16 GB of RAM. My system supports hardware acceleration (Intel Quick Sync), and it's properly enabled using VAAPI + QSV (iHD driver).

The issue:

• 4K HDR videos encoded in H.265 (HEVC) fail to transcode correctly — the screen stays black or playback never starts.
  
• Transcoding is triggered (e.g., logs show
  "Transcode (v:h264 a:aac)"
  ), but playback doesn't work.
  
• These videos include 10-bit, BT.2020, SMPTE2084 streams — for example, Moana (2016) in 4K HEVC.
  
• I also record videos in 4K HEVC with my phone, and Jellyfin is unable to play or transcode them either.
  

Playback works perfectly for AV1 and H.264 videos. There are no visible errors in the logs, but transcoding seems to silently fail or get stuck. Jellyfin is correctly using
ffmpeg with hardware acceleration (vaapi,qsv,hwmap, etc.).

I’ve attached logs to help with debugging. Let me know if I can help test anything further — thanks for your support!