Choppy transcoding of high bitrate files with Hardware Acceleration enabled

Hi there,

I’m experiencing choppy transcoding of high bitrate files with Hardware Acceleration enabled. The file Jellyfin is transcoding has the following properties:

Containermkv
Path/data/movies/2160p.UHD.Remux.HEVC.DTS-HD.MA.5.1-playBD.mkv
Size47167 MB
Video
TitleMPEG-H HEVC Video / 64973 kbps / 2160p / 24 fps / 16:9 / Main 10 @ Level 5.1 @ High / 10 bits / HDR10 / BT.2020 - 4K
CodecHEVC
ProfileMain 10
Level153
Resolution3840x2160
Aspect ratio16:9
InterlacedNo
Framerate24
Bitrate68814 kbps
Video rangeHDR
Color transfersmpte2084
Color primariesbt2020
Pixel formatyuv420p10le
Ref frames1
Audio
TitleDTS-HD Master Audio / 5.1 / 48 kHz / 4174 kbps / 24-bit - English - Default
Languageeng
CodecDTS
ProfileDTS-HD MA
Layout5.1
Channels6 ch
Sample rate48000 Hz
Bit depth24 bit
DefaultYes

Jellyfin transcodes this to 1080p but at just 16fps. However, the bitrate is 115Mbps, even though I’ve set the streaming bitrate limit to 75Mbps. The transcode log can be found here: https://pastebin.com/rzQvnPnR
I’ve played around with various settings, such as the FFMpeg presets ultrafast, slow etc. as well as the CRF numbers to no avail.

The client is the Jellyfin webinterface running in FireFox on Windows 10. The server is running Jellyfin version: 10.7.0 on Docker (Operating System: Linux, Architecture: X64) on my Synology DS918+. This NAS is capable of hardware transcoding the following codcs:

I’ve enabled hardware transcoding by mapping the appropriate devices to the Docker container, assigning them the correct permissions, and selecting the appropriate codecs in in Jellyfin itself.

What could this be causing? Is the source file simple too big to handle? I don’t see my CPU suffering, it hovers between 40-70 percent. I would’ve expected to see 100% concurrent CPU usage if it was CPU limited.

Thank you!