Hardware Transcoding

[jppars]

Hi! My docker jellyfin instance is failing or struggling to transcode using a Quadro P400.

My understanding is that this GPU should be able to handle multiple 1080p transcodes at once, but I'm seeing just one max out the GPU with a large CPU load on top of that. Direct playback is working perfectly, so this makes me think I must have something configured wrong, but I can't seem to figure out what.

If anyone has any suggestions or insight that would greatly appreciated!


[Some more info about the problem and my setup]

This was originally observed on the Firefox web UI. I would open the media file, jellyfin displays a still image from metadata,
then closes the media file and displays this error with the following FFmpeg log:

# Firefox Web UI Error
   

FFmpeg log associated with the error

Output of nvidia-smi during attempted transcoding

Abbreviated output of top during same test


I've tried with multiple different files with different codecs (typically the source file is either AVC or HEVC),
all with the same failure.

I've also seen similar behavior using Swiftfin on iPhone and AppleTV, but they would just sit on a black screen, or start playback and stutter every second or so. Below is an FFmpeg log of a stuttering stream to Swiftfin on iPhone:

Stuttering Swiftfin iPhone FFmpeg log


I installed the NVIDIA and CUDA drivers through apt following this guide: 
https://wiki.debian.org/NvidiaGraphicsDrivers
I also installed and configured the nvidia-container-toolkit for docker following this guide: 
https://docs.nvidia.com/datacenter/cloud...guide.html


Lastly, I've included my jellyfin hardware transcoding settings, docker compose file and the resources allocated to the VM below:

   

Jellyfin docker compose file

I'm running Proxmox with the following hardware:
CPU:  AMD Ryzen 5 4650G
Mobo: Asus Pro B550M-C
GPU:  Quadro P400

And docker Jellyfin on a headless Debian 12 VM with the following resources:
4 Cores
8Gb RAM
Quadro P400 using VFIO passthrough


Thanks!