Dropping frames/lag when using hardware acceleration

[ttsuam]

Setup
Hardware
Intel® Xeon® CPU E5-2678 v3 @ 2.50GHz
Nvidia GeForce GTX 1660 Super
32 GB ECC memory
a bunch of drives

Software
Hypervisor: Proxmox version 8.1.3 (running kernel: 6.5.11-7-pve)
VM: Ubuntu 22.04.3 LTS
Docker container: jellyfin/jellyfin:latest  (dashboard says jellyfin version 10.8.13)

Issue Description
When I play a 4K movie with hardware acceleration on, it transcodes it to something Firefox can handle but drops frames or stutters. If I turn off hardware acceleration, it may occasionally drop a frame or stutter, but less than with using the GPU. It seems to be the same results with different movies.

I suspect it could be related to the virtualization/containerization, but I am open to suggestions. I have passed my GPU through from Proxmox to the VM to the Jellyfin container. I have "confirmed" that the container is using the GPU by running nvidia-smi in the container while transcoding. If there is a better way to confirm it's working correctly, I'm willing to try it. I attached my FFMpeg and system logs.

[TheDreadPirate]

Can you attach an ffmpeg where the system is using NVENC. The log you attached is CPU transcoding.

Also, may I ask why you are both using a VM and a Docker container? Is that VM running more than one container?

[ttsuam]

Thank you for your reply. I have attached the FFMpeg logs using NVENC. As far as the container-inside-a-VM goes, I previously had multiple docker containers but have simplified down to just Jellyfin recently. I am using a hypervisor and VM so I have more flexibility in running other projects that are compartmentalized separate from the media. I've found Docker/docker compose to be convenient for keeping track of what I have installed and for getting a quick and easy fresh start when I mess something up.

[TheDreadPirate]

From a jellyfin perspective, everything looks normal. As far as I can tell anyway. It is transcoding fast enough. Both on the CPU and GPU.

So there is some other system bottleneck or problem somewhere that isn't obvious from the logs. Running out of memory? Some network issue?

[ttsuam]

The server VM is allocated 20GB of memory of the 32GB total on the host. When transcoding, it uses almost all of the 20GB, but I assume that's to be expected and that 20GB is enough.

Network-wise, it's fairly simple. I'm on the same LAN, just connecting based on IP with no DNS using an average off the shelf router. I can ping the VM with no problems. I'm not sure what I would be looking for if there was an issue here.

The portion of the CPU allocated to the VM hardly breaks 20% use when transcoding using the GPU, so I don't think that's the issue.

The only other thing I could think of is the hard drive throughput/access speed, but I'm not sure how I would test that. I have three Seagate Ironwolf st4000vn008-2dr166 and one Western Digital wdc_wd40ezrz-00gxcb0 in a ZFS pool on Proxmox.

Edit: Looking at Proxmox, the IODelay is as high as 25% and averages about 10-15% while transcoding, which in looking at other forums, seems to point to drives that are too slow. Do that seem reasonable? From what I'm reading, it seems like upgrading the drive might be the solution, but I'd like to avoid that if possible, $$$.

Edit2: It plays smoothly on the xbox, and the server IO delay is even lower, about 1-10%. That makes me think it's just the client struggling. But I'm not sure how that would change the IO delay

[TheDreadPirate]

ffmpeg should not be consuming that much memory. Is it actually using all that memory? Or is it being used as cache?

[ttsuam]

Now that I look more closely using the free command, it's only using 1 GB and the rest is buffer/cache. Before I was just looking at the Proxmox graph.