Transcoding Works Perfectly until It Suddenly Doesn't

[familiarglyphs]

Hi all,

I've been at this for a year now and I'm finally ready to admit defeat and ask for help. 

SPECS:
Proxmox base, running two VMs: 1. TrueNAS 2. Ubuntu Server for Docker containers
Intel i3 12-100
32GB DDR5 RAM
GTX 1660 GPU
256GB SSD (runs Proxmox only)
1TB SSD (runs TrueNAS and Ubuntu VMs, plenty of headroom)
12TB CMR HDDs for storage (also plenty of headroom)

Here is a pastebin of my most recent transcode log that failed after a few minutes of working totally fine: https://pastebin.com/ZAKwxmu8

--

I've had this issue for a while now, but lately it seems to happen on every single watch. If I have a 4K HEVC file with multi-channel audio and HDR going on, attempting to stream to a less capable client like an iOS device or a 1080p PC running in a browser window causes the system to completely crash, taking other Docker containers down with it (including my DNS server, which makes monitoring difficult because Portainer goes down as well). In some crashes, even the TrueNAS VM dies and only Proxmox is left standing. So far, if I can avoid transcoding, everything works great, but occasionally I do watch from a mobile device and would like to at least understand why the crashes are happening. I asked for help in the Jellyfin Discord server, and they pointed me here after we tried to troubleshoot for a while.

During the last failed transcode (detailed in the pastebin above) I also had a terminal window with watch nvidia-smi going and a terminal window with htop for monitoring. I also had Jellyfin tell me playback info in the window itself while it was playing, and it appeared to be transcoding just fine. GPU utilization was only at 32% with no thermal issues whatsoever, and htop showed 14% CPU utilization across 4 threads I have allocated to the entire VM (thus considering all other containers running concurrently) with RAM at 2GB used out of 16GB available. The movie in question (The Batman 2019) played perfectly fine in 1080p with no tone mapping and downmixed to stereo from atmos for a few minutes before unceremoniously crashing. Linux logs show no out of memory errors or anything of the sort. Closest I was able to find was ffmpeg exiting with code 137. 

The only reason I even have a dedicated GPU in this system was because I couldn't get anything to play without it, despite research online pointing to the iGPU's capability for at least 1 4K stream without issue, and a clear overhead for memory in case that became an issue. My /cache folder is bound outside the container to a folder with half a TB of free space, and I've verified that it clears itself out. I'm at a total loss and not sure what to check next! Any help is appreciated.

Thanks