Transcoding Works Perfectly until It Suddenly Doesn't

[familiarglyphs]

Hi all,

I've been at this for a year now and I'm finally ready to admit defeat and ask for help. 

SPECS:
Proxmox base, running two VMs: 1. TrueNAS 2. Ubuntu Server for Docker containers
Intel i3 12-100
32GB DDR5 RAM
GTX 1660 GPU
256GB SSD (runs Proxmox only)
1TB SSD (runs TrueNAS and Ubuntu VMs, plenty of headroom)
12TB CMR HDDs for storage (also plenty of headroom)

Here is a pastebin of my most recent transcode log that failed after a few minutes of working totally fine: https://pastebin.com/ZAKwxmu8

--

I've had this issue for a while now, but lately it seems to happen on every single watch. If I have a 4K HEVC file with multi-channel audio and HDR going on, attempting to stream to a less capable client like an iOS device or a 1080p PC running in a browser window causes the system to completely crash, taking other Docker containers down with it (including my DNS server, which makes monitoring difficult because Portainer goes down as well). In some crashes, even the TrueNAS VM dies and only Proxmox is left standing. So far, if I can avoid transcoding, everything works great, but occasionally I do watch from a mobile device and would like to at least understand why the crashes are happening. I asked for help in the Jellyfin Discord server, and they pointed me here after we tried to troubleshoot for a while.

During the last failed transcode (detailed in the pastebin above) I also had a terminal window with watch nvidia-smi going and a terminal window with htop for monitoring. I also had Jellyfin tell me playback info in the window itself while it was playing, and it appeared to be transcoding just fine. GPU utilization was only at 32% with no thermal issues whatsoever, and htop showed 14% CPU utilization across 4 threads I have allocated to the entire VM (thus considering all other containers running concurrently) with RAM at 2GB used out of 16GB available. The movie in question (The Batman 2019) played perfectly fine in 1080p with no tone mapping and downmixed to stereo from atmos for a few minutes before unceremoniously crashing. Linux logs show no out of memory errors or anything of the sort. Closest I was able to find was ffmpeg exiting with code 137. 

The only reason I even have a dedicated GPU in this system was because I couldn't get anything to play without it, despite research online pointing to the iGPU's capability for at least 1 4K stream without issue, and a clear overhead for memory in case that became an issue. My /cache folder is bound outside the container to a folder with half a TB of free space, and I've verified that it clears itself out. I'm at a total loss and not sure what to check next! Any help is appreciated.

Thanks

[bitmap]

What issues did you have with Quicksync? A 12th gen i3 should definitely be able to handle this sort of job. Are you interested in root cause analysis or trying to get QSV hardware acceleration working?

[familiarglyphs]

At the moment I would like to understand why the NVENC workflow suddenly goes down and takes the server with it when all signs point to a well-oiled machine. I only have the GPU in the system at all to support hardware accelerated transcoding, so getting things to work with just QuickSync would be a great stretch goal for afterwards. Being able to run the system without a GPU would save on power consumption, so I'd definitely go that route, it's just a lot of setup through proxmox before I can even test it, as opposed to the GPU which is already correctly isolated and passed through to the container.

[bitmap]

Can you provide your Jellyfin logs or clear things out, recreate the issue, and then provide the Jellyfin log? Remove domains, leave all else. There are no errors at all in the transcode log you provided...in fact, it looks like it finished successfully. So the issue might lie elsewhere.

ETA: Keep an eye on your Ubuntu version as well, since #4 of the known issues will hinder your progress on Quicksync.

https://jellyfin.org/docs/general/post-i...l-on-linux

[Saturng2005]

Hey! I was having the same issue, and I was able to fix it. In my case, it had to do with some GPU check causing the transcoding to fail every few hours, forcing me to reboot.

Try something for me and tell me what happens:

While your transcoding is working, stop the video, go to your server's terminal and type in "sudo systemctl daemon-reload". Now go back to the video stream and see if transcoding is working still. If it's not, I know what's up

[familiarglyphs]

Hey! I was having the same issue, and I was able to fix it. In my case, it had to do with some GPU check causing the transcoding to fail every few hours, forcing me to reboot.

Try something for me and tell me what happens:

While your transcoding is working, stop the video, go to your server's terminal and type in "sudo systemctl daemon-reload". Now go back to the video stream and see if transcoding is working still. If it's not, I know what's up

I tried this and transcoding didn't stop at all. Might not be this then.

UPDATE: I closed out of the video stream and tried starting it again after the daemon-reload, now I'm getting fatal playback error. Maybe this is the issue after all?

[familiarglyphs]

Can you provide your Jellyfin logs or clear things out, recreate the issue, and then provide the Jellyfin log? Remove domains, leave all else. There are no errors at all in the transcode log you provided...in fact, it looks like it finished successfully. So the issue might lie elsewhere.

ETA: Keep an eye on your Ubuntu version as well, since #4 of the known issues will hinder your progress on Quicksync.

https://jellyfin.org/docs/general/post-i...l-on-linux

Recreated the issue and here are the Jellyfin logs.
https://pastebin.com/Gqbznn4Z

I was monitoring htop and the Docker container's logs in real time during the transcode, and everything seemed normal. Once playback failed, the ffmpeg process seemed to continue for a while in htop before eventually giving up. My Pi-hole container broke as usual when Jellyfin breaks.. very odd as resources clearly did not run out and the two containers do not share a network stack.