SOLVED: Server becomes unresponsive when transcoding for my phone

[magguzu]

Hi folks.

I have an issue that has been causing a lot of pain.

Whenever I try to watch a video on some devices that require codec transcoding, notably my phone, I experience freezing and my server becomes unresponsive until I reboot.

This doesn't seem to be for ALL transcoding though. If it's transcoding for bitrate, it seems to work well.

For those problematic streams, it almost seems as if the server is software transcoding, since CPU spikes and the server becomes unusable.

Jellyfin is running in a Docker container, on an Ubuntu VM in Proxmox.

I have transcoding configured through my Intel iGPU. UHD 770 (CPU is Intel Core i5-13500)

12 cores, 24GB RAM assigned to the VM.



I have the iGPU passed through to the VM, then again to the container like so:


  jellyfin:
    image: jellyfin/jellyfin:latest
    container_name: jellyfin
    ports:
      - 8096:8096
    group_add:
      - "109" # render group
      - "44" # video group
    devices:
      - /dev/dridev/dri #iGPU for transcoding
    volumes:
      - type: bind
        source: /services/jellyfin/jellyfin/jellyfin-web/config.json
        target: /jellyfin/jellyfin-web/config.json
      - type: bind
        source: /services/jellyfin/jellyfin/system.xml
        target: /etc/jellyfin/system.xml
      - /services/jellyfin/configconfig
      - /services/jellyfin/cachecache
      - /services/jellyfin/fontsfonts
      - /mnt/mediamedia
    restart: always
    environment:
      - TZ=America/New_York
      - JELLYFIN_PublishedServerUrl=xxxx
    logging:
      driver: "json-file"
      options:
        max-file: "10"
        max-size: "200k"



My transcoding configs:

   

I believe the i915 Intel low power transcoding is configured, though my paths are slightly different from what's in the Jellyfin doc:


root@jellyfintempsys/kernel/debug/dri/1/gt0/uc# cat huc_info
HuC firmware: i915/tgl_huc.bin
status: RUNNING
version: found 7.9.3
uCode: 589504 bytes
RSA: 256 bytes
HuC status: 0x00090001
root@jellyfintempsys/kernel/debug/dri/1/gt0/uc# cat guc_info
GuC firmware: i915/tgl_guc_70.bin
status: RUNNING
version: found 70.20.0
uCode: 334784 bytes
RSA: 256 bytes
GuC status 0x8003f0ec:
Bootrom status = 0x76
uKernel status = 0xf0
MIA Core status = 0x3
Scratch registers:
0: 0x0
1: 0x1633d7
2: 0x44000
3: 0x4000
4: 0x40
5: 0x4ec
6: 0x4680000c
7: 0x0
8: 0x0
9: 0x0
10: 0x0
11: 0x0
12: 0x0
13: 0x0
14: 0x0
15: 0x0

GuC logging stats:
Relay full count: 0
DEBUG: flush count          0, overflow count          0
CRASH: flush count          0, overflow count          0
CAPTURE: flush count          0, overflow count          0


Here is my ffmpeg log from my most recent reproducing:
https://pastebin.com/rpm2KBRB

And the container log:
https://pastebin.com/9vMhWFb6

Server config, database, metadata, etc are all on an NVME SSD. The media is on an HDD over NFS.

That's everything I can think of right now. Let me know if I need to provide anything else. Thanks in advance!

[TheDreadPirate]

Can you try disabling the Playback Reporting plugin and restarting?

[magguzu]

Thanks. I did, and reproduced the issue again afterwards.

Container log:

https://pastebin.com/PfU6jqtP

ffmpeg:

https://pastebin.com/tAutJbNh

This time I was running gputop while testing, and no activity was shown, which I think backs up my theory that it's not being used.

[TheDreadPirate]

What is the purpose of these two bind mounts? In particular, the system.xml bind.

Code:

- type: bind
        source: /services/jellyfin/jellyfin/jellyfin-web/config.json
        target: /jellyfin/jellyfin-web/config.json
      - type: bind
        source: /services/jellyfin/jellyfin/system.xml
        target: /etc/jellyfin/system.xml

For a docker container, the system.xml files is located in /config/config. So, as is, this bind does nothing.

[magguzu]

I bind mount config.json because I have some custom links in the side menu.

The system.xml was because I was trying to enable Prometheus metrics, which didn't work and I never undid it. And I see it's disabled in that directory you mentioned, so thanks for showing me why, haha.

[TheDreadPirate]

Enable debug logging by editing /config/config/logging.default.json. Replace "Information" and "Warning" in the top section with "Debug". You do not need to restart, the change should take effect immediately.

Replicate the problem and then share the new log. You may need to zip the log and attach it to a forum post since the log will get big very quickly and exceed pastebin's size limits.

Once you've done this undo the debug logging changes.

[magguzu]

Okay, I've made some progress.

it seems like the iGPU isn't used at all, even for bitrate transcoding. I guess my CPU was handling it okay enough that I never noticed, and maybe codec transcoding is more intensive?

I tested by transcoding in both JF and Plex and only Plex shows activity in gputop!

I know that when I select QSV in JF, there's no option to specify the device, unlike with VAAPI which does. On the Plex side I had to manually select my AlderLake device from a dropdown. Could this be why?

I tried passing /dev/dri/renderD128 directly, but this made no difference.

Here are the DEBUG logs you requested, while reproducing the issue.

Container:

https://drive.google.com/file/d/1o87163z...sp=sharing


ffmpeg for good measure:

https://pastebin.com/T3QZvk6R

[TheDreadPirate]

Your system is using the iGPU.

Code:

Stream mapping:
  Stream #0:0 -> #0:0 (hevc (hevc_qsv) -> h264 (h264_qsv))
  Stream #0:1 -> #0:1 (copy)

And the previous ffmpeg logs also indicated this. Any CPU usage may be from audio transcoding (not the case for this last one) or from subtitle burn in.

One thing I did notice is your passing in of a font directory. And the boat load of font attachments in this last video.

Are fonts an issue? Did you enable fallback fonts in Dashboard > Playback?

[magguzu]

The font directory was an experimental thing. It's been happening since before that. I can remove that.

yeah never mind on the iGPU thing...

gputop does show activity if I check it as root. I'm running Jellyfin as root and Plex as my own user. So I guess that was a red herring.

New findings though. I've been testing this with an anime show, which is HEVC, AAC and has SSA subtitles.
I'm doing another test with a much heavier 4K, HEVC, HDR, with SRT subs video, and all is fine. Server runs great while I play it and it's transcoding.

ffmpeg only briefly appears for the anime test, and is around for a lot longer for the 4K show, perhaps it just transcodes quickly for the former and exits.

So now I'm stumped again.

[TheDreadPirate]

There's a lot of noise in the debug log, but nothing stood out.

If you've configured fallback fonts in Dashboard > Playback, remove them if you have not done so already.

The issue could also be subtitle extraction. I'm assuming you are attempting to watch One Piece with the subtitles enabled. Does it play without issue if you watch it without subtitles?