Thread Pool Starvation

[natzilla]

I am facing a situation where my system is being flooded with processes from /usr/bin/jellyfin



There are hundreds, possible thousands of these entries in htop. It's obvious something is hung here and I'd like to know some ways to further investigate it. I have not rebooted the server which in my experience does clear it, but I want to root cause this first.

More confirmation details regarding CPU usage being starved.





● jellyfin.service - Jellyfin Media Server
    Loaded: loaded (/lib/systemd/system/jellyfin.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/jellyfin.service.d
            └─jellyfin.service.conf
    Active: active (running) since Fri 2023-09-01 19:29:02 UTC; 2 days ago
  Main PID: 732 (jellyfin)
      Tasks: 3636 (limit: 18546)
    Memory: 13.7G
        CPU: 2d 19h 15min 58.966s
    CGroup: /system.slice/jellyfin.service
            └─732 /usr/bin/jellyfin --webdir=/usr/share/jellyfin/web --restartpath=/usr/lib/jellyfin/restart.sh --ffmpeg=/usr/lib/jellyfin-ffmpeg/ffmpeg

Sep 04 14:59:29 jellyfin jellyfin[732]: [14:59:29] [WRN] As of "09/04/2023 14:59:09 +00:00", the heartbeat has been running for "00:00:20.7576155" which is longer than "00:00:01". This could be caused by thread pool starvation.
Sep 04 14:59:50 jellyfin jellyfin[732]: [14:59:50] [WRN] As of "09/04/2023 14:59:31 +00:00", the heartbeat has been running for "00:00:10.8371778" which is longer than "00:00:01". This could be caused by thread pool starvation.
Sep 04 15:00:13 jellyfin jellyfin[732]: [15:00:13] [WRN] As of "09/04/2023 14:59:51 +00:00", the heartbeat has been running for "00:00:21.2132112" which is longer than "00:00:01". This could be caused by thread pool starvation.
Sep 04 15:00:42 jellyfin jellyfin[732]: [15:00:42] [WRN] As of "09/04/2023 15:00:22 +00:00", the heartbeat has been running for "00:00:20.3549039" which is longer than "00:00:01". This could be caused by thread pool starvation.
Sep 04 15:00:53 jellyfin jellyfin[732]: [15:00:53] [WRN] As of "09/04/2023 15:00:43 +00:00", the heartbeat has been running for "00:00:10.0880467" which is longer than "00:00:01". This could be caused by thread pool starvation.
Sep 04 15:01:16 jellyfin jellyfin[732]: [15:01:16] [WRN] As of "09/04/2023 15:00:55 +00:00", the heartbeat has been running for "00:00:21.4189816" which is longer than "00:00:01". This could be caused by thread pool starvation.
Sep 04 15:01:25 jellyfin jellyfin[732]: [15:01:25] [WRN] As of "09/04/2023 15:01:18 +00:00", the heartbeat has been running for "00:00:07.2229611" which is longer than "00:00:01". This could be caused by thread pool starvation.
Sep 04 15:01:37 jellyfin jellyfin[732]: [15:01:37] [WRN] As of "09/04/2023 15:01:26 +00:00", the heartbeat has been running for "00:00:10.6905181" which is longer than "00:00:01". This could be caused by thread pool starvation.
Sep 04 15:01:39 jellyfin jellyfin[732]: [15:01:39] [WRN] As of "09/04/2023 15:01:38 +00:00", the heartbeat has been running for "00:00:01.5668541" which is longer than "00:00:01". This could be caused by thread pool starvation.
Sep 04 15:01:43 jellyfin jellyfin[732]: [15:01:43] [WRN] As of "09/04/2023 15:01:41 +00:00", the heartbeat has been running for "00:00:02.4147117" which is longer than "00:00:01". This could be caused by thread pool starvation.

System details

No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 22.04.3 LTS
Release: 22.04
Codename: jammy

NVIDIA-SMI 525.125.06  Driver Version: 525.125.06  CUDA Version: 12.0