SOLVED: Disk full because of transcode directory

[Duvel]

Hi

Running 10.9.11 in docker : jellyfin/jellyfin:latest on a Ubuntu 22.04 VM.

I've got a mature server with tenths of users running for years with no issues.

This morning that server was KO because the system disk was full

tom@www-prd:~$ df
Filesystem      1K-blocks        Used  Available Use% Mounted on
tmpfs              1224444        5560    1218884  1% /run
/dev/sda2        102656700    98453460          0 100% /
tmpfs              6122208          28    6122180  1% /dev/shm
tmpfs                5120          0      5120  0% /run/lock
/dev/sdb1      1073216516  545110924  528105592  51% /var/www
/dev/sdc1      16104039404 15023022980 1081016424  94% /media/plex
tmpfs              1224440          60    1224380  1% /run/user/128

I've got a 100TB SDD for the system and its where Jellyfin files lies, including the transcode directory

There were more than 13k files in the directory and all files were created on yesterday and 2 days ago.
like:
-rw-r--r--  1 jellyfin jellyfin  2443293 Oct  2 21:29 f27db32e08a850103ccdaf0b5e93e9b35.mp4
-rw-r--r--  1 jellyfin jellyfin  2317467 Oct  2 21:29 f27db32e08a850103ccdaf0b5e93e9b360.mp4
-rw-r--r--  1 jellyfin jellyfin  2652326 Oct  2 21:29 f27db32e08a850103ccdaf0b5e93e9b361.mp4
-rw-r--r--  1 jellyfin jellyfin  3282917 Oct  2 21:29 f27db32e08a850103ccdaf0b5e93e9b362.mp4
-rw-r--r--  1 jellyfin jellyfin  2614603 Oct  2 21:29 f27db32e08a850103ccdaf0b5e93e9b363.mp4
-rw-r--r--  1 jellyfin jellyfin  2999447 Oct  2 21:29 f27db32e08a850103ccdaf0b5e93e9b364.mp4
-rw-r--r--  1 jellyfin jellyfin  2785210 Oct  2 21:29 f27db32e08a850103ccdaf0b5e93e9b365.mp4
-rw-r--r--  1 jellyfin jellyfin  2418162 Oct  2 21:29 f27db32e08a850103ccdaf0b5e93e9b366.mp4
-rw-r--r--  1 jellyfin jellyfin  2461479 Oct  2 21:29 f27db32e08a850103ccdaf0b5e93e9b367.mp4
-rw-r--r--  1 jellyfin jellyfin  2489552 Oct  2 21:29 f27db32e08a850103ccdaf0b5e93e9b368.mp4
-rw-r--r--  1 jellyfin jellyfin  2464962 Oct  2 21:29 f27db32e08a850103ccdaf0b5e93e9b369.mp4
-rw-r--r--  1 jellyfin jellyfin  2285016 Oct  2 21:29 f27db32e08a850103ccdaf0b5e93e9b36.mp4
-rw-r--r--  1 jellyfin jellyfin  2456811 Oct  2 21:29 f27db32e08a850103ccdaf0b5e93e9b370.mp4
-rw-r--r--  1 jellyfin jellyfin  2520562 Oct  2 21:29 f27db32e08a850103ccdaf0b5e93e9b371.mp4
-rw-r--r--  1 jellyfin jellyfin  2570875 Oct  2 21:29 f27db32e08a850103ccdaf0b5e93e9b372.mp4
-rw-r--r--  1 jellyfin jellyfin  2911734 Oct  2 21:29 f27db32e08a850103ccdaf0b5e93e9b373.mp4
-rw-r--r--  1 jellyfin jellyfin  2606474 Oct  2 21:29 f27db32e08a850103ccdaf0b5e93e9b374.mp4
-rw-r--r--  1 jellyfin jellyfin  2675693 Oct  2 21:29 f27db32e08a850103ccdaf0b5e93e9b375.mp4
-rw-r--r--  1 jellyfin jellyfin  2890279 Oct  2 21:29 f27db32e08a850103ccdaf0b5e93e9b376.mp4
-rw-r--r--  1 jellyfin jellyfin  2769607 Oct  2 21:29 f27db32e08a850103ccdaf0b5e93e9b377.mp4
-rw-r--r--  1 jellyfin jellyfin  2905585 Oct  2 21:29 f27db32e08a850103ccdaf0b5e93e9b378.mp4
-rw-r--r--  1 jellyfin jellyfin  2449224 Oct  2 21:29 f27db32e08a850103ccdaf0b5e93e9b379.mp4
-rw-r--r--  1 jellyfin jellyfin  1690650 Oct  2 21:29 f27db32e08a850103ccdaf0b5e93e9b37.mp4
-rw-r--r--  1 jellyfin jellyfin  2111243 Oct  2 21:29 f27db32e08a850103ccdaf0b5e93e9b380.mp4
-rw-r--r--  1 jellyfin jellyfin  1927557 Oct  2 21:29 f27db32e08a850103ccdaf0b5e93e9b381.mp4
-rw-r--r--  1 jellyfin jellyfin  2007292 Oct  2 21:29 f27db32e08a850103ccdaf0b5e93e9b382.mp4
-rw-r--r--  1 jellyfin jellyfin  2684367 Oct  2 21:29 f27db32e08a850103ccdaf0b5e93e9b383.mp4
-rw-r--r--  1 jellyfin jellyfin  2396827 Oct  2 21:29 f27db32e08a850103ccdaf0b5e93e9b384.mp4
-rw-r--r--  1 jellyfin jellyfin  2360938 Oct  2 21:29 f27db32e08a850103ccdaf0b5e93e9b385.mp4
-rw-r--r--  1 jellyfin jellyfin  2843802 Oct  2 21:29 f27db32e08a850103ccdaf0b5e93e9b386.mp4
-rw-r--r--  1 jellyfin jellyfin  3176584 Oct  2 21:29 f27db32e08a850103ccdaf0b5e93e9b387.mp4
-rw-r--r--  1 jellyfin jellyfin  2668085 Oct  2 21:29 f27db32e08a850103ccdaf0b5e93e9b388.mp4
-rw-r--r--  1 jellyfin jellyfin  2994367 Oct  2 21:29 f27db32e08a850103ccdaf0b5e93e9b389.mp4
-rw-r--r--  1 jellyfin jellyfin  2063018 Oct  2 21:29 f27db32e08a850103ccdaf0b5e93e9b38.mp4
-rw-r--r--  1 jellyfin jellyfin  2744151 Oct  2 21:29 f27db32e08a850103ccdaf0b5e93e9b390.mp4
-rw-r--r--  1 jellyfin jellyfin  2591348 Oct  2 21:29 f27db32e08a850103ccdaf0b5e93e9b391.mp4
-rw-r--r--  1 jellyfin jellyfin  2768626 Oct  2 21:29 f27db32e08a850103ccdaf0b5e93e9b392.mp4
-rw-r--r--  1 jellyfin jellyfin  2782224 Oct  2 21:29 f27db32e08a850103ccdaf0b5e93e9b393.mp4
-rw-r--r--  1 jellyfin jellyfin  2853041 Oct  2 21:29 f27db32e08a850103ccdaf0b5e93e9b394.mp4
-rw-r--r--  1 jellyfin jellyfin  3026344 Oct  2 21:29 f27db32e08a850103ccdaf0b5e93e9b395.mp4
-rw-r--r--  1 jellyfin jellyfin  3116120 Oct  2 21:29 f27db32e08a850103ccdaf0b5e93e9b396.mp4
-rw-r--r--  1 jellyfin jellyfin  2333962 Oct  2 21:29 f27db32e08a850103ccdaf0b5e93e9b397.mp4
-rw-r--r--  1 jellyfin jellyfin  2372371 Oct  2 21:29 f27db32e08a850103ccdaf0b5e93e9b398.mp4

I purged the whole directory and regained 30TB

2 days ago (1st october evening)  I found that the jellyfin server was slow to respond, while I had 5 users watching something on it. I had a look at the log and did not find anything obvious. I thus decided to restart the container. I rarely have to do this. The server answered much better after that.

All the transcode files were newer than that restart, meaning that it created 13k transcode files in 24 hours and never deleted them itself.

There's obviously a big issue somewhere and I'd like that you help me to find the root cause in my log files (I'd like to avoid to post them all publicly if possible).

Cheers