SOLVED: Disk full because of transcode directory - Printable Version +- Jellyfin Forum (https://forum.jellyfin.org) +-- Forum: Support (https://forum.jellyfin.org/f-support) +--- Forum: Troubleshooting (https://forum.jellyfin.org/f-troubleshooting) +--- Thread: SOLVED: Disk full because of transcode directory (/t-solved-disk-full-because-of-transcode-directory) |
Disk full because of transcode directory - Duvel - 2024-10-03 Hi Running 10.9.11 in docker : jellyfin/jellyfin:latest on a Ubuntu 22.04 VM. I've got a mature server with tenths of users running for years with no issues. This morning that server was KO because the system disk was full Quote:tom@www-prd:~$ df I've got a 100TB SDD for the system and its where Jellyfin files lies, including the transcode directory There were more than 13k files in the directory and all files were created on yesterday and 2 days ago. like: -rw-r--r-- 1 jellyfin jellyfin 2443293 Oct 2 21:29 f27db32e08a850103ccdaf0b5e93e9b35.mp4 -rw-r--r-- 1 jellyfin jellyfin 2317467 Oct 2 21:29 f27db32e08a850103ccdaf0b5e93e9b360.mp4 -rw-r--r-- 1 jellyfin jellyfin 2652326 Oct 2 21:29 f27db32e08a850103ccdaf0b5e93e9b361.mp4 -rw-r--r-- 1 jellyfin jellyfin 3282917 Oct 2 21:29 f27db32e08a850103ccdaf0b5e93e9b362.mp4 -rw-r--r-- 1 jellyfin jellyfin 2614603 Oct 2 21:29 f27db32e08a850103ccdaf0b5e93e9b363.mp4 -rw-r--r-- 1 jellyfin jellyfin 2999447 Oct 2 21:29 f27db32e08a850103ccdaf0b5e93e9b364.mp4 -rw-r--r-- 1 jellyfin jellyfin 2785210 Oct 2 21:29 f27db32e08a850103ccdaf0b5e93e9b365.mp4 -rw-r--r-- 1 jellyfin jellyfin 2418162 Oct 2 21:29 f27db32e08a850103ccdaf0b5e93e9b366.mp4 -rw-r--r-- 1 jellyfin jellyfin 2461479 Oct 2 21:29 f27db32e08a850103ccdaf0b5e93e9b367.mp4 -rw-r--r-- 1 jellyfin jellyfin 2489552 Oct 2 21:29 f27db32e08a850103ccdaf0b5e93e9b368.mp4 -rw-r--r-- 1 jellyfin jellyfin 2464962 Oct 2 21:29 f27db32e08a850103ccdaf0b5e93e9b369.mp4 -rw-r--r-- 1 jellyfin jellyfin 2285016 Oct 2 21:29 f27db32e08a850103ccdaf0b5e93e9b36.mp4 -rw-r--r-- 1 jellyfin jellyfin 2456811 Oct 2 21:29 f27db32e08a850103ccdaf0b5e93e9b370.mp4 -rw-r--r-- 1 jellyfin jellyfin 2520562 Oct 2 21:29 f27db32e08a850103ccdaf0b5e93e9b371.mp4 -rw-r--r-- 1 jellyfin jellyfin 2570875 Oct 2 21:29 f27db32e08a850103ccdaf0b5e93e9b372.mp4 -rw-r--r-- 1 jellyfin jellyfin 2911734 Oct 2 21:29 f27db32e08a850103ccdaf0b5e93e9b373.mp4 -rw-r--r-- 1 jellyfin jellyfin 2606474 Oct 2 21:29 f27db32e08a850103ccdaf0b5e93e9b374.mp4 -rw-r--r-- 1 jellyfin jellyfin 2675693 Oct 2 21:29 f27db32e08a850103ccdaf0b5e93e9b375.mp4 -rw-r--r-- 1 jellyfin jellyfin 2890279 Oct 2 21:29 f27db32e08a850103ccdaf0b5e93e9b376.mp4 -rw-r--r-- 1 jellyfin jellyfin 2769607 Oct 2 21:29 f27db32e08a850103ccdaf0b5e93e9b377.mp4 -rw-r--r-- 1 jellyfin jellyfin 2905585 Oct 2 21:29 f27db32e08a850103ccdaf0b5e93e9b378.mp4 -rw-r--r-- 1 jellyfin jellyfin 2449224 Oct 2 21:29 f27db32e08a850103ccdaf0b5e93e9b379.mp4 -rw-r--r-- 1 jellyfin jellyfin 1690650 Oct 2 21:29 f27db32e08a850103ccdaf0b5e93e9b37.mp4 -rw-r--r-- 1 jellyfin jellyfin 2111243 Oct 2 21:29 f27db32e08a850103ccdaf0b5e93e9b380.mp4 -rw-r--r-- 1 jellyfin jellyfin 1927557 Oct 2 21:29 f27db32e08a850103ccdaf0b5e93e9b381.mp4 -rw-r--r-- 1 jellyfin jellyfin 2007292 Oct 2 21:29 f27db32e08a850103ccdaf0b5e93e9b382.mp4 -rw-r--r-- 1 jellyfin jellyfin 2684367 Oct 2 21:29 f27db32e08a850103ccdaf0b5e93e9b383.mp4 -rw-r--r-- 1 jellyfin jellyfin 2396827 Oct 2 21:29 f27db32e08a850103ccdaf0b5e93e9b384.mp4 -rw-r--r-- 1 jellyfin jellyfin 2360938 Oct 2 21:29 f27db32e08a850103ccdaf0b5e93e9b385.mp4 -rw-r--r-- 1 jellyfin jellyfin 2843802 Oct 2 21:29 f27db32e08a850103ccdaf0b5e93e9b386.mp4 -rw-r--r-- 1 jellyfin jellyfin 3176584 Oct 2 21:29 f27db32e08a850103ccdaf0b5e93e9b387.mp4 -rw-r--r-- 1 jellyfin jellyfin 2668085 Oct 2 21:29 f27db32e08a850103ccdaf0b5e93e9b388.mp4 -rw-r--r-- 1 jellyfin jellyfin 2994367 Oct 2 21:29 f27db32e08a850103ccdaf0b5e93e9b389.mp4 -rw-r--r-- 1 jellyfin jellyfin 2063018 Oct 2 21:29 f27db32e08a850103ccdaf0b5e93e9b38.mp4 -rw-r--r-- 1 jellyfin jellyfin 2744151 Oct 2 21:29 f27db32e08a850103ccdaf0b5e93e9b390.mp4 -rw-r--r-- 1 jellyfin jellyfin 2591348 Oct 2 21:29 f27db32e08a850103ccdaf0b5e93e9b391.mp4 -rw-r--r-- 1 jellyfin jellyfin 2768626 Oct 2 21:29 f27db32e08a850103ccdaf0b5e93e9b392.mp4 -rw-r--r-- 1 jellyfin jellyfin 2782224 Oct 2 21:29 f27db32e08a850103ccdaf0b5e93e9b393.mp4 -rw-r--r-- 1 jellyfin jellyfin 2853041 Oct 2 21:29 f27db32e08a850103ccdaf0b5e93e9b394.mp4 -rw-r--r-- 1 jellyfin jellyfin 3026344 Oct 2 21:29 f27db32e08a850103ccdaf0b5e93e9b395.mp4 -rw-r--r-- 1 jellyfin jellyfin 3116120 Oct 2 21:29 f27db32e08a850103ccdaf0b5e93e9b396.mp4 -rw-r--r-- 1 jellyfin jellyfin 2333962 Oct 2 21:29 f27db32e08a850103ccdaf0b5e93e9b397.mp4 -rw-r--r-- 1 jellyfin jellyfin 2372371 Oct 2 21:29 f27db32e08a850103ccdaf0b5e93e9b398.mp4 I purged the whole directory and regained 30TB 2 days ago (1st october evening) I found that the jellyfin server was slow to respond, while I had 5 users watching something on it. I had a look at the log and did not find anything obvious. I thus decided to restart the container. I rarely have to do this. The server answered much better after that. All the transcode files were newer than that restart, meaning that it created 13k transcode files in 24 hours and never deleted them itself. There's obviously a big issue somewhere and I'd like that you help me to find the root cause in my log files (I'd like to avoid to post them all publicly if possible). Cheers RE: Disk full because of transcode directory - gnattu - 2024-10-03 The server should have a scheduled task called "clean transcode directory" that will automatically run every 24 hours, and will clean all files created more than 24 hours ago. If the client behaves correctly the server will clean the files immediately after the playback session ends, but if some of your users are using a client that handles the session unreliably which will make the server to rely on this scheduled task to clean transcode cache. The max amount you need in this unideal scenario is the amount of disks for 24hours playback. BTW, your size looks like GB not TB to me, and 30 GB cache is nothing for a busy server streaming high bitrate videos. If make the cache larger is not an option, you can go to dashboard->Playback->Transcoding, scroll down and check delete segments. This will delete the segments during the playback after the clients reported it has already played that part, but a reminder is that that option is not always reliable and the functionality is very client dependent. Some client may even fail to playback with that enabled as it does not accurately report playback time. RE: Disk full because of transcode directory - Duvel - 2024-10-03 yes, I more or less knew all of this. But how do you explain that with max 10 users/clients active during 24 hours, Jellyfin has generated 13,750 transcode files? It just doesnt make sense unless there's an issue somewhere, server-side. One user alone could not have triggered that amount of transcode files unless he tried to chain-start thousands of different videos purposely. All my users are good friends that are using my server for years, and none of them is an idiot. Or it might be one of their clients that bugged and sent thousands of requests ??? How to determine this ? RE: Disk full because of transcode directory - TheDreadPirate - 2024-10-03 That file count is not abnormally high. Each transcode file is NOT the full movie/show. Transcode files are a few seconds of the original video so that they can be served in small chunks. This is how Youtube/Twitch/Netflix/etc. serves up content for their clients. Depending on the length of the video, this could mean dozens or hundreds of segments per movie/show. So the fact that you have 13K for 10 users over 24 hours is not unusual. As gnattu mentioned, the NORMAL behavior is that the client gracefully ends the watch sessions and Jellyfin IMMEDIATELY cleans up the transcode files. If the client is not terminating the sessions gracefully then Jellyfin does not clean up immediate and this cleanup job will come around once a day to remove them. What would cause a client to NOT gracefully end a session? Could be several reasons and it varies from client to client. If you could provide your full server log via pastebin, we might be able to narrow down the cause. RE: Disk full because of transcode directory - Duvel - 2024-10-04 Its back this morning. Hopefully I thought to check my disks before my server was KO. Its at 94% again, so it grew up of 25TB, and its excatly the same issue: there are 13285 files in the transcode directory (while I deleted everything yesterday same time) Code: tom@www-prd:/opt/jellyfin/cache/transcodes$ ls | wc -l The files seems to be created in several bursts at exactly the same time, the most recent was one hour ago. Very little subset of what happened at 06:38: -rw-r--r-- 1 jellyfin jellyfin 7542252 Oct 4 06:38 fef670eab219f77bcc87a3001ef1f4a4554.mp4 -rw-r--r-- 1 jellyfin jellyfin 181356 Oct 4 06:38 fef670eab219f77bcc87a3001ef1f4a4555.mp4 -rw-r--r-- 1 jellyfin jellyfin 1859096 Oct 4 06:38 fef670eab219f77bcc87a3001ef1f4a455.mp4 -rw-r--r-- 1 jellyfin jellyfin 3366230 Oct 4 06:38 fef670eab219f77bcc87a3001ef1f4a456.mp4 -rw-r--r-- 1 jellyfin jellyfin 1276070 Oct 4 06:38 fef670eab219f77bcc87a3001ef1f4a457.mp4 -rw-r--r-- 1 jellyfin jellyfin 2309881 Oct 4 06:38 fef670eab219f77bcc87a3001ef1f4a458.mp4 -rw-r--r-- 1 jellyfin jellyfin 1320000 Oct 4 06:38 fef670eab219f77bcc87a3001ef1f4a459.mp4 -rw-r--r-- 1 jellyfin jellyfin 3082858 Oct 4 06:38 fef670eab219f77bcc87a3001ef1f4a45.mp4 -rw-r--r-- 1 jellyfin jellyfin 3578340 Oct 4 06:38 fef670eab219f77bcc87a3001ef1f4a460.mp4 -rw-r--r-- 1 jellyfin jellyfin 1883282 Oct 4 06:38 fef670eab219f77bcc87a3001ef1f4a461.mp4 -rw-r--r-- 1 jellyfin jellyfin 3421939 Oct 4 06:38 fef670eab219f77bcc87a3001ef1f4a462.mp4 -rw-r--r-- 1 jellyfin jellyfin 1901723 Oct 4 06:38 fef670eab219f77bcc87a3001ef1f4a463.mp4 -rw-r--r-- 1 jellyfin jellyfin 3810550 Oct 4 06:38 fef670eab219f77bcc87a3001ef1f4a464.mp4 -rw-r--r-- 1 jellyfin jellyfin 2183319 Oct 4 06:38 fef670eab219f77bcc87a3001ef1f4a465.mp4 -rw-r--r-- 1 jellyfin jellyfin 5222503 Oct 4 06:38 fef670eab219f77bcc87a3001ef1f4a466.mp4 -rw-r--r-- 1 jellyfin jellyfin 2288383 Oct 4 06:38 fef670eab219f77bcc87a3001ef1f4a467.mp4 -rw-r--r-- 1 jellyfin jellyfin 4677151 Oct 4 06:38 fef670eab219f77bcc87a3001ef1f4a468.mp4 -rw-r--r-- 1 jellyfin jellyfin 2444315 Oct 4 06:38 fef670eab219f77bcc87a3001ef1f4a469.mp4 -rw-r--r-- 1 jellyfin jellyfin 8441371 Oct 4 06:38 fef670eab219f77bcc87a3001ef1f4a46.mp4 -rw-r--r-- 1 jellyfin jellyfin 4027794 Oct 4 06:38 fef670eab219f77bcc87a3001ef1f4a470.mp4 -rw-r--r-- 1 jellyfin jellyfin 1808794 Oct 4 06:38 fef670eab219f77bcc87a3001ef1f4a471.mp4 -rw-r--r-- 1 jellyfin jellyfin 2707219 Oct 4 06:38 fef670eab219f77bcc87a3001ef1f4a472.mp4 -rw-r--r-- 1 jellyfin jellyfin 1527641 Oct 4 06:38 fef670eab219f77bcc87a3001ef1f4a473.mp4 -rw-r--r-- 1 jellyfin jellyfin 3060941 Oct 4 06:38 fef670eab219f77bcc87a3001ef1f4a474.mp4 -rw-r--r-- 1 jellyfin jellyfin 1807027 Oct 4 06:38 fef670eab219f77bcc87a3001ef1f4a475.mp4 -rw-r--r-- 1 jellyfin jellyfin 4163836 Oct 4 06:38 fef670eab219f77bcc87a3001ef1f4a476.mp4 -rw-r--r-- 1 jellyfin jellyfin 1846293 Oct 4 06:38 fef670eab219f77bcc87a3001ef1f4a477.mp4 -rw-r--r-- 1 jellyfin jellyfin 3731164 Oct 4 06:38 fef670eab219f77bcc87a3001ef1f4a478.mp4 -rw-r--r-- 1 jellyfin jellyfin 1712534 Oct 4 06:38 fef670eab219f77bcc87a3001ef1f4a479.mp4 -rw-r--r-- 1 jellyfin jellyfin 4066813 Oct 4 06:38 fef670eab219f77bcc87a3001ef1f4a47.mp4 -rw-r--r-- 1 jellyfin jellyfin 2998610 Oct 4 06:38 fef670eab219f77bcc87a3001ef1f4a480.mp4 -rw-r--r-- 1 jellyfin jellyfin 1075912 Oct 4 06:38 fef670eab219f77bcc87a3001ef1f4a481.mp4 -rw-r--r-- 1 jellyfin jellyfin 3042324 Oct 4 06:38 fef670eab219f77bcc87a3001ef1f4a482.mp4 -rw-r--r-- 1 jellyfin jellyfin 1467281 Oct 4 06:38 fef670eab219f77bcc87a3001ef1f4a483.mp4 -rw-r--r-- 1 jellyfin jellyfin 2915908 Oct 4 06:38 fef670eab219f77bcc87a3001ef1f4a484.mp4 -rw-r--r-- 1 jellyfin jellyfin 2178227 Oct 4 06:38 fef670eab219f77bcc87a3001ef1f4a485.mp4 -rw-r--r-- 1 jellyfin jellyfin 4653001 Oct 4 06:38 fef670eab219f77bcc87a3001ef1f4a486.mp4 -rw-r--r-- 1 jellyfin jellyfin 2180141 Oct 4 06:38 fef670eab219f77bcc87a3001ef1f4a487.mp4 -rw-r--r-- 1 jellyfin jellyfin 3384733 Oct 4 06:38 fef670eab219f77bcc87a3001ef1f4a488.mp4 -rw-r--r-- 1 jellyfin jellyfin 2828708 Oct 4 06:38 fef670eab219f77bcc87a3001ef1f4a489.mp4 -rw-r--r-- 1 jellyfin jellyfin 8932967 Oct 4 06:38 fef670eab219f77bcc87a3001ef1f4a48.mp4 -rw-r--r-- 1 jellyfin jellyfin 2192415 Oct 4 06:38 fef670eab219f77bcc87a3001ef1f4a490.mp4 -rw-r--r-- 1 jellyfin jellyfin 4685762 Oct 4 06:38 fef670eab219f77bcc87a3001ef1f4a491.mp4 -rw-r--r-- 1 jellyfin jellyfin 2533681 Oct 4 06:38 fef670eab219f77bcc87a3001ef1f4a492.mp4 -rw-r--r-- 1 jellyfin jellyfin 3927124 Oct 4 06:38 fef670eab219f77bcc87a3001ef1f4a493.mp4 -rw-r--r-- 1 jellyfin jellyfin 1908308 Oct 4 06:38 fef670eab219f77bcc87a3001ef1f4a494.mp4 -rw-r--r-- 1 jellyfin jellyfin 3408503 Oct 4 06:38 fef670eab219f77bcc87a3001ef1f4a495.mp4 -rw-r--r-- 1 jellyfin jellyfin 1675484 Oct 4 06:38 fef670eab219f77bcc87a3001ef1f4a496.mp4 -rw-r--r-- 1 jellyfin jellyfin 4292158 Oct 4 06:38 fef670eab219f77bcc87a3001ef1f4a497.mp4 -rw-r--r-- 1 jellyfin jellyfin 2040123 Oct 4 06:38 fef670eab219f77bcc87a3001ef1f4a498.mp4 -rw-r--r-- 1 jellyfin jellyfin 4392426 Oct 4 06:38 fef670eab219f77bcc87a3001ef1f4a499.mp4 -rw-r--r-- 1 jellyfin jellyfin 6957379 Oct 4 06:38 fef670eab219f77bcc87a3001ef1f4a49.mp4 -rw-r--r-- 1 jellyfin jellyfin 32307 Oct 4 06:38 fef670eab219f77bcc87a3001ef1f4a4.m3u8 I have 4 or 5 other bursts from yesterday around 21:00 and then after. I didnt check who was there and what they did. But I currently have one friend streaming a movie from an Ipad with Safari, which might be the "culprit" (but IMO the real culprit is the server) Stream: Transcode Video: Transcode (h264 h264 1920) Audio: Transcode (eac3 eac3) Product: Jellyfin Web Device: Safari iPad Not exactly an "exotic" client. 2 other streams are on pause, probably since yesterday night, and both were not transcoding (direct play on a Nokia box and Jellyfin Web Firefox) I notice also that the number of "normal logs" files (ie non-transcode-logs) is abnormally high for my server. I should normally have one log file per day. Code: tom@www-prd:/var/log/jellyfin$ ll -t There are really too much different log files for paste bin. I attach a zip of the last dozen of them here. Note that I just extracted them now and did not took any measure yet like deleting transcode files or restarting the server. Thank for looking in to it ! Something important to add : I did not change anything to my Jellyfin config for like 2 weeks when I upgrated the container to the latest Jellyfin version. And I never had this issue in the past. This bug appeared from nowhere suddenly. RE: Disk full because of transcode directory - Efficient_Good_5784 - 2024-10-04 Have you tried watching a few of the transcoded clips so you can try to identify which files are being placed there? Also, looks like you have the playback reporting plugin. Try disabling it and see if it solves your issue. RE: Disk full because of transcode directory - Duvel - 2024-10-04 I am monitoring the transcode folder. 2 friends have come back within the last hour and they were both transcoding and are both on pause right now I cleaned up the transcode directory 45 minutes ago. And there's already 1871 files in it. For 2 streams in less than 1 hour... is that possible? Video: Transcode (h264 h264 1920) Audio: Transcode (ac3 ac3) Product: Jellyfin Web Device: Safari iPhone Video: Transcode (h264 h264 1920) Audio: Transcode (eac3 eac3) Product: Jellyfin Web Device: Safari iPad And I've got the 2 movies that they were watching. What do you want me to do exactly @Efficient_Good_5784 ? I also disabled the playback reporting pluggin and restarted the server, and got this: Code: [09:00:56] [INF] Skipping disabled plugin 17.0.0.0 of LDAP-Auth while I had disabled it first, looks weird RE: Disk full because of transcode directory - Efficient_Good_5784 - 2024-10-04 (2024-10-04, 07:05 AM)Duvel Wrote: I am monitoring the transcode folder. 2 friends have come back within the last hour and they were both transcoding and are both on pause right nowAs was pointed to you earlier in this thread, yes that is possible. Each transcode is broken down roughly into small 6-second segments. Note that they can still be shorter intervals too. If we do the math for a transcode of a simple 1.5 hour long movie: 1.5 hours * 3600 seconds = 5400 seconds 5400 seconds / 6 seconds = 900 segments Multiply that example by 2 and you get 1800 segments. As for the plugin, it states it disabled itself so all should be good. Try running it like that and see if the transcodes remove themselves after a client exits the stream. RE: Disk full because of transcode directory - Duvel - 2024-10-11 After one week, no more issue so I guess it is solved. I have uninstalled the playback reporting pluggin, and it was not an easy task.... the thing appeared 5 times in my pluggin list, disable/remove had no effect on some of them and I had to restart the server multiple times and retry removal to get rid of everything. Since then, no issue ! Not sure if it was really the problem, but I havent changed anything else and my users neither. Thanks! |