SOLVED: can't start jellyfin after disk full

[kro]

Here is the full log as attached.

[TheDreadPirate]

Can you run this command and share the output? I'm hoping there is output different from what is in the jellyfin.log

Code:

sudo docker logs jellyfin

[kro]

crosspost

[TheDreadPirate]

I saw, but my post still stands. "docker logs" will also show any logs the container generates that isn't related to Jellyfin. But they may likely be identical.

You may just need to start from scratch. Running out of disk space is a death sentence for a database.

[kro]

the attachment is the log for this container. the exact thing that your command would return (and does, I've just ensured).

[TheDreadPirate]

Then it is quite likely your database is beyond rescue. You could try using your backup, but those are pretty old.

[kro]

funny update : my backup disk is dying and it's very likely I won't be able to restore jellyfin.

I wonder how databases still get corrupted as soon as storage get full in 2024. What happened in 70 years of computing ?

I have another backup that is one year old on a raid 5 array though. So to anybody reading this : we probably should always save triple backups of everything. or abort technical projects.

A big thank you anyway @TheDreadPirate for you time and help !

[Efficient_Good_5784]

I wonder how databases still get corrupted as soon as storage get full in 2024. What happened in 70 years of computing ?

Having no space to work with means no more space to store changes or new additions to the database.
At best, having the database become locked and the service frozen would save the database from breaking down.

[kro]

woot !

finally I could restore the 3 months old database backup and combine it with the recent metadata directory (I could not get from the backup disk anymore).

everything seems to work, I guess I just have lost 3 months of metadata and recently added users, but it's really better than I went to think.
For the record I also got the reason behind the lack of free space : I was taking snapshots (with rsnapshot) on a directory that was supposedly a mountpoint for an external support, except it was not mounted. So I've just created quickly a bunch of unnoticed content on my system part. I really should be more careful with that, and add condition for mounts in my scripts.
A lot of stress for luckily not much harm.
Now is the time to setup reliable automatic backups.

... and still : yes, I guess the system should abort before creating those 0 bit files everywhere ! It's such a basic and critical issue...

[Cremator]

I had the same issue, and I could fix it fairly easy. I am using lsio jellyfin docker.

1. Stop the bad jellyfin container:

Code:

docker stop jellyfin

2. Start a new jellyfin container:

Code:

docker run -d --name=new-jellyfin -e PUID=1000 -e PGUID=1000 -e TZ=Europe/Sofia -v /tmp/new-jellyfin:/config lscr.io/linuxserver/jellyfin:latest

3. Dump the new-jellyfin migrations table into a sql file:

Code:

sqlite3 /tmp/new-jellyfin/data/data/jellyfin.db
sqlite> .output /tmp/mytable_dump.sql
sqlite> .dump __EFMigrationsHistory
sqlite> .quit

4. Drop and recreate __EFMigrationsHistory in the bad jellyfin:

Code:

sqlite3 /var/lib/docker/volumes/bad_jellyfin/_data/data/data/jellyfin.db
sqlite> drop table __EFMigrationsHistory;
sqlite> .read /tmp/mytable_dump.sql
sqlite> .quit 

5. Copy migrations.xml from new to bad:

Code:

cp /tmp/new-jellyfin/migrations.xml /var/lib/docker/volumes/bad_jellyfin/_data/migrations.xml

6. Start the bad, hopefully now good jellyfin container:

Code:

docker start jellyfin