Jellyfin Forum
SOLVED: new ubuntu update today leaves server unreachable - Printable Version

+- Jellyfin Forum (https://forum.jellyfin.org)
+-- Forum: Support (https://forum.jellyfin.org/f-support)
+--- Forum: Troubleshooting (https://forum.jellyfin.org/f-troubleshooting)
+--- Thread: SOLVED: new ubuntu update today leaves server unreachable (/t-solved-new-ubuntu-update-today-leaves-server-unreachable)

new ubuntu update today leaves server unreachable - farotherside - 2024-11-16

hey all,

this morning updated my jellyfin instance using 'apt', pulling the most recent update from the 'https://repo.jellyfin.org/ubuntu jammy' repo. after updating my web interface would no longer work, so I started digging.

here's my system details:
Code:
Distributor ID: Ubuntu
Description: Ubuntu 24.04.1 LTS
Release: 24.04
Codename: noble

Linux xwing 6.8.0-44-generic #44-Ubuntu SMP PREEMPT_DYNAMIC Tue Aug 13 13:35:26 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Package: jellyfin
Version: 10.10.2+ubu2204

looks like the most recent update broke things in two different ways... the first way was what appears to be a well known old bug, where the newly-installed config(?) references /usr/lib/jellyfin/bin/jellyfin-web (which does not exist) instead of /usr/share/jellyfin/web (which does exist).

after creating a symlink to redirect to the real directory (sudo ln -s /usr/share/jellyfin/web /usr/lib/jellyfin/bin/jellyfin-web), the server at least tried to start, but now it is failing because apparently a migration process is trying to do something and crashing...?

Code:
[2024-11-17 10:32:04.020 +13:00] [INF] Loaded plugin: "TMDb" "10.10.2.0"
[2024-11-17 10:32:04.021 +13:00] [INF] Loaded plugin: "Studio Images" "10.10.2.0"
[2024-11-17 10:32:04.022 +13:00] [INF] Loaded plugin: "OMDb" "10.10.2.0"
[2024-11-17 10:32:04.062 +13:00] [INF] Loaded plugin: "MusicBrainz" "10.10.2.0"
[2024-11-17 10:32:04.063 +13:00] [INF] Loaded plugin: "AudioDB" "10.10.2.0"
[2024-11-17 10:32:04.118 +13:00] [INF] Applying migration '"RemoveDuplicatePlaylistChildren"'
[2024-11-17 10:32:04.558 +13:00] [ERR] Could not apply migration '"RemoveDuplicatePlaylistChildren"'
System.UnauthorizedAccessException: Access to the path '/media/MEDIA/Audio/INCOMING/Coldplay - Essentials (2024) [FLAC] 88/Coldplay - Essentials.m3u' is denied.
---> System.IO.IOException: Permission denied
  --- End of inner exception stack trace ---
  at Interop.ThrowExceptionForIoErrno(ErrorInfo errorInfo, String path, Boolean isDirError)
  at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String path, OpenFlags flags, Int32 mode, Boolean failForSymlink, Boolean& wasSymlink, Func`4 createOpenException)
  at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize, UnixFileMode openPermissions, Int64& fileLength, UnixFileMode& filePermissions, Boolean failForSymlink, Boolean& wasSymlink, Func`4 createOpenException)
  at System.IO.File.OpenHandle(String path, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize)
  at System.IO.File.WriteToFile(String path, FileMode mode, String contents, Encoding encoding)
  at Emby.Server.Implementations.Playlists.PlaylistManager.SavePlaylistFile(Playlist item)
  at Jellyfin.Server.Migrations.Routines.RemoveDuplicatePlaylistChildren.Perform()
  at Jellyfin.Server.Migrations.MigrationRunner.PerformMigrations(IMigrationRoutine[] migrations, MigrationOptions migrationOptions, Action`1 saveConfiguration, ILogger logger)
[2024-11-17 10:32:04.563 +13:00] [FTL] Error while starting server
System.UnauthorizedAccessException: Access to the path '/media/MEDIA/Audio/INCOMING/Coldplay - Essentials (2024) [FLAC] 88/Coldplay - Essentials.m3u' is denied.
---> System.IO.IOException: Permission denied
  --- End of inner exception stack trace ---
  at Interop.ThrowExceptionForIoErrno(ErrorInfo errorInfo, String path, Boolean isDirError)
  at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String path, OpenFlags flags, Int32 mode, Boolean failForSymlink, Boolean& wasSymlink, Func`4 createOpenException)
  at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize, UnixFileMode openPermissions, Int64& fileLength, UnixFileMode& filePermissions, Boolean failForSymlink, Boolean& wasSymlink, Func`4 createOpenException)
  at System.IO.File.OpenHandle(String path, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize)
  at System.IO.File.WriteToFile(String path, FileMode mode, String contents, Encoding encoding)
  at Emby.Server.Implementations.Playlists.PlaylistManager.SavePlaylistFile(Playlist item)
  at Jellyfin.Server.Migrations.Routines.RemoveDuplicatePlaylistChildren.Perform()
  at Jellyfin.Server.Migrations.MigrationRunner.PerformMigrations(IMigrationRoutine[] migrations, MigrationOptions migrationOptions, Action`1 saveConfiguration, ILogger logger)
  at Jellyfin.Server.Migrations.MigrationRunner.Run(CoreAppHost host, ILoggerFactory loggerFactory)
  at Jellyfin.Server.Program.StartServer(IServerApplicationPaths appPaths, StartupOptions options, IConfiguration startupConfig)
[2024-11-17 10:32:04.563 +13:00] [INF] Running query planner optimizations in the database... This might take a while
[2024-11-17 10:32:04.586 +13:00] [INF] Disposing "CoreAppHost"
[2024-11-17 10:32:04.586 +13:00] [INF] Disposing "MusicBrainzArtistProvider"
[2024-11-17 10:32:04.586 +13:00] [INF] Disposing "MusicBrainzAlbumProvider"
[2024-11-17 10:32:04.587 +13:00] [INF] Disposing "PluginManager"

(I know, I know. I don't even really like Coldplay.)

the jellyfin user does have read access to that directory and to that specific .M3U file (I even tested to be sure), but it does not have write access which made me a little suspicious; I don't see any reason for a mediaserver to have unsupervised write access to the media files. I'm willing to hear reasoning though.

testing by removing that file completely from the media collection makes no difference; the server still crashes with the same message, despite (or because of?) the fact that there's no file there anymore. seems like the sort of thing that should fail gracefully instead of throwing a system exception.

thoughts?

RE: new ubuntu update today leaves server unreachable - TheDreadPirate - 2024-11-16

Can you share the entire Jellyfin log via pastebin?

RE: new ubuntu update today leaves server unreachable - TheDreadPirate - 2024-11-16

Looks like someone else has the same issue. I'm setting up my test instance to try to replicate.

https://github.com/jellyfin/jellyfin/issues/13048

RE: new ubuntu update today leaves server unreachable - farotherside - 2024-11-16

sure. here's the whole log, freshly generated, including the 30 or so lines before the previous cutoff.

Code:
[2024-11-17 12:00:30.867 +13:00] [INF] Jellyfin version: "10.10.2"
[2024-11-17 12:00:30.897 +13:00] [INF] Environment Variables: ["[JELLYFIN_ARGS, $JELLYFIN_WEB_OPT $JELLYFIN_FFMPEG_OPT $JELLYFIN_SERVICE_OPT $JELLYFIN_NOWEBAPP_OPT $JELLFIN_ADDITIONAL_OPTS --datadir $JELLYFIN_DATA_DIR --configdir $JELLYFIN_CONFIG_DIR --logdir $JELLYFIN_LOG_DIR --cachedir $JELLYFIN_CACHE_DIR]", "[JELLYFIN_FFMPEG_OPT, --ffmpeg=/usr/lib/jellyfin-ffmpeg/ffmpeg]", "[JELLYFIN_ADDITIONAL_OPTS, ]", "[JELLYFIN_USER, jellyfin]", "[JELLYFIN_DATA_DIR, /var/lib/jellyfin]", "[JELLYFIN_LOG_DIR, /var/log/jellyfin]", "[JELLYFIN_CONFIG_DIR, /etc/jellyfin]", "[JELLYFIN_CACHE_DIR, /var/cache/jellyfin]", "[JELLYFIN_WEB_OPT, --webdir=/usr/share/jellyfin/web]"]
[2024-11-17 12:00:30.899 +13:00] [INF] Arguments: ["/usr/lib/jellyfin/bin/jellyfin.dll", "--webdir=/usr/share/jellyfin/web", "--ffmpeg=/usr/lib/jellyfin-ffmpeg/ffmpeg"]
[2024-11-17 12:00:30.899 +13:00] [INF] Operating system: "Ubuntu 24.04.1 LTS"
[2024-11-17 12:00:30.900 +13:00] [INF] Architecture: X64
[2024-11-17 12:00:30.900 +13:00] [INF] 64-Bit Process: True
[2024-11-17 12:00:30.900 +13:00] [INF] User Interactive: True
[2024-11-17 12:00:30.900 +13:00] [INF] Processor count: 4
[2024-11-17 12:00:30.901 +13:00] [INF] Program data path: "/var/lib/jellyfin"
[2024-11-17 12:00:30.901 +13:00] [INF] Log directory path: "/var/log/jellyfin"
[2024-11-17 12:00:30.901 +13:00] [INF] Config directory path: "/etc/jellyfin"
[2024-11-17 12:00:30.901 +13:00] [INF] Cache path: "/var/cache/jellyfin"
[2024-11-17 12:00:30.901 +13:00] [INF] Temp directory path: "/tmp/jellyfin"
[2024-11-17 12:00:30.901 +13:00] [INF] Web resources path: "/usr/share/jellyfin/web"
[2024-11-17 12:00:30.901 +13:00] [INF] Application directory: "/usr/lib/jellyfin/bin/"
[2024-11-17 12:00:31.040 +13:00] [INF] Setting cache path: "/var/cache/jellyfin"
[2024-11-17 12:00:31.111 +13:00] [INF] Loading assemblies
[2024-11-17 12:00:31.121 +13:00] [INF] Loaded assembly "TvMaze.Api.Client, Version=0.1.72.0, Culture=neutral, PublicKeyToken=null" from "/var/lib/jellyfin/plugins/TVmaze_11.0.0.0/TvMaze.Api.Client.dll"
[2024-11-17 12:00:31.123 +13:00] [INF] Loaded assembly "Jellyfin.Plugin.TvMaze, Version=11.0.0.0, Culture=neutral, PublicKeyToken=null" from "/var/lib/jellyfin/plugins/TVmaze_11.0.0.0/Jellyfin.Plugin.TvMaze.dll"
[2024-11-17 12:00:31.126 +13:00] [INF] Loaded assembly "Flurl.Http, Version=3.0.1.0, Culture=neutral, PublicKeyToken=null" from "/var/lib/jellyfin/plugins/TVmaze_11.0.0.0/Flurl.Http.dll"
[2024-11-17 12:00:31.135 +13:00] [INF] Loaded assembly "Polly, Version=7.0.0.0, Culture=neutral, PublicKeyToken=c8a3ffc3f8f825cc" from "/var/lib/jellyfin/plugins/TVmaze_11.0.0.0/Polly.dll"
[2024-11-17 12:00:31.137 +13:00] [INF] Loaded assembly "Flurl, Version=3.0.1.0, Culture=neutral, PublicKeyToken=null" from "/var/lib/jellyfin/plugins/TVmaze_11.0.0.0/Flurl.dll"
[2024-11-17 12:00:31.217 +13:00] [INF] Defined LAN subnets: ["127.0.0.1/8", "10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16"]
[2024-11-17 12:00:31.217 +13:00] [INF] Defined LAN exclusions: []
[2024-11-17 12:00:31.218 +13:00] [INF] Used LAN subnets: ["127.0.0.1/8", "10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16"]
[2024-11-17 12:00:31.219 +13:00] [INF] Filtered interface addresses: ["127.0.0.1", "192.168.1.218"]
[2024-11-17 12:00:31.219 +13:00] [INF] Bind Addresses ["0.0.0.0"]
[2024-11-17 12:00:31.219 +13:00] [INF] Remote IP filter is "Allowlist"
[2024-11-17 12:00:31.219 +13:00] [INF] Filtered subnets: []
[2024-11-17 12:00:48.482 +13:00] [INF] Loaded plugin: "TVmaze" "11.0.0.0"
[2024-11-17 12:00:48.484 +13:00] [INF] Loaded plugin: "TMDb" "10.10.2.0"
[2024-11-17 12:00:48.484 +13:00] [INF] Loaded plugin: "Studio Images" "10.10.2.0"
[2024-11-17 12:00:48.485 +13:00] [INF] Loaded plugin: "OMDb" "10.10.2.0"
[2024-11-17 12:00:48.520 +13:00] [INF] Loaded plugin: "MusicBrainz" "10.10.2.0"
[2024-11-17 12:00:48.521 +13:00] [INF] Loaded plugin: "AudioDB" "10.10.2.0"
[2024-11-17 12:00:48.577 +13:00] [INF] Applying migration '"RemoveDuplicatePlaylistChildren"'
[2024-11-17 12:00:48.995 +13:00] [ERR] Could not apply migration '"RemoveDuplicatePlaylistChildren"'
System.UnauthorizedAccessException: Access to the path '/media/MEDIA/Audio/INCOMING/Coldplay - Essentials (2024) [FLAC] 88/Coldplay - Essentials.m3u' is denied.
---> System.IO.IOException: Permission denied
   --- End of inner exception stack trace ---
   at Interop.ThrowExceptionForIoErrno(ErrorInfo errorInfo, String path, Boolean isDirError)
   at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String path, OpenFlags flags, Int32 mode, Boolean failForSymlink, Boolean& wasSymlink, Func`4 createOpenException)
   at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize, UnixFileMode openPermissions, Int64& fileLength, UnixFileMode& filePermissions, Boolean failForSymlink, Boolean& wasSymlink, Func`4 createOpenException)
   at System.IO.File.OpenHandle(String path, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize)
   at System.IO.File.WriteToFile(String path, FileMode mode, String contents, Encoding encoding)
   at Emby.Server.Implementations.Playlists.PlaylistManager.SavePlaylistFile(Playlist item)
   at Jellyfin.Server.Migrations.Routines.RemoveDuplicatePlaylistChildren.Perform()
   at Jellyfin.Server.Migrations.MigrationRunner.PerformMigrations(IMigrationRoutine[] migrations, MigrationOptions migrationOptions, Action`1 saveConfiguration, ILogger logger)
[2024-11-17 12:00:49.000 +13:00] [FTL] Error while starting server
System.UnauthorizedAccessException: Access to the path '/media/MEDIA/Audio/INCOMING/Coldplay - Essentials (2024) [FLAC] 88/Coldplay - Essentials.m3u' is denied.
---> System.IO.IOException: Permission denied
   --- End of inner exception stack trace ---
   at Interop.ThrowExceptionForIoErrno(ErrorInfo errorInfo, String path, Boolean isDirError)
   at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String path, OpenFlags flags, Int32 mode, Boolean failForSymlink, Boolean& wasSymlink, Func`4 createOpenException)
   at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize, UnixFileMode openPermissions, Int64& fileLength, UnixFileMode& filePermissions, Boolean failForSymlink, Boolean& wasSymlink, Func`4 createOpenException)
   at System.IO.File.OpenHandle(String path, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize)
   at System.IO.File.WriteToFile(String path, FileMode mode, String contents, Encoding encoding)
   at Emby.Server.Implementations.Playlists.PlaylistManager.SavePlaylistFile(Playlist item)
   at Jellyfin.Server.Migrations.Routines.RemoveDuplicatePlaylistChildren.Perform()
   at Jellyfin.Server.Migrations.MigrationRunner.PerformMigrations(IMigrationRoutine[] migrations, MigrationOptions migrationOptions, Action`1 saveConfiguration, ILogger logger)
   at Jellyfin.Server.Migrations.MigrationRunner.Run(CoreAppHost host, ILoggerFactory loggerFactory)
   at Jellyfin.Server.Program.StartServer(IServerApplicationPaths appPaths, StartupOptions options, IConfiguration startupConfig)
[2024-11-17 12:00:49.001 +13:00] [INF] Running query planner optimizations in the database... This might take a while
[2024-11-17 12:00:49.022 +13:00] [INF] Disposing "CoreAppHost"
[2024-11-17 12:00:49.022 +13:00] [INF] Disposing "MusicBrainzArtistProvider"
[2024-11-17 12:00:49.022 +13:00] [INF] Disposing "MusicBrainzAlbumProvider"
[2024-11-17 12:00:49.022 +13:00] [INF] Disposing "PluginManager"


RE: new ubuntu update today leaves server unreachable - TheDreadPirate - 2024-11-16

I was able to reproduce. Both with the docker volume mounted as read only and by setting the individual m3u file's permissions so that the container user had read only permissions.

RE: new ubuntu update today leaves server unreachable - farotherside - 2024-11-17

thanks. temporary fix of enabling write access to the drive got things running, if others are reading this.

reposting my github comments here though because I think there's a couple of larger issues implied by this bug:

Quote:just as another couple of datapoints, I was able to get jellyfin running by adjusting the umask in fstab to temporarily allow the jellyfin user to write to the drive - but even though this updating of m3u files is harmless and well-intentioned, I still think it's a highly problematic stance to take for a project to quietly modify files on a mediaserver without first asking permission from (or even notifying!!) the owner. sketchy behavior, guys.

secondly, the server was not able to come up without further help, because the migration process was referencing missing/inaccessible M3U files on a drive that I had removed last week from the mediaserver. I had removed all mention of that drive from the various libraries, but apparently the "playlists" library/database still had entries for those M3Us.

honestly I don't ever use M3Us and I don't use the 'playlists' library feature, but I have neither come across anything in the settings to configure it nor disable it. at the very least it would seem that whatever database playlists are in is not being updated by nightly library scans.