SOLVED: Can no longer reach jellyfin after reboot

[nonprofitparrot]

I rebooted my machine today and noticed that I couldn't connect to jellyfin. After some investigation, I realized that my media directory/disk had not properly mounted and fixed the mounting issue. I thought that would fix jellyfin but after hours of debugging I can't seem to figure out why it won't respond. I've tried CURLing jellyfin, it responds 503.

Please let me know if you have any ideas!

Code:

[18:53:26] [INF] [1] Main: Jellyfin version: 10.9.11

[18:53:26] [INF] [1] Main: Environment Variables: ["[JELLYFIN_LOG_DIR, /config/log]", "[JELLYFIN_WEB_DIR, /usr/share/jellyfin/web]", "[JELLYFIN_CONFIG_DIR, /config]", "[JELLYFIN_CACHE_DIR, /config/cache]", "[JELLYFIN_DATA_DIR, /config/data]"]
[18:53:26] [INF] [1] Main: Arguments: ["/usr/lib/jellyfin/bin/jellyfin.dll", "--ffmpeg=/usr/lib/jellyfin-ffmpeg/ffmpeg"]
[18:53:26] [INF] [1] Main: Operating system: Ubuntu 24.04.1 LTS
[18:53:26] [INF] [1] Main: Architecture: X64
[18:53:26] [INF] [1] Main: 64-Bit Process: True
[18:53:26] [INF] [1] Main: User Interactive: True
[18:53:26] [INF] [1] Main: Processor count: 16
[18:53:26] [INF] [1] Main: Program data path: /config/data
[18:53:26] [INF] [1] Main: Log directory path: /config/log
[18:53:26] [INF] [1] Main: Config directory path: /config
[18:53:26] [INF] [1] Main: Cache path: /config/cache
[18:53:26] [INF] [1] Main: Web resources path: /usr/share/jellyfin/web
[18:53:26] [INF] [1] Main: Application directory: /usr/lib/jellyfin/bin/
[18:53:26] [INF] [1] Emby.Server.Implementations.AppBase.BaseConfigurationManager: Setting cache path: /config/cache
[18:53:26] [INF] [1] Emby.Server.Implementations.ApplicationHost: Loading assemblies
[18:53:26] [INF] [1] Jellyfin.Networking.Manager.NetworkManager: Defined LAN subnets: ["::1/128", "fe80::/10", "fc00::/7", "127.0.0.1/8", "10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16"]
[18:53:26] [INF] [1] Jellyfin.Networking.Manager.NetworkManager: Defined LAN exclusions: []
[18:53:26] [INF] [1] Jellyfin.Networking.Manager.NetworkManager: Used LAN subnets: ["::1/128", "fe80::/10", "fc00::/7", "127.0.0.1/8", "10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16"]
[18:53:26] [INF] [1] Jellyfin.Networking.Manager.NetworkManager: Filtered interface addresses: ["127.0.0.1", "10.88.0.16", "::1", "fe80::84b1:6ff:fe4c:88b5%2"]
[18:53:26] [INF] [1] Jellyfin.Networking.Manager.NetworkManager: Bind Addresses ["::"]
[18:53:26] [INF] [1] Jellyfin.Networking.Manager.NetworkManager: Remote IP filter is Allowlist
[18:53:26] [INF] [1] Jellyfin.Networking.Manager.NetworkManager: Filtered subnets: ["100.64.0.0/10", "100.1.16.211/32"]
[18:53:26] [INF] [1] Microsoft.EntityFrameworkCore.Database.Command: Executed DbCommand (4ms) [Parameters=[], CommandType='Text', CommandTimeout='30']
SELECT COUNT(*) FROM "sqlite_master" WHERE "name" = '__EFMigrationsHistory' AND "type" = 'table';
[18:53:26] [INF] [1] Microsoft.EntityFrameworkCore.Database.Command: Executed DbCommand (0ms) [Parameters=[], CommandType='Text', CommandTimeout='30']
SELECT "MigrationId", "ProductVersion"
FROM "__EFMigrationsHistory"
ORDER BY "MigrationId";
[18:53:26] [INF] [1] Microsoft.EntityFrameworkCore.Database.Command: Executed DbCommand (0ms) [Parameters=[], CommandType='Text', CommandTimeout='30']
SELECT "u"."Id", "u"."AudioLanguagePreference", "u"."AuthenticationProviderId", "u"."CastReceiverId", "u"."DisplayCollectionsView", "u"."DisplayMissingEpisodes", "u"."EnableAutoLogin", "u"."EnableLocalPassword", "u"."EnableNextEpisodeAutoPlay", "u"."EnableUserPreferenceAccess", "u"."HidePlayedInLatest", "u"."InternalId", "u"."InvalidLoginAttemptCount", "u"."LastActivityDate", "u"."LastLoginDate", "u"."LoginAttemptsBeforeLockout", "u"."MaxActiveSessions", "u"."MaxParentalAgeRating", "u"."MustUpdatePassword", "u"."Password", "u"."PasswordResetProviderId", "u"."PlayDefaultAudioTrack", "u"."RememberAudioSelections", "u"."RememberSubtitleSelections", "u"."RemoteClientBitrateLimit", "u"."RowVersion", "u"."SubtitleLanguagePreference", "u"."SubtitleMode", "u"."SyncPlayAccess", "u"."Username", "i"."Id", "i"."LastModified", "i"."Path", "i"."UserId"
FROM "Users" AS "u"
LEFT JOIN "ImageInfos" AS "i" ON "u"."Id" = "i"."UserId"
ORDER BY "u"."Id", "i"."Id"
[18:53:27] [INF] [1] Microsoft.EntityFrameworkCore.Database.Command: Executed DbCommand (0ms) [Parameters=[], CommandType='Text', CommandTimeout='30']
SELECT "p"."Id", "p"."Kind", "p"."Permission_Permissions_Guid", "p"."RowVersion", "p"."UserId", "p"."Value", "u"."Id", "i"."Id"
FROM "Users" AS "u"
LEFT JOIN "ImageInfos" AS "i" ON "u"."Id" = "i"."UserId"
INNER JOIN "Permissions" AS "p" ON "u"."Id" = "p"."UserId"
ORDER BY "u"."Id", "i"."Id"
[18:53:27] [INF] [1] Microsoft.EntityFrameworkCore.Database.Command: Executed DbCommand (0ms) [Parameters=[], CommandType='Text', CommandTimeout='30']
SELECT "p"."Id", "p"."Kind", "p"."Preference_Preferences_Guid", "p"."RowVersion", "p"."UserId", "p"."Value", "u"."Id", "i"."Id"
FROM "Users" AS "u"
LEFT JOIN "ImageInfos" AS "i" ON "u"."Id" = "i"."UserId"
INNER JOIN "Preferences" AS "p" ON "u"."Id" = "p"."UserId"
ORDER BY "u"."Id", "i"."Id"
[18:53:27] [INF] [1] Microsoft.EntityFrameworkCore.Database.Command: Executed DbCommand (0ms) [Parameters=[], CommandType='Text', CommandTimeout='30']
SELECT "a"."Id", "a"."DayOfWeek", "a"."EndHour", "a"."StartHour", "a"."UserId", "u"."Id", "i"."Id"
FROM "Users" AS "u"
LEFT JOIN "ImageInfos" AS "i" ON "u"."Id" = "i"."UserId"
INNER JOIN "AccessSchedules" AS "a" ON "u"."Id" = "a"."UserId"
ORDER BY "u"."Id", "i"."Id"
[18:53:27] [INF] [1] Emby.Server.Implementations.Plugins.PluginManager: Loaded plugin: TMDb 10.9.11.0
[18:53:27] [INF] [1] Emby.Server.Implementations.Plugins.PluginManager: Loaded plugin: Studio Images 10.9.11.0
[18:53:27] [INF] [1] Emby.Server.Implementations.Plugins.PluginManager: Loaded plugin: OMDb 10.9.11.0
[18:53:27] [INF] [1] Emby.Server.Implementations.Plugins.PluginManager: Loaded plugin: MusicBrainz 10.9.11.0
[18:53:27] [INF] [1] Emby.Server.Implementations.Plugins.PluginManager: Loaded plugin: AudioDB 10.9.11.0
[18:53:27] [INF] [1] Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager: User profile is available. Using '/config/.aspnet/DataProtection-Keys' as key repository; keys will not be encrypted at rest.
[18:53:27] [INF] [1] Microsoft.EntityFrameworkCore.Database.Command: Executed DbCommand (0ms) [Parameters=[], CommandType='Text', CommandTimeout='30']
SELECT "d"."Id", "d"."AccessToken", "d"."AppName", "d"."AppVersion", "d"."DateCreated", "d"."DateLastActivity", "d"."DateModified", "d"."DeviceId", "d"."DeviceName", "d"."IsActive", "d"."UserId"
FROM "Devices" AS "d"
ORDER BY "d"."Id"
[18:53:27] [INF] [1] Microsoft.EntityFrameworkCore.Database.Command: Executed DbCommand (0ms) [Parameters=[], CommandType='Text', CommandTimeout='30']
SELECT "d"."Id", "d"."CustomName", "d"."DeviceId"
FROM "DeviceOptions" AS "d"
ORDER BY "d"."Id"
[18:53:27] [INF] [1] Main: Kestrel is listening on all interfaces
[18:53:27] [WRN] [1] Microsoft.AspNetCore.StaticFiles.StaticFileMiddleware: The WebRootPath was not found: /run/s6-rc:s6-rc-init:fEDfcJ/servicedirs/svc-jellyfin/wwwroot. Static files may be unavailable.
[18:53:27] [INF] [1] Microsoft.Hosting.Lifetime: Now listening on: http://[::]:8096
[18:53:27] [INF] [1] Microsoft.Hosting.Lifetime: Application started. Press Ctrl+C to shut down.
[18:53:27] [INF] [1] Microsoft.Hosting.Lifetime: Hosting environment: Production
[18:53:27] [INF] [1] Microsoft.Hosting.Lifetime: Content root path: /run/s6-rc:s6-rc-init:fEDfcJ/servicedirs/svc-jellyfin
[18:53:27] [INF] [1] Emby.Server.Implementations.ApplicationHost: Running startup tasks
[18:53:27] [INF] [11] Emby.Server.Implementations.IO.LibraryMonitor: Watching directory /data/media/movies
[18:53:27] [INF] [10] Emby.Server.Implementations.IO.LibraryMonitor: Watching directory /data/media/tv
[18:53:27] [INF] [1] Emby.Server.Implementations.ScheduledTasks.TaskManager: Daily trigger for Generate Trickplay Images set to fire at 2024-10-15 03:00:00.000 +00:00, which is 08:06:32.5231851 from now.
[18:53:27] [INF] [1] Emby.Server.Implementations.ScheduledTasks.TaskManager: Daily trigger for Extract Chapter Images set to fire at 2024-10-15 02:00:00.000 +00:00, which is 07:06:32.5214962 from now.
[18:53:27] [INF] [1] MediaBrowser.MediaEncoding.Encoder.MediaEncoder: Found ffmpeg version 6.0.1
[18:53:27] [INF] [1] MediaBrowser.MediaEncoding.Encoder.MediaEncoder: Available decoders: ["libdav1d", "av1", "av1_cuvid", "av1_qsv", "h264", "h264_qsv", "h264_cuvid", "hevc", "hevc_qsv", "hevc_cuvid", "mpeg2video", "mpeg2_qsv", "mpeg2_cuvid", "mpeg4", "mpeg4_cuvid", "msmpeg4", "vc1_qsv", "vc1_cuvid", "vp8", "libvpx", "vp8_cuvid", "vp8_qsv", "vp9", "libvpx-vp9", "vp9_cuvid", "vp9_qsv", "aac", "ac3", "dca", "flac", "mp3", "truehd"]
[18:53:27] [INF] [1] MediaBrowser.MediaEncoding.Encoder.MediaEncoder: Available encoders: ["libsvtav1", "av1_nvenc", "av1_qsv", "av1_amf", "av1_vaapi", "libx264", "h264_amf", "h264_nvenc", "h264_qsv", "h264_v4l2m2m", "h264_vaapi", "libx265", "hevc_amf", "hevc_nvenc", "hevc_qsv", "hevc_vaapi", "mjpeg_qsv", "mjpeg_vaapi", "mpeg4", "msmpeg4", "libvpx", "libvpx-vp9", "aac", "libfdk_aac", "ac3", "alac", "dca", "flac", "libmp3lame", "libopus", "truehd", "libvorbis", "srt"]
[18:53:27] [INF] [1] MediaBrowser.MediaEncoding.Encoder.MediaEncoder: Available filters: ["deinterlace_qsv", "deinterlace_vaapi", "hwupload_cuda", "hwupload_vaapi", "libplacebo", "overlay_opencl", "overlay_qsv", "overlay_vaapi", "overlay_vulkan", "overlay_cuda", "procamp_vaapi", "scale_cuda", "scale_opencl", "scale_qsv", "scale_vaapi", "scale_vulkan", "tonemap_cuda", "tonemap_opencl", "tonemap_vaapi", "vpp_qsv", "yadif_cuda", "zscale", "alphasrc"]
[18:53:27] [INF] [1] MediaBrowser.MediaEncoding.Encoder.MediaEncoder: Available hwaccel types: ["cuda", "vaapi", "qsv", "drm", "opencl", "vulkan"]
[18:53:27] [INF] [1] MediaBrowser.MediaEncoding.Encoder.MediaEncoder: VAAPI device /dev/dri/renderD128 is AMD GPU
[18:53:27] [INF] [1] MediaBrowser.MediaEncoding.Encoder.MediaEncoder: VAAPI device /dev/dri/renderD128 supports Vulkan DRM interop
[18:53:27] [INF] [1] MediaBrowser.MediaEncoding.Encoder.MediaEncoder: FFmpeg: /usr/lib/jellyfin-ffmpeg/ffmpeg
[18:53:27] [INF] [1] Emby.Server.Implementations.ApplicationHost: ServerId: 9bdc0356f9b64a3cb0a0a71783f6b70c
[18:53:27] [INF] [1] Emby.Server.Implementations.ApplicationHost: Core startup complete
[18:53:27] [INF] [1] Main: Startup complete 0:00:01.8914438
[18:53:27] [INF] [11] Microsoft.AspNetCore.Hosting.Diagnostics: Request starting HTTP/1.1 GET http://localhost:8096/health - null null
[18:53:27] [INF] [11] Microsoft.AspNetCore.Routing.EndpointMiddleware: Executing endpoint 'Health checks'
[18:53:27] [INF] [8] Microsoft.AspNetCore.Routing.EndpointMiddleware: Executed endpoint 'Health checks'
[18:53:27] [INF] [8] Microsoft.AspNetCore.Hosting.Diagnostics: Request finished HTTP/1.1 GET http://localhost:8096/health - 200 null text/plain 85.5522ms
[ls.io-init] done.
[18:53:30] [INF] [10] System.Net.Http.HttpClient.Default.LogicalHandler: Start processing HTTP request GET https://repo.jellyfin.org/files/plugin/manifest.json
[18:53:30] [INF] [10] System.Net.Http.HttpClient.Default.ClientHandler: Sending HTTP request GET https://repo.jellyfin.org/files/plugin/manifest.json
[18:53:30] [INF] [11] Emby.Server.Implementations.ScheduledTasks.TaskManager: Clean up collections and playlists Completed after 0 minute(s) and 0 seconds
[18:53:35] [INF] [11] System.Net.Http.HttpClient.Default.ClientHandler: Received HTTP response headers after 5423.7502ms - 200
[18:53:35] [INF] [11] System.Net.Http.HttpClient.Default.LogicalHandler: End processing HTTP request after 5428.3518ms - 200
[18:53:35] [INF] [10] Emby.Server.Implementations.ScheduledTasks.TaskManager: Update Plugins Completed after 0 minute(s) and 5 seconds
[19:03:33] [INF] [23] Microsoft.AspNetCore.Hosting.Diagnostics: Request starting HTTP/1.1 GET http://100.1.16.211:8096/ - null null
[19:03:33] [INF] [23] Microsoft.AspNetCore.Hosting.Diagnostics: Request finished HTTP/1.1 GET http://100.1.16.211:8096/ - 302 0 null 0.9943ms
[19:03:42] [INF] [3] Microsoft.AspNetCore.Hosting.Diagnostics: Request starting HTTP/1.1 GET http://100.1.16.211:8096/ - null null
[19:03:42] [INF] [3] Microsoft.AspNetCore.Hosting.Diagnostics: Request finished HTTP/1.1 GET http://100.1.16.211:8096/ - 302 0 null 0.4813ms
[19:03:48] [INF] [3] Microsoft.AspNetCore.Hosting.Diagnostics: Request starting HTTP/1.1 GET http://100.1.16.211:8096/web/ - null null
[19:03:48] [INF] [3] Microsoft.AspNetCore.StaticFiles.StaticFileMiddleware: Sending file. Request path: '/index.html'. Physical path: '/usr/share/jellyfin/web/index.html'
[19:03:48] [INF] [3] Microsoft.AspNetCore.Hosting.Diagnostics: Request finished HTTP/1.1 GET http://100.1.16.211:8096/web/ - 200 null text/html 8.3406ms
[19:03:49] [INF] [23] Microsoft.AspNetCore.Hosting.Diagnostics: Request starting HTTP/1.1 GET http://100.1.16.211:8096/f5bbb798cb2c65908633.png - null null
[19:03:49] [INF] [23] Microsoft.AspNetCore.Hosting.Diagnostics: Request finished HTTP/1.1 GET http://100.1.16.211:8096/f5bbb798cb2c65908633.png - 503 0 null 6.4616ms
[19:03:49] [INF] [23] Microsoft.AspNetCore.Hosting.Diagnostics: Request starting HTTP/1.1 GET http://100.1.16.211:8096/bc8d51405ec040305a87.ico - null null
[19:03:49] [INF] [23] Microsoft.AspNetCore.Hosting.Diagnostics: Request finished HTTP/1.1 GET http://100.1.16.211:8096/bc8d51405ec040305a87.ico - 503 0 null 2.6915ms
[19:03:49] [INF] [3] Microsoft.AspNetCore.Hosting.Diagnostics: Request starting HTTP/1.1 GET http://100.1.16.211:8096/favicon.ico - null null
[19:03:49] [INF] [3] Microsoft.AspNetCore.Hosting.Diagnostics: Request finished HTTP/1.1 GET http://100.1.16.211:8096/favicon.ico - 53 0 null 0.8139ms
[19:03:50] [INF] [21] Microsoft.AspNetCore.Hosting.Diagnostics: Request starting HTTP/1.1 GET http://100.1.16.211:8096/ - null null
[19:03:50] [INF] [21] Microsoft.AspNetCore.Hosting.Diagnostics: Request finished HTTP/1.1 GET http://100.1.16.211:8096/ - 302 0 null 0.4316ms
[19:04:08] [INF] [23] Microsoft.AspNetCore.Hosting.Diagnostics: Request starting HTTP/1.1 GET http://localhost:8096/health - null null
[19:04:08] [INF] [23] Microsoft.AspNetCore.Hosting.Diagnostics: Request finished HTTP/1.1 GET http://localhost:8096/health - 503 0 null 0.7947ms

[TheDreadPirate]

Are you using a reverse proxy?

And are you using a VPN?

[nonprofitparrot]

Are you using a reverse proxy?

And are you using a VPN?

Hi @TheDreadPirate! I'm not using a reverse proxy or a VPN. This is all on my LAN. CURL makes me think networking isn't the issue because I can hit the server.

[TheDreadPirate]

Can you share the contents of /config/config/network.xml?

Also, you say this is all on the same LAN, but the request that was logged was from a remote IP. And this remote IP is on the allow list, but why would the request be coming from your WAN IP? Are you using a domain to access your Jellyfin?

[nonprofitparrot]

To elaborate, jellyfin is running in a podman container on a nixos server attached to my router. I was CURLing from the same box, and attempting to access the web ui from my laptop via the WAN.

This is my network configuration:

Code:

<?xml version="1.0" encoding="utf-8"?>
<NetworkConfiguration xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
  <BaseUrl />
  <EnableHttps>false</EnableHttps>
  <RequireHttps>false</RequireHttps>
  <InternalHttpPort>8096</InternalHttpPort>
  <InternalHttpsPort>8920</InternalHttpsPort>
  <PublicHttpPort>8096</PublicHttpPort>
  <PublicHttpsPort>8920</PublicHttpsPort>
  <AutoDiscovery>true</AutoDiscovery>
  <EnableUPnP>false</EnableUPnP>
  <EnableIPv4>true</EnableIPv4>
  <EnableIPv6>true</EnableIPv6>
  <EnableRemoteAccess>true</EnableRemoteAccess>
  <LocalNetworkSubnets />
  <LocalNetworkAddresses />
  <KnownProxies />
  <IgnoreVirtualInterfaces>true</IgnoreVirtualInterfaces>
  <VirtualInterfaceNames>
    <string>veth</string>
  </VirtualInterfaceNames>
  <EnablePublishedServerUriByRequest>false</EnablePublishedServerUriByRequest>
  <PublishedServerUriBySubnet />
  <RemoteIPFilter>
    <string>100.64.0.0/10</string>
    <string>100.1.16.211</string>
  </RemoteIPFilter>
  <IsRemoteIPFilterBlacklist>false</IsRemoteIPFilterBlacklist>
</NetworkConfiguration>

Thanks a lot for taking a look!

[TheDreadPirate]

For testing, can you replace this section

Code:

<RemoteIPFilter>
    <string>100.64.0.0/10</string>
    <string>100.1.16.211</string>
  </RemoteIPFilter>

With this

Code:

<RemoteIPFilter />

[nonprofitparrot]

@TheDreadPirate that was it!! I must have configured it to only allow access from my two main devices some time ago... I'm not sure why that stopped working today. Thanks a bunch!