SOLVED: "Startup probe failed" failed after updating TrueNAS & Jellyfin

[jymotion]

I was running Jellyfin on TrueNAS Scale as recently as this morning without issue. I saw today that both had updates available (they were both only 1 release behind or so), so I updated TrueNAS Scale, then Jellyfin. Now when I try to start Jellyfin back up again, it gets stuck in DEPLOYING with the following events:

Code:

2023-10-14 17:13:24
Startup probe failed: Get "http://172.16.0.63:8096/health": dial tcp 172.16.0.63:8096: connect: connection refused
2023-10-14 17:13:14
Started container jellyfin
2023-10-14 17:13:13
Created container jellyfin
2023-10-14 17:13:12
Container image "jellyfin/jellyfin:10.8.11" already present on machine
2023-10-14 17:13:11
Created container jellyfin-init-01-permissions
2023-10-14 17:13:11
Started container jellyfin-init-01-permissions
2023-10-14 17:13:10
Add eth0 [172.16.0.63/16] from ix-net
2023-10-14 17:13:10
Container image "bash:4.4.23" already present on machine
2023-10-14 17:13:09
Scaled up replica set jellyfin-844545c6b5 to 1
2023-10-14 17:13:09
Created pod: jellyfin-844545c6b5-nqfxw
2023-10-14 17:13:09
Successfully assigned ix-jellyfin/jellyfin-844545c6b5-nqfxw to ix-truenas

I don't recognize the IP address it's trying to reach. When I watch content, I'm connecting at http://192.168.0.86:30013, but maybe that's irrelevant. When I look at my Kubernetes Settings, I see:

Code:

Cluster CIDR: 172.16.0.0/16
Service CIDR: 172.17.0.0/16
Cluster DNS IP: 172.17.0.10

I'm not familiar with what this is, but it seems similar to what is listed in the Events log. Sorry if this is a stupid issue. Would appreciate any help.

[bitmap]

Those are Docker assigned IP addresses for your bridge or other network (i.e., docker networking, not a different physical or external connection). The health check on the container itself is failing, that's a probe of whether Jellyfin is up (can it access the home page) and the answer is no. So this log doesn't have enough info, you'll probably need to find the docker logs.

[jymotion]

Those are Docker assigned IP addresses for your bridge or other network (i.e., docker networking, not a different physical or external connection). The health check on the container itself is failing, that's a probe of whether Jellyfin is up (can it access the home page) and the answer is no. So this log doesn't have enough info, you'll probably need to find the docker logs.

Thanks for taking the time to reply. Here are the logs for the Jellyfin container, if I'm looking in the right place:

Code:

2023-10-14 22:40:56.660221+00:00[22:40:56] [INF] [1] Main: Jellyfin version: 10.8.11
2023-10-14 22:40:56.664405+00:00[22:40:56] [INF] [1] Main: Environment Variables: ["[JELLYFIN_WEB_DIR, /jellyfin/jellyfin-web]", "[JELLYFIN_DATA_DIR, /config]", "[JELLYFIN_CACHE_DIR, /cache]", "[DOTNET_SYSTEM_GLOBALIZATION_INVARIANT, 1]", "[JELLYFIN_CONFIG_DIR, /config/config]", "[JELLYFIN_LOG_DIR, /config/log]", "[JELLYFIN_FFMPEG, /usr/lib/jellyfin-ffmpeg/ffmpeg]"]
2023-10-14 22:40:56.664423+00:00[22:40:56] [INF] [1] Main: Arguments: ["/jellyfin/jellyfin.dll"]
2023-10-14 22:40:56.664868+00:00[22:40:56] [INF] [1] Main: Operating system: Linux
2023-10-14 22:40:56.665224+00:00[22:40:56] [INF] [1] Main: Architecture: X64
2023-10-14 22:40:56.665236+00:00[22:40:56] [INF] [1] Main: 64-Bit Process: True
2023-10-14 22:40:56.665242+00:00[22:40:56] [INF] [1] Main: User Interactive: True
2023-10-14 22:40:56.665244+00:00[22:40:56] [INF] [1] Main: Processor count: 4
2023-10-14 22:40:56.665246+00:00[22:40:56] [INF] [1] Main: Program data path: /config
2023-10-14 22:40:56.665251+00:00[22:40:56] [INF] [1] Main: Web resources path: /jellyfin/jellyfin-web
2023-10-14 22:40:56.665253+00:00[22:40:56] [INF] [1] Main: Application directory: /jellyfin/
2023-10-14 22:40:56.777117+00:00[22:40:56] [INF] [1] Emby.Server.Implementations.AppBase.BaseConfigurationManager: Setting cache path: /cache
2023-10-14 22:40:56.784680+00:00[22:40:56] [INF] [1] Emby.Server.Implementations.ApplicationHost: Loading assemblies
2023-10-14 22:40:56.820797+00:00[22:40:56] [INF] [1] Jellyfin.Networking.Manager.NetworkManager: Defined LAN addresses : [10.0.0.0/8,172.16.0.0/12,192.168.0.0/16]
2023-10-14 22:40:56.820819+00:00[22:40:56] [INF] [1] Jellyfin.Networking.Manager.NetworkManager: Defined LAN exclusions : []
2023-10-14 22:40:56.821204+00:00[22:40:56] [INF] [1] Jellyfin.Networking.Manager.NetworkManager: Using LAN addresses: [10.0.0.0/8,172.16.0.0/12,192.168.0.0/16]
2023-10-14 22:40:56.826073+00:00[22:40:56] [INF] [1] Jellyfin.Networking.Manager.NetworkManager: Using bind addresses: []
2023-10-14 22:40:56.826090+00:00[22:40:56] [INF] [1] Jellyfin.Networking.Manager.NetworkManager: Using bind exclusions: []
2023-10-14 22:40:59.159642+00:00[22:40:59] [INF] [1] Emby.Server.Implementations.Plugins.PluginManager: Loaded plugin: TMDb 10.8.11.0
2023-10-14 22:40:59.159841+00:00[22:40:59] [INF] [1] Emby.Server.Implementations.Plugins.PluginManager: Loaded plugin: Studio Images 10.8.11.0
2023-10-14 22:40:59.159978+00:00[22:40:59] [INF] [1] Emby.Server.Implementations.Plugins.PluginManager: Loaded plugin: OMDb 10.8.11.0
2023-10-14 22:40:59.160089+00:00[22:40:59] [INF] [1] Emby.Server.Implementations.Plugins.PluginManager: Loaded plugin: MusicBrainz 10.8.11.0
2023-10-14 22:40:59.160201+00:00[22:40:59] [INF] [1] Emby.Server.Implementations.Plugins.PluginManager: Loaded plugin: AudioDB 10.8.11.0
2023-10-14 22:40:59.191524+00:00[22:40:59] [INF] [1] Main: Kestrel listening on Any IP4 Address
2023-10-14 22:40:59.214603+00:00[22:40:59] [WRN] [1] Microsoft.AspNetCore.DataProtection.Repositories.EphemeralXmlRepository: Using an in-memory repository. Keys will not be persisted to storage.
2023-10-14 22:40:59.214733+00:00[22:40:59] [WRN] [1] Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager: Neither user profile nor HKLM registry available. Using an ephemeral key repository. Protected data will be unavailable when application exits.
2023-10-14 22:40:59.310675+00:00[22:40:59] [WRN] [1] Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager: No XML encryptor configured. Key {24a784b9-84ec-4910-8988-ce6337382ab6} may be persisted to storage in unencrypted form.
2023-10-14 22:40:59.320976+00:00[22:40:59] [INF] [1] Emby.Server.Implementations.ApplicationHost: Running startup tasks
2023-10-14 22:40:59.341576+00:00[22:40:59] [INF] [1] Emby.Server.Implementations.ScheduledTasks.TaskManager: Daily trigger for Extract Chapter Images set to fire at 2023-10-15 02:00:00.000 +00:00, which is 03:19:00.6587561 from now.
2023-10-14 22:40:59.366815+00:00[22:40:59] [INF] [1] MediaBrowser.MediaEncoding.Encoder.MediaEncoder: Found ffmpeg version 5.1.3
2023-10-14 22:40:59.385421+00:00[22:40:59] [INF] [1] MediaBrowser.MediaEncoding.Encoder.MediaEncoder: Available decoders: ["libdav1d", "av1", "av1_cuvid", "av1_qsv", "h264", "h264_qsv", "h264_cuvid", "hevc", "hevc_qsv", "hevc_cuvid", "mpeg2video", "mpeg2_qsv", "mpeg2_cuvid", "mpeg4", "mpeg4_cuvid", "msmpeg4", "vc1_qsv", "vc1_cuvid", "vp8", "libvpx", "vp8_cuvid", "vp8_qsv", "vp9", "libvpx-vp9", "vp9_cuvid", "vp9_qsv", "aac", "ac3", "dca", "flac", "mp3", "truehd"]
2023-10-14 22:40:59.399530+00:00[22:40:59] [INF] [1] MediaBrowser.MediaEncoding.Encoder.MediaEncoder: Available encoders: ["libx264", "h264_amf", "h264_nvenc", "h264_qsv", "h264_v4l2m2m", "h264_vaapi", "libx265", "hevc_amf", "hevc_nvenc", "hevc_qsv", "hevc_vaapi", "mpeg4", "msmpeg4", "libvpx", "libvpx-vp9", "aac", "libfdk_aac", "ac3", "dca", "flac", "libmp3lame", "libopus", "truehd", "libvorbis", "srt"]
2023-10-14 22:40:59.412146+00:00[22:40:59] [INF] [1] MediaBrowser.MediaEncoding.Encoder.MediaEncoder: Available filters: ["deinterlace_qsv", "deinterlace_vaapi", "hwupload_cuda", "hwupload_vaapi", "overlay_opencl", "overlay_qsv", "overlay_vaapi", "overlay_cuda", "procamp_vaapi", "scale_cuda", "scale_opencl", "scale_qsv", "scale_vaapi", "tonemap_cuda", "tonemap_opencl", "tonemap_vaapi", "vpp_qsv", "yadif_cuda", "zscale", "alphasrc"]
2023-10-14 22:40:59.480607+00:00[22:40:59] [INF] [1] MediaBrowser.MediaEncoding.Encoder.MediaEncoder: Available hwaccel types: ["cuda", "vaapi", "qsv", "drm", "opencl", "vulkan"]
2023-10-14 22:40:59.541272+00:00[22:40:59] [INF] [1] MediaBrowser.MediaEncoding.Encoder.MediaEncoder: FFmpeg: /usr/lib/jellyfin-ffmpeg/ffmpeg
2023-10-14 22:40:59.541661+00:00[22:40:59] [INF] [1] Emby.Server.Implementations.ApplicationHost: ServerId: bbe9eb1e2a1d435d8264e25e4a277300
2023-10-14 22:40:59.587435+00:00[22:40:59] [INF] [1] Emby.Server.Implementations.ApplicationHost: Executed all pre-startup entry points in 0:00:00.0412321
2023-10-14 22:40:59.587455+00:00[22:40:59] [INF] [1] Emby.Server.Implementations.ApplicationHost: Core startup complete
2023-10-14 22:40:59.674527+00:00[22:40:59] [INF] [10] Emby.Server.Implementations.IO.LibraryMonitor: Watching directory /mymedia/movies
2023-10-14 22:40:59.676538+00:00[22:40:59] [INF] [1] Emby.Server.Implementations.ApplicationHost: Executed all post-startup entry points in 0:00:00.0889959
2023-10-14 22:40:59.676567+00:00[22:40:59] [INF] [1] Main: Startup complete 0:00:03.1057256
2023-10-14 22:40:59.685823+00:00[22:40:59] [INF] [8] Emby.Server.Implementations.IO.LibraryMonitor: Watching directory /mymedia/tv
2023-10-14 22:41:02.347471+00:00[22:41:02] [INF] [8] Emby.Server.Implementations.ScheduledTasks.TaskManager: StartupTrigger fired for task: Update Plugins
2023-10-14 22:41:02.349110+00:00[22:41:02] [INF] [8] Emby.Server.Implementations.ScheduledTasks.TaskManager: Queuing task PluginUpdateTask
2023-10-14 22:41:02.351769+00:00[22:41:02] [INF] [10] Emby.Server.Implementations.ScheduledTasks.TaskManager: Executing Update Plugins
2023-10-14 22:41:03.308365+00:00[22:41:03] [INF] [8] Emby.Server.Implementations.ScheduledTasks.TaskManager: Update Plugins Completed after 0 minute(s) and 0 seconds
2023-10-14 22:41:03.312433+00:00[22:41:03] [INF] [8] Emby.Server.Implementations.ScheduledTasks.TaskManager: ExecuteQueuedTasks

The only lines labeled as warnings are:

Code:

2023-10-14 22:40:59.214603+00:00[22:40:59] [WRN] [1] Microsoft.AspNetCore.DataProtection.Repositories.EphemeralXmlRepository: Using an in-memory repository. Keys will not be persisted to storage.
2023-10-14 22:40:59.214733+00:00[22:40:59] [WRN] [1] Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager: Neither user profile nor HKLM registry available. Using an ephemeral key repository. Protected data will be unavailable when application exits.
2023-10-14 22:40:59.310675+00:00[22:40:59] [WRN] [1] Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager: No XML encryptor configured. Key {24a784b9-84ec-4910-8988-ce6337382ab6} may be persisted to storage in unencrypted form.

Is this relevant or is there somewhere else I should look?

[jymotion]

Tried updating TrueNAS Scale to the latest RC branch and got it working. Can't honestly say I understand how. Sorry to anyone who might read this in the future.

[BourbonDoc]

Having the exact same problem in TrueNAS 24.04. Only thing that works is entirely deleting and then reinstalling Jellyfin. Which is a huge PIA.