SOLVED: Jellyfin failing to start due to Kestrel failing to start

[csaavedra]

I have Jellyfin 10.10.6 in a bare Nano Pi with Armbian, I only use it in the LAN. Yesterday the OOM killer killed jellyfin due to high memory usage, and after that it has refused to start with the logs showing that Kestrel fails to start. Nothing has changed in my configuration, so I have no idea what's going on here.

First, here are excerpts from jellyfin starting and failing:

Code:

Mar 02 20:49:46 nanopi systemd[1]: Started jellyfin.service - Jellyfin Media Server.
Mar 02 20:49:49 nanopi jellyfin[3453]: [20:49:48] [INF] Jellyfin version: 10.10.6
Mar 02 20:49:49 nanopi jellyfin[3453]: [20:49:49] [INF] Environment Variables: ["[JELLYFIN_ADDITIONAL_OPTS, ]", "[JELLYFIN_LOG_DIR, /var/log/jellyfin]", "[JELLYFIN_CONFIG_DIR, /etc/jellyfin]", "[JELLYFIN_FFMPEG>
Mar 02 20:49:49 nanopi jellyfin[3453]: [20:49:49] [INF] Arguments: ["/usr/lib/jellyfin/bin/jellyfin.dll", "--webdir=/usr/share/jellyfin/web", "--ffmpeg=/usr/lib/jellyfin-ffmpeg/ffmpeg"]
Mar 02 20:49:49 nanopi jellyfin[3453]: [20:49:49] [INF] Operating system: Armbian 25.2.1 bookworm
Mar 02 20:49:49 nanopi jellyfin[3453]: [20:49:49] [INF] Architecture: Arm64
Mar 02 20:49:49 nanopi jellyfin[3453]: [20:49:49] [INF] 64-Bit Process: True
Mar 02 20:49:49 nanopi jellyfin[3453]: [20:49:49] [INF] User Interactive: True
Mar 02 20:49:49 nanopi jellyfin[3453]: [20:49:49] [INF] Processor count: 4
Mar 02 20:49:49 nanopi jellyfin[3453]: [20:49:49] [INF] Program data path: /var/lib/jellyfin
Mar 02 20:49:49 nanopi jellyfin[3453]: [20:49:49] [INF] Log directory path: /var/log/jellyfin
Mar 02 20:49:49 nanopi jellyfin[3453]: [20:49:49] [INF] Config directory path: /etc/jellyfin
Mar 02 20:49:49 nanopi jellyfin[3453]: [20:49:49] [INF] Cache path: /var/cache/jellyfin
Mar 02 20:49:49 nanopi jellyfin[3453]: [20:49:49] [INF] Temp directory path: /tmp/jellyfin
Mar 02 20:49:49 nanopi jellyfin[3453]: [20:49:49] [INF] Web resources path: /usr/share/jellyfin/web
Mar 02 20:49:49 nanopi jellyfin[3453]: [20:49:49] [INF] Application directory: /usr/lib/jellyfin/bin/
Mar 02 20:49:50 nanopi jellyfin[3453]: [20:49:50] [INF] Setting cache path: /var/cache/jellyfin
Mar 02 20:49:50 nanopi jellyfin[3453]: [20:49:50] [INF] Loading assemblies
...
Mar 02 20:49:52 nanopi jellyfin[3453]: [20:49:52] [INF] Defined LAN subnets: ["127.0.0.1/8", "10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16"]
Mar 02 20:49:52 nanopi jellyfin[3453]: [20:49:52] [INF] Defined LAN exclusions: []
Mar 02 20:49:52 nanopi jellyfin[3453]: [20:49:52] [INF] Used LAN subnets: ["127.0.0.1/8", "10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16"]
Mar 02 20:49:52 nanopi jellyfin[3453]: [20:49:52] [INF] Filtered interface addresses: ["127.0.0.1", "192.168.42.136"]
Mar 02 20:49:52 nanopi jellyfin[3453]: [20:49:52] [INF] Bind Addresses ["0.0.0.0"]
Mar 02 20:49:52 nanopi jellyfin[3453]: [20:49:52] [INF] Remote IP filter is Allowlist
Mar 02 20:49:52 nanopi jellyfin[3453]: [20:49:52] [INF] Filtered subnets: []
...
Mar 02 20:50:44 nanopi jellyfin[3453]: [20:50:44] [INF] Kestrel is listening on 0.0.0.0
...
Mar 02 20:50:51 nanopi jellyfin[3453]: [20:50:51] [WRN] The WebRootPath was not found: /var/lib/jellyfin/wwwroot. Static files may be unavailable.
Mar 02 20:50:51 nanopi jellyfin[3453]: [20:50:51] [ERR] Hosting failed to start
Mar 02 20:50:51 nanopi jellyfin[3453]: System.InvalidOperationException: Server has already started.
Mar 02 20:50:51 nanopi jellyfin[3453]:    at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServerImpl.StartAsync[TContext](IHttpApplication`1 application, CancellationToken cancellationToken)
Mar 02 20:50:51 nanopi jellyfin[3453]:    at Microsoft.AspNetCore.Hosting.GenericWebHostService.StartAsync(CancellationToken cancellationToken)
Mar 02 20:50:51 nanopi jellyfin[3453]:    at Microsoft.Extensions.Hosting.Internal.Host.<StartAsync>b__15_1(IHostedService service, CancellationToken token)
Mar 02 20:50:51 nanopi jellyfin[3453]:    at Microsoft.Extensions.Hosting.Internal.Host.ForeachService[T](IEnumerable`1 services, CancellationToken token, Boolean concurrent, Boolean abortOnFirstException, List>
Mar 02 20:50:51 nanopi jellyfin[3453]: [20:50:51] [ERR] Kestrel failed to start! This is most likely due to an invalid address or port bind - correct your bind configuration in network.xml and try again
Mar 02 20:50:51 nanopi jellyfin[3453]: [20:50:51] [FTL] Error while starting server
Mar 02 20:50:51 nanopi jellyfin[3453]: System.InvalidOperationException: Server has already started.
Mar 02 20:50:51 nanopi jellyfin[3453]:    at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServerImpl.StartAsync[TContext](IHttpApplication`1 application, CancellationToken cancellationToken)
Mar 02 20:50:51 nanopi jellyfin[3453]:    at Microsoft.AspNetCore.Hosting.GenericWebHostService.StartAsync(CancellationToken cancellationToken)
Mar 02 20:50:51 nanopi jellyfin[3453]:    at Microsoft.Extensions.Hosting.Internal.Host.<StartAsync>b__15_1(IHostedService service, CancellationToken token)
Mar 02 20:50:51 nanopi jellyfin[3453]:    at Microsoft.Extensions.Hosting.Internal.Host.ForeachService[T](IEnumerable`1 services, CancellationToken token, Boolean concurrent, Boolean abortOnFirstException, List>
Mar 02 20:50:51 nanopi jellyfin[3453]:    at Microsoft.Extensions.Hosting.Internal.Host.StartAsync(CancellationToken cancellationToken)
Mar 02 20:50:51 nanopi jellyfin[3453]:    at Jellyfin.Server.Program.StartServer(IServerApplicationPaths appPaths, StartupOptions options, IConfiguration startupConfig)
Mar 02 20:50:52 nanopi jellyfin[3453]: [20:50:52] [INF] Disposing CoreAppHost
Mar 02 20:50:52 nanopi jellyfin[3453]: [20:50:52] [INF] Disposing MusicBrainzArtistProvider
Mar 02 20:50:52 nanopi jellyfin[3453]: [20:50:52] [INF] Disposing MusicBrainzAlbumProvider
Mar 02 20:50:52 nanopi jellyfin[3453]: [20:50:52] [INF] Disposing CoverArtArchiveImageProvider
Mar 02 20:50:52 nanopi jellyfin[3453]: [20:50:52] [INF] Disposing KodiSyncQueuePlugin
Mar 02 20:50:52 nanopi jellyfin[3453]: [20:50:52] [INF] Disposing PluginManager
Mar 02 20:50:52 nanopi systemd[1]: jellyfin.service: Deactivated successfully.
Mar 02 20:50:52 nanopi systemd[1]: jellyfin.service: Consumed 45.489s CPU time.

Here are the contents of my network.xml:

Code:

<?xml version="1.0" encoding="utf-8"?>
<NetworkConfiguration xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
  <BaseUrl />
  <EnableHttps>false</EnableHttps>
  <RequireHttps>false</RequireHttps>
  <CertificatePath />
  <CertificatePassword />
  <InternalHttpPort>8096</InternalHttpPort>
  <InternalHttpsPort>8920</InternalHttpsPort>
  <PublicHttpPort>8096</PublicHttpPort>
  <PublicHttpsPort>8920</PublicHttpsPort>
  <AutoDiscovery>true</AutoDiscovery>
  <EnableUPnP>false</EnableUPnP>
  <EnableIPv4>true</EnableIPv4>
  <EnableIPv6>false</EnableIPv6>
  <EnableRemoteAccess>false</EnableRemoteAccess>
  <LocalNetworkSubnets />
  <LocalNetworkAddresses />
  <KnownProxies />
  <IgnoreVirtualInterfaces>true</IgnoreVirtualInterfaces>
  <VirtualInterfaceNames/>
  <EnablePublishedServerUriByRequest>false</EnablePublishedServerUriByRequest>
  <PublishedServerUriBySubnet />
  <RemoteIPFilter />
  <IsRemoteIPFilterBlacklist>false</IsRemoteIPFilterBlacklist>
</NetworkConfiguration>

And finally this is the network configuration in this device:

Code:

root@nanopi:/etc/jellyfin# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
      valid_lft forever preferred_lft forever
2: end0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 16:fb:00:b7:5f:74 brd ff:ff:ff:ff:ff:ff
    inet 192.168.42.136/24 metric 1024 brd 192.168.42.255 scope global dynamic end0
      valid_lft 73470sec preferred_lft 73470sec
    inet6 fe80::14fb:ff:feb7:5f74/64 scope link
      valid_lft forever preferred_lft forever

[minijackson]

I encountered this today, and it turned out to be a plugin. Through a process of trial and error, by manually moving directories in and out from ${datadir}/plugins/, I found out it was the ListenBrainz plugin. The problem was reported here: https://github.com/lyarenei/jellyfin-plu...issues/121 and was fixed in the latest version

[csaavedra]

Thanks for the reply! And to think that I left out the plugins initialization because I thought it was irrelevant. I'll remove the listenbrainz plugin.