Inaccessible after upgrade from 10.8.13.0 to 10.9.11.0

[Riesling]

Thank you Pirate,
interesting: in proxmox GUI v6 is not configured for this container, but it was actually using it, as you noticed.
Switched it off, ping to Google is v4 now, no more v6 in ifconfig or logs.
But: no change, no access after reboot (v6 was probably also active before the upgrade).
Yes, no client connections logged at all - not even the ones from the redirects... Obviously no connections getting through. And that is exactly the problem

Cheers,
~R.

[TheDreadPirate]

Can you describe how your networking is setup? Anything unusual? Do you have any VLANs configured or something? Are your clients in the same subnet? Can your clients even ping your jellyfin LXC and get a response?

[Riesling]

yes, I have a 10.x.x.x net w. a 255.0.0.0 netmask - a bit BIG, but less numbers to type then w. the usual 192.168.x.x nets
Jelly is on the 10.3.3.3, router is 10.0.0.1, same subnet, no VLANS, all quite simple and straight forward.
I can ping the VE w. 0.276ms, I access it via ssh, all good.

But I don't think this is the source of the problem, as it worked perfectly fine until yesterday and all I did was an "apt dist-upgrade" in and (meanwhile several) reboots of the LXC - I bet it is some entry in the (new?) config files...

~R.

[TheDreadPirate]

Config files of what? Jellyfin? Jellyfin isn't receiving connections to drop or deny. Did the LXC firewall get reset or something? If you can ping and ssh, double check that the firewall is still allowing 8096 in.

Code:

sudo ufw status

Make sure it is enabled and re-add the firewall rule if it was cleared out.

Code:

sudo ufw enable
sudo ufw allow 8096

Also, for the record, you should be using "apt upgrade" instead of dist-upgrade for maintenance updates. Dist-upgrade can break dependencies.

Code:

upgrade
    upgrade is used to install the newest versions of all packages
    currently installed on the system from the sources enumerated in
    /etc/apt/sources.list. Packages currently installed with new
    versions available are retrieved and upgraded; under no
    circumstances are currently installed packages removed, or packages
    not already installed retrieved and installed. New versions of
    currently installed packages that cannot be upgraded without
    changing the install status of another package will be left at
    their current version. An update must be performed first so that
    apt-get knows that new versions of packages are available.

dist-upgrade
    dist-upgrade in addition to performing the function of upgrade,
    also intelligently handles changing dependencies with new versions
    of packages; apt-get has a "smart" conflict resolution system, and
    it will attempt to upgrade the most important packages at the
    expense of less important ones if necessary. So, dist-upgrade
    command may remove some packages. The /etc/apt/sources.list file
    contains a list of locations from which to retrieve desired package
    files. See also apt_preferences(5) for a mechanism for overriding
    the general settings for individual packages.

[Riesling]

Thanks Pirate,
yes, config files of Jellyfin, either new ones added, old ones left over or modified and not compatible w. the new version?
No firewall on this VE. As http://10.3.3.3:8096 is accessible (an redirects) port 8096 seems to be open.
Double checked from other VE in the same network:

Code:

root@sandbox:~# wget http://10.3.3.3:8096/
--2024-10-10 07:18:18--  http://10.3.3.3:8096/
Connecting to 10.3.3.3:8096... connected.
HTTP request sent, awaiting response... 302 Found
Location: web/ [following]
--2024-10-10 07:18:20--  http://10.3.3.3:8096/web/
Reusing existing connection to 10.3.3.3:8096.
HTTP request sent, awaiting response... 200 OK
Length: 9970 (9.7K) [text/html]
Saving to: ‘index.html’

index.html            100%[=========================>]  9.74K  --.-KB/s    in 0s     

2024-10-10 07:18:24 (96.4 MB/s) - ‘index.html’ saved [9970/9970]

root@sandbox:~#

[TheDreadPirate]

Let's experiment. Let's backup your current Jellyfin server files, purge all of Jellyfin, reinstall Jellyfin, test the fresh 10.9.11 server, then restore the old server and test it with fresh binaries.

On the Jellyfin LXC.

Code:

sudo systemctl stop jellyfin
cd /some/backup/directory  # Just find a directory to store your jellyfin backup in
mkdir -p etc var/lib var/cache
sudo rsync -a -p --progress /etc/jellyfin etc/
sudo rsync -a -p --progress /var/lib/jellyfin var/lib/
sudo rsync -a -p --progress /var/cache/jellyfin var/cache/
sudo apt purge jellyfin*
sudo rm -rf /etc/jellyfin /var/lib/jellyfin /var/cache/jellyfin /usr/lib/jellyfin /usr/lib/jellyfin-ffmpeg /usr/share/jellyfin
curl https://repo.jellyfin.org/install-debuntu.sh | sudo bash

Now you should have a fresh install to test with. Clear browser cache on web clients before connecting or use private/incognito windows.

Once you've verified you can reach it with other clients, let's restore your original server instance. The "--delete" at the end of the rsync commands below ensures that any new files that the test instance generated, that don't also exist in the original, are deleted so you don't have stray files.

Code:

sudo systemctl stop jellyfin
cd /etc/
sudo rsync -a -p --progress /some/backup/directory/etc/jellyfin .
cd /var/cache
sudo rsync -a -p --progress /some/backup/directory/var/cache/jellyfin . --delete
cd /var/lib
sudo rsync -a -p --progress /some/backup/directory/var/lib/jellyfin . --delete
sudo systemctl start jellyfin

[Riesling]

Thank you very much dear Pirate,
I'll try that over the weekend and get back.
Cheers,
~R.

[Riesling]

Thank you Pirate,

I did as you wrote*.

Result: same situation as previously described - server inaccessible. 

Freshly installed server worked fine and was perfectly accessible until/before the restore of the original instance:

After I restored the ~8GB etc, cache and lib, server is - yet again - inaccessible, showing again splash-screen -> selectserver.html -> "cannot connect, try later". 

I see "[ERR] [1] Main: The server is expected to host the web client, but the provided content directory is either invalid or empty: /usr/lib/jellyfin/bin/jellyfin-web. If you do not want to host the web client with the server, you may set the '--nowebclient' command line flag, or set'hostwebclient=false' in your config settings"

On " : / # jellyfin --nowebclient" on the server I get http://10.3.3.3:8096/api-docs/swagger/index.html in Browser which I have never seen before. Seems to be about the API. But at least I get a response, even though not the one expected...

In logs:

Code:

[WRN] An operation of type '"SqlOperation"' will be attempted while a rebuild of table '"ImageInfos"' is pending. The database may not be in an expected state. Review the SQL generated by this migration to help diagnose any failures. Consider moving these operations to a subsequent migration.
[WRN] An operation of type '"SqlOperation"' will be attempted while a rebuild of table '"Permissions"' is pending. The database may not be in an expected state. Review the SQL generated by this migration to help diagnose any failures. Consider moving these operations to a subsequent migration.
[WRN] An operation of type '"SqlOperation"' will be attempted while a rebuild of table '"Preferences"' is pending. The database may not be in an expected state. Review the SQL generated by this migration to help diagnose any failures. Consider moving these operations to a subsequent migration.
[WRN] An operation of type '"SqlOperation"' will be attempted while a rebuild of table '"Users"' is pending. The database may not be in an expected state. Review the SQL generated by this migration to help diagnose any failures. Consider moving these operations to a subsequent migration.
(each message shows twice)
[WRN] No XML encryptor configured. Key {654762f9-d002-4cd0-b5bf-031745238eec} may be persisted to storage in unencrypted form.
...and the 404 for some manifests

*) For the record a lil detail:
When purging the old install also the jelly apt-sources and keys should be removed to prevent this error triggered by the install-debuntu.sh: 

Code:

E: Conflicting values set for option Signed-By regarding source https://repo.jellyfin.org/debian/ bullseye: /var/lib/extrepo/keys/jellyfin.asc != /etc/apt/keyrings/jellyfin.gpg
E: The list of sources could not be read.
ERROR: Failed to update APT repositories. Something is wrong with your APT sources, GPG keys, or Internet connection. Try again shortly or use https://jellyfin.org/contact to find us for troubleshooting.

[TheDreadPirate]

How are you starting Jellyfin? It sounds like you're simply typing "jellyfin". You should be using systemctl.

sudo systemctl start jellyfin

[Riesling]

yes, with "systemctl start jellyfin" - and it is of course running.

---

but you can also type just "jellyfin" to see its status I guess:

Code:

root@jellyfin:/# jellyfin
[20:00:09] [INF] [1] Main: Jellyfin version: 10.9.11
[20:00:09] [INF] [1] Main: Environment Variables: ["[JELLYFIN_LOG_DIR, /root/.local/share/jellyfin/log]"]
[20:00:09] [INF] [1] Main: Arguments: ["/usr/lib/jellyfin/bin/jellyfin.dll"]
[20:00:09] [INF] [1] Main: Operating system: Debian GNU/Linux 11 (bullseye)
[20:00:09] [INF] [1] Main: Architecture: X64
[20:00:09] [INF] [1] Main: 64-Bit Process: True
[20:00:09] [INF] [1] Main: User Interactive: True
[20:00:09] [INF] [1] Main: Processor count: 8
[20:00:09] [INF] [1] Main: Program data path: /root/.local/share/jellyfin
[20:00:09] [INF] [1] Main: Log directory path: /root/.local/share/jellyfin/log
[20:00:09] [INF] [1] Main: Config directory path: /root/.config/jellyfin
[20:00:09] [INF] [1] Main: Cache path: /root/.cache/jellyfin
[20:00:09] [INF] [1] Main: Web resources path: /usr/lib/jellyfin/bin/jellyfin-web
[20:00:09] [INF] [1] Main: Application directory: /usr/lib/jellyfin/bin/
[20:00:09] [ERR] [1] Main: The server is expected to host the web client, but the provided content directory is either invalid or empty: /usr/lib/jellyfin/bin/jellyfin-web. If you do not want to host the web client with the server, you may set the '--nowebclient' command line flag, or set'hostwebclient=false' in your config settings