Android TV cannot reach jellyfin server

[jgx0]

I've set up my home server with jellyfin and I am starting to get PTSD from the login screen on my Android TV. It almost never works, but sometimes it does. Connecting with my devices likes Phone or Tablet works almost always.
My server is discovered under http://192.168.1.61:8096 but if I try to click on it, it will say "Unable to connect to server https://...". So, for some reason it tries https (which is disabled in my dashboard). But also putting in the direct IP address (with or without port, with or without http://) will not let me connect.
The server is up and running, I can connect to it with my other devices. I reinstalled the app, I cleared the cache and data, nothing works.
I run jellyfin in a docker image via:

Code:

sudo docker run -d \
--name jellyfin \
-p 8096:8096 \
-p 7359:7359/udp \
-p 1900:1900/udp \
-v /home/user/jellyfin/config:/config \
-v /home/user/jellyfin/cache:/cache \
-v /mnt/mediahdd:/media \
-e JELLYFIN_PublishedServerUrl="http://192.168.1.61:8096/" \
--restart unless-stopped \
jellyfin/jellyfin:latest

I am getting quite desperate here, nothing seems to work, the server seems to be there and the TV even finds the correct IP address...

EDIT: I sideloaded Whophin on my TV and this also doesn't work. Again, same symptom: Server is discovered, but connection times out even if putting the correct IP http://192.168.1.61:8096

EDIT2: It seems that the server is briefly reachable from the TV if I restart the whole server(!). The process is always the same. I need to restart the server. Then I can log into Jellyfin on my TV. But then, I'll get a playback error. I need to restart jellyfin again on my server. Then it actually works for a while. At some point, the playback freezes and the thing is gone again.