VPS, nginx, ssh tunnel to local server

[raccoonsummer]

I'm trying to set up a VPS such that I can allow my parents to use their Roku to load / play things from my Jellyfin server. The VPS is running an nginx server with a reverse proxy setup as per the Jellyfin documentation, as far as I can tell, but I think some parts are misconfigured, and there's some differences between how I have my system set up versus the official documentation, and so I'm not really sure how to get it all working correctly.

The general idea that I'm going for is:
internet client -> https to my VPS -> proxy connection via SSH tunnel -> my synology -> docker container running Jellyfin

My starting point for this was this thread on Reddit: https://www.reddit.com/r/jellyfin/commen..._to_allow/

My Environment:
Local:
Jellyfin via Docker, running on a Synology DS920+ which handles hardware transcoding and has all the local storage for media.

VPS:
Linode VPS, static IP, nginx installed and configured, Let's Encrypt SSL cert is OK, custom DNS and domain name all set up (using a subdomain setup, ie jellyfin.cloud.mydomain.com), secured with only SSH Certificate login and Fail2Ban set up / secured.

Where I went off script / tried customizing things:
I can't get the Docker container to handle the SSH Tunnel configuration from the Jellyfin container to the VPS, so I figured I could do that from the Synology itself via command line for testing, then later set it up via scheduled task to make sure that the tunnel is set up and running on reboot, and/or resets daily at some time, so that it's sure to be up and running. I haven't been able to confirm that this is actually working or not, but it appears to be?

Configuration stuff:
VPS:
/etc/nginx/conf.d/jellyfin.conf

Code:

# jellyfin configuration, taken from https://jellyfin.org/docs/general/networking/nginx/
# uncomment sections once https is set up.

server {
    if ($host = jellyfin.cloud.mydomain.tld) {
        return 301 https://$host$request_uri;
    } # managed by Certbot


listen 80;
listen [::]:80;
server_name jellyfin.cloud.mydomain.tld;

# Uncomment to redirect HTTP to HTTPS:
return 301 https://$host$request_uri;
}

server {
listen 443 ssl http2;
listen [::]:443 ssl http2;
server_name jellyfin.cloud.mydomain.tld;

## The default `client_max_body_size` is 1M, this might not be enough for some posters, etc.
client_max_body_size 20M;

# use a variable to store the upstream proxy
# in this example, we are using a hostname which is resolved via DNS
# (if you aren't using DNS, remove the resolver line and change the variable to point to an ip address  e.g `set $jellyfin 127.0.0.1`)
set $jellyfin 127.0.0.1;
#resolver 127.0.0.1 valid=30;
    ssl_certificate /etc/letsencrypt/live/jellyfin.cloud.mydomain.tld/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/jellyfin.cloud.mydomain.tld/privkey.pem; # managed by Certbot
include /etc/letsencrypt/options-ssl-nginx.conf;
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;
add_header Strict-Transport-Security "max-age=31536000" always;
ssl_trusted_certificate /etc/letsencrypt/live/jellyfin.cloud.mydomain.tld/chain.pem;
ssl_stapling on;
ssl_stapling_verify on;

# Security / XSS Mitigation Headers
# NOTE: X-Frame-Options may cause issues with the webOS app
add_header X-Frame-Options "SAMEORIGIN";
add_header X-XSS-Protection "0"; # Do NOT enable. This is obsolete/dangerous.
add_header X-Content-Type-Options "nosniff";

# COOP/COEP. Disable if you use external plugins/images/assets
add_header Cross-Origin-Opener-Policy "same-origin" always;
add_header Cross-Origin-Embedder-Policy "require-corp" always;
add_header Cross-Origin-Resource-Policy "same-origin" always;

# Permissions policy. May cause issues on some clients
add_header Permissions-Policy "accelerometer=(), ambient-light-sensor=(), battery=(), bluetooth=(), camera=(), clipboard-read=(), display-capture=(), document-domain=(), encrypted-media=(), gamepad=(), geolocation=(), gyroscope=(), hid=(), idle-detection=(), interest-cohort=(), keyboard-map=(), local-fonts=(), magnetometer=(), microphone=(), payment=(), publickey-credentials-get=(), serial=(), sync-xhr=(), usb=(), xr-spatial-tracking=()" always;

# Tell browsers to use per-origin process isolation
add_header Origin-Agent-Cluster "?1" always;

# Content Security Policy
# See: https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP
# Enforces https content and restricts JS/CSS to origin
# External Javascript (such as cast_sender.js for Chromecast) must be whitelisted.
# NOTE: The default CSP headers may cause issues with the webOS app
add_header Content-Security-Policy "default-src https: data: blob: http://image.tmdb.org; style-src 'self' 'unsafe-inline'; script-src 'self' 'unsafe-inline' https://www.gstatic.com/cv/js/sender/v1/cast_sender.js https://www.gstatic.com/eureka/clank/95/cast_sender.js https://www.gstatic.com/eureka/clank/96/cast_sender.js https://www.gstatic.com/eureka/clank/97/cast_sender.js https://www.youtube.com blob:; worker-src 'self' blob:; connect-src 'self'; object-src 'none'; frame-ancestors 'self'";

location = / {
return 302 http://$host/web/;
return 302 https://$host/web/;
}

location / {
# Proxy main jellyfin traffic
proxy_pass http://$jellyfin:8096;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Protocol $scheme;
proxy_set_header X-Forwarded-Host $http_host;

# Disable buffering when the nginx proxy gets very resource heavy upon streaming
proxy_buffering off;
}

# location block for /web -- This is purely for aesthetics so that /web/#!/ works instead of having to to /web/index.html/#!/
location = /web/ {
# Proxy main Jellyfin traffic
proxy_pass http://$jellyfin:8096/web/index.html;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Protocol $scheme;
proxy_set_header X-Forwarded-Host $http_host;
}

location /socket {
# proxy jellyfin websockets traffic
proxy_pass http://$jellyfin:8086;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Protocol $scheme;
proxy_set_header X-Forwarded-Host $http_host;
}
}

On the Synology via command line, I'm running the below command to establish the ssh tunnel:

Code:

ssh -NTC -o ServerAliveInterval=60 -o ExitOnForwardFailure=yes -R 127.0.0.1:8096:127.0.0.1:8096 nginx-ssh@jellyfin.cloud.mydomain.tld

I can see that the connection is established on the VPS:
[vpshost]$ sudo netstat -at
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address          Foreign Address        State     
tcp        0      0 localhost:8096          0.0.0.0:*              LISTEN   
tcp        0      0 0.0.0.0:https          0.0.0.0:*              LISTEN   
tcp        0      0 0.0.0.0:http            0.0.0.0:*              LISTEN   
tcp        0      0 0.0.0.0:ssh            0.0.0.0:*              LISTEN   
tcp        0    288 cloud.mydomain.tld:ssh redacted.ip.addr:15100    ESTABLISHED
tcp        0      0 cloud.mydomain.tld:ssh redacted.ip.addr:54494      ESTABLISHED
tcp6      0      0 [::]:https              [::]:*                  LISTEN   
tcp6      0      0 [::]:http              [::]:*                  LISTEN   
tcp6      0      0 [::]:ssh                [::]:*                  LISTEN   

However, when I try to load my jellyfin instance, I am getting a 502 Bad Gateway error from the web server:
https://jellyfin.cloud.mydomain.tld/ forwards to https://jellyfin.cloud.mydomain.tld/web/ ---> returns 502 Bad Gateway nginx/1.20.1

/var/log/nginx/error.log
2023/08/18 14:02:40 [crit] 699#699: connect() to 23.214.95.221:80 failed (13: Permission denied) while requesting certificate status, responder: r3.o.lencr.org, peer: 23.214.95.221:80, certificate: "/etc/letsencrypt/live/jellyfin.cloud.mydomain.tld/fullchain.pem"
2023/08/18 14:02:40 [crit] 699#699: connect() to 23.214.95.212:80 failed (13: Permission denied) while requesting certificate status, responder: r3.o.lencr.org, peer: 23.214.95.212:80, certificate: "/etc/letsencrypt/live/jellyfin.cloud.mydomain.tld/fullchain.pem"
2023/08/18 14:02:40 [crit] 699#699: connect() to [2600:1406:5600:3::17d6:5fdd]:80 failed (13: Permission denied) while requesting certificate status, responder: r3.o.lencr.org, peer: [2600:1406:5600:3::17d6:5fdd]:80, certificate: "/etc/letsencrypt/live/jellyfin.cloud.mydomain.tld/fullchain.pem"
2023/08/18 14:02:40 [crit] 699#699: connect() to [2600:1406:5600:3::17d6:5fd4]:80 failed (13: Permission denied) while requesting certificate status, responder: r3.o.lencr.org, peer: [2600:1406:5600:3::17d6:5fd4]:80, certificate: "/etc/letsencrypt/live/jellyfin.cloud.mydomain.tld/fullchain.pem"
2023/08/18 14:02:41 [crit] 699#699: *1 connect() to 127.0.0.1:8096 failed (13: Permission denied) while connecting to upstream, client: 146.70.174.92, server: jellyfin.cloud.mydomain.tld, request: "GET /web/ HTTP/2.0", upstream: "http://127.0.0.1:8096/web/index.html", host: "jellyfin.cloud.mydomain.tld"
2023/08/18 14:02:41 [crit] 699#699: *1 connect() to 127.0.0.1:8096 failed (13: Permission denied) while connecting to upstream, client: 146.70.174.92, server: jellyfin.cloud.mydomain.tld, request: "GET /favicon.ico HTTP/2.0", upstream: "http://127.0.0.1:8096/favicon.ico", host: "jellyfin.cloud.mydomain.tld", referrer: "https://jellyfin.cloud.mydomain.tld/web/"

---

Potential problems that I'm considering, but don't know how to troubleshoot:

1. There's some issue with how nginx is set up / allowed to communicate with the ssh tunnel that's been set up? But I don't know how to confirm if that's the case or how to resolve it if that is indeed the situation. 
   
2. The ssh tunnel command is setting something up incorrectly? I don't fully understand the "-R 127.0.0.1:8096:127.0.0.1:8096" part.
   
3. There's a permission issue on the VPS between the nginx user (built in / is what nginx is running under) and the nginx-ssh user account (that I set up for certificate-based ssh login)
   
4. The fact that Jellyfin is running in a docker container is causing weirdness? I don't have any other hardware (currently) that I could run a full Jellyfin installation on, so I'm hoping there's a way to keep it on the Synlolgy within a docker container. (I could maybe use a full VM on the Synology, but I think that that will eat up a lot more system resources...)
   

NOTE: I've replaced all instances of my actual domain name with "mydomain.tld"

Thank you reading and for any suggestions or troubleshooting steps! I'd have put this all in the Jellyfin Reddit thread on the subject, but it appears that that's no longer possible. I'm hoping that this setup isn't too weird, since I'd really like to avoid having to forward ports from my home network.

[TheDreadPirate]

For the SSH command, I think it the "-R" part should be

Code:

-R 8096:127.0.0.1:8096

The documentation says it should be <remote port>:<localhost>:<local port>.

I think it makes more sense to use something like wireguard to tunnel between your VPS and your Synology box.

Also, is there a reason you are using a VPS instead of hosting nginx in another container on your Synology NAS? Or did you already have the VPS for other reasons?

[raccoonsummer]

TheDreadPirate,
I just tried your suggested change, and unfortunately I'm still getting the 502 Bad Gateway page.

I'd like to use wireguard, but I'm even less familiar with how to arrange that to allow VPS traffic to be proxied to my local system.

The reason behind trying to put it up on a VPS is that the VPS gets a static IP that I can point my domain name at, is a separate system from anything at my home, and if anything goes wrong with the VPS, I can just nuke it and set up a new one. I like the idea of the separation in devices for security's sake. Also getting https:// working properly at home is kinda funky with the DDNS stuff, I think? If I were to use my local hardware to host externally, I'd have to open a port, firewall it properly (which isn't so bad), and also run some DDNS service to keep my IP / DNS entries up do date. Not the worst thing... and if it'd be significantly easier to set up I might consider it - I just also haven't found a good, clear guide on how to do it securely in such a way that I can be pretty confident that a security issue won't expose my whole home network.

[TheDreadPirate]

With LetsEncrypt and certbot, the fact that you are using DDNS is completely transparent. The process is exactly the same for the certificate request process. Certbot manages both of my certs, and both certs use my NoIP DDNS addresses.

As for security concerns. If you use non-standard ports you significantly reduce your chance someone attempting to break in. Most cyber-actors/script kiddies are scanning common ports to run their cookie cutter attacks against. Port 80, 443, 22, 25, 3389, etc. If you run your services on ephemeral ports (49152-65535) you are unlikely to be noticed. If you keep your software up to date, cookie cutter exploit scripts will be ineffective. And, let's be real, none of us are worth someone going out of their way to scan every port. We are not worth someone using their secret zero-day exploit against.

If I'm a hacker of a high enough caliber that I discovered a zero-day vulnerability and developed the capability to exploit it I'm using that against some massive multi-national company, defense contractor, government, or intelligence agency. If a I'm bottom of the barrel script kiddie, I'm going after easy targets and letting this random script I found on a hacker forum do all the work.

And if you setup your nginx container properly, it kind of is separate from your other containers and devices. Even on an entirely bare metal setup like mine, as long as you properly use groups and permissions you are requiring this hypothetical hacker to have additional exploits to break out to system wide access.