SOLVED: Server Unavailable on haproxy

SOLVED: Server Unavailable on haproxy - Printable Version

+- Jellyfin Forum (https://forum.jellyfin.org)
+-- Forum: Support (https://forum.jellyfin.org/f-support)
+--- Forum: Troubleshooting (https://forum.jellyfin.org/f-troubleshooting)
+---- Forum: Networking & Access (https://forum.jellyfin.org/f-networking-access)
+---- Thread: SOLVED: Server Unavailable on haproxy (/t-solved-server-unavailable-on-haproxy)

Server Unavailable on haproxy - sEVacitU - 2024-03-02

Setup:
- Public IP: [Removed public IP - TheDreadPirate]
- Jellyfin Server: Windows PC 192:168.1.12:8096 no https, no docker
- haproxy Server: Rocky 9 10.0.0.12
- DNS: a record points "jellyfin.mydomain.com" to [Removed public IP - TheDreadPirate]

Notes:
- When doing a port scan port 443 is open at my public ip
- When monitoring my firewall rules, traffic is successfully being passed to my Rocky server
- I cannot ping my jellyfin server from my Rocky server, but if I log into the GUI and navigate to 192:168.1.12:8096 I can access my jellyfin library.

When attempting to navigate to jellyfin.mydomain.com I get a 503 Server Unavailable error.

Code:
[root@localhost user]# systemctl status haproxy

[b]●[/b] haproxy.service - HAProxy Load Balancer

     Loaded: loaded (/usr/lib/systemd/system/haproxy.service; [b]disabled[/b]; preset: [b]disabled[/b])

     Active: [b]active (running)[/b] since Sat 2024-03-02 02:08:08 CST; 7h ago

    Process: 2164 ExecStartPre=/usr/sbin/haproxy -f $CONFIG -f $CFGDIR -c -q $OPTIONS (code=exited, status=0/SUCCESS)

   Main PID: 2166 (haproxy)

      Tasks: 5 (limit: 22944)

     Memory: 10.3M

        CPU: 3.263s

     CGroup: /system.slice/haproxy.service

             ├─2166 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -f /etc/haproxy/conf.d -p /run/haproxy.pid

             └─2168 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -f /etc/haproxy/conf.d -p /run/haproxy.pid

Mar 02 02:08:08 localhost.localdomain systemd[1]: Starting HAProxy Load Balancer...

Mar 02 02:08:08 localhost.localdomain haproxy[2166]: [NOTICE]   (2166) : New worker #1 (2168) forked

Mar 02 02:08:08 localhost.localdomain systemd[1]: Started HAProxy Load Balancer.

Mar 02 02:08:08 localhost.localdomain haproxy[2168]: [WARNING]  (2168) : Server jellyfin_backend/jellyfin is DOWN, reason: Layer4 connection problem, info: "General socket error (Permission denied)", check duratio>

Mar 02 02:08:08 localhost.localdomain haproxy[2168]: [NOTICE]   (2168) : haproxy version is 2.4.22-f8e3218

Mar 02 02:08:08 localhost.localdomain haproxy[2168]: [NOTICE]   (2168) : path to executable is /usr/sbin/haproxy

Mar 02 02:08:08 localhost.localdomain haproxy[2168]: [ALERT]    (2168) : sendmsg()/writev() failed in logger #2: No such file or directory (errno=2)

Mar 02 02:08:08 localhost.localdomain haproxy[2168]: [ALERT]    (2168) : backend 'jellyfin_backend' has no server available!

Code:
#---------------------------------------------------------------------

# common defaults that all the 'listen' and 'backend' sections will

# use if not designated in their block

#---------------------------------------------------------------------

defaults

    mode                    http

    log                     global

    option                  httplog

    option                  dontlognull

    option http-server-close

    option forwardfor       except 127.0.0.0/8

    option                  redispatch

    retries                 3

    timeout http-request    10s

    timeout queue           1m

    timeout connect         10s

    timeout client          1m

    timeout server          1m

    timeout http-keep-alive 10s

    timeout check           10s

    maxconn                 3000

#---------------------------------------------------------------------

# main frontend which proxys to the backends

#---------------------------------------------------------------------

frontend jellyfin_proxy

    bind *:443 ssl crt /etc/letsencrypt/live/jellyfin.mydomain.com/fullchain.pem alpn h2,http/1.1

    redirect scheme https if !{ ssl_fc }

    option forwardfor

    acl letsencrypt_auth path_beg /.well-known/acme-challenge/

    acl is_jellyfin hdr(host) -i jellyfin.henrilogon.com

    use_backend jellyfin_backend if is_jellyfin

#---------------------------------------------------------------------

# static backend for serving up images, stylesheets and such

#---------------------------------------------------------------------

backend jellyfin_backend

    option httpchk

    option forwardfor

    http-check send meth GET uri /health

    http-check expect string Healthy

    server jellyfin 192.168.1.12:8096 check

backend letsencrypt

    server letsencrypt 127.0.0.1:8888

#---------------------------------------------------------------------

listen stats

bind *:9000

stats enable

stats uri /stats

stats refresh 10s

stats admin if TRUE

Code:
[root@localhost tim]# curl -v 192.168.1.12:8096

*   Trying 192.168.1.12:8096...

* Connected to 192.168.1.12 (192.168.1.12) port 8096 (#0)

> GET / HTTP/1.1

> Host: 192.168.1.12:8096

> User-Agent: curl/7.76.1

> Accept: */*

> 

* Mark bundle as not supporting multiuse

< HTTP/1.1 302 Found

< Content-Length: 0

< Date: Sat, 02 Mar 2024 16:26:27 GMT

< Server: Kestrel

< Location: /web/index.html

< 

* Connection #0 to host 192.168.1.12 left intact

Jellyfin Networking Settings:
Server Address Settings
Local HTTP port number: 8096
Enable HTTPS: Not Selected
Local HTTPS port number: 8920
Base URL: empty
Bind to local network address: 192.168.1.12
LAN networks: 192.168.1.0/24, 192.168.2.0/24
Known proxies: 10.0.0.12
HTTPS Settings:
*Disabled*
Remote Access Settings
Allow remote connections to this server: checked
Remote IP address filter: empty
Remote IP address filter mode: Whitelist
Enable automatic port mapping: unchecked
Public HTTP port number: 8096
Public HTTPS port number: 8920
IP Protocols
*IPv4 Only*
Firewall and Proxy Settings
jellyfin.mydomain.com

---- Edit ----
I did finally find the following log

Code:
Mar  2 10:51:50 localhost setroubleshoot[2096]: SELinux is preventing /usr/sbin/haproxy from name_connect 

access on the tcp_socket port 8096.#012#012*****  Plugin connect_ports (85.9 confidence) suggests   

*********************#012#012If you want to allow /usr/sbin/haproxy to connect to network port 8096#012

Then you need to modify the port type.#012Do#012# semanage port -a -t PORT_TYPE -p tcp 8096#012 

where PORT_TYPE is one of the following: commplex_link_port_t, commplex_main_port_t, dns_port_t, 

dnssec_port_t, fmpro_internal_port_t, http_cache_port_t, http_port_t, kerberos_port_t, ocsp_port_t, 

rtp_media_port_t.#012#012*****  Plugin catchall_boolean (7.33 confidence) suggests   ******************#012#012

If you want to allow nis to enabled#012Then you must tell SELinux about this by enabling the 'nis_enabled' boolean.

#012#012Do#012setsebool -P nis_enabled 1#012#012*****  Plugin catchall_boolean (7.33 confidence) suggests   

******************#012#012If you want to allow haproxy to connect any#012Then you must tell SELinux about this by 

enabling the 'haproxy_connect_any' boolean.#012#012Do#012setsebool -P haproxy_connect_any 1#012#012*****  

Plugin catchall (1.35 confidence) suggests   **************************#012#012If you believe that haproxy should be 

allowed name_connect access on the port 8096 tcp_socket by default.#012Then you should report this as a bug.#012

You can generate a local policy module to allow this access.#012Do#012allow this access for now by executing:#012#

ausearch -c 'haproxy' --raw | audit2allow -M my-haproxy#012# semodule -X 300 -i my-haproxy.pp#012

RE: Server Unavailable on haproxy - TheDreadPirate - 2024-03-02

Is the address 10.12 or 1.12? Your curl command is using 10.12 but all your configs are 1.12.

RE: Server Unavailable on haproxy - sEVacitU - 2024-03-03

(2024-03-02, 06:23 PM)TheDreadPirate Wrote: Is the address 10.12 or 1.12? Your curl command is using 10.12 but all your configs are 1.12.

It's 10.21. I just get into the habbit of changing all IPs even though those privates legit don't really matter. I will edit the curl to match the rest.

RE: Server Unavailable on haproxy - TheDreadPirate - 2024-03-03

Do you have a firewall rule on your Jellyfin host that only allows connections to port 8096 from the 192.168.1.0/24 subnet?

RE: Server Unavailable on haproxy - sEVacitU - 2024-03-03

(2024-03-03, 02:32 AM)TheDreadPirate Wrote: Do you have a firewall rule on your Jellyfin host that only allows connections to port 8096 from the 192.168.1.0/24 subnet?

Nope. The firewall rules I have set up on the host allows local networks set to 192.168.1.0/24, 192.168.2.0/24, 10.0.0.0/24 and any remote network. (I know this isn't a great idea, currently I am just port forwarding from my public IP to my jellyfin server with no proxy. I will change that firewall rule once I get my proxy working.

When using a browser on 10.0.0.1 I can get to my jellyfin server without issue, but I get a layer 4 rejection message when attempting to use the proxy. I didn't know if for some reason my proxy was trying to use HTTPS on the backend instead of HTTP, otherwise I'm pretty lost about why it doesn't work.

RE: Server Unavailable on haproxy - sEVacitU - 2024-03-03

After parsing out the log message I recieved I added the following command and it resolved the issue for me.

Code:
sudo semanage port -a -t http_port_t -p tcp 8096