SOLVED: Server Unavailable on haproxy

[sEVacitU]

Setup: 
- Public IP: [Removed public IP - TheDreadPirate]
- Jellyfin Server: Windows PC 192:168.1.12:8096 no https, no docker
- haproxy Server: Rocky 9 10.0.0.12
- DNS: a record points "jellyfin.mydomain.com" to [Removed public IP - TheDreadPirate]

Notes:
- When doing a port scan port 443 is open at my public ip
- When monitoring my firewall rules, traffic is successfully being passed to my Rocky server
- I cannot ping my jellyfin server from my Rocky server, but if I log into the GUI and navigate to 192:168.1.12:8096 I can access my jellyfin library.


When attempting to navigate to jellyfin.mydomain.com I get a 503 Server Unavailable error. 

Code:

[root@localhost user]# systemctl status haproxy
[b]●[/b] haproxy.service - HAProxy Load Balancer
     Loaded: loaded (/usr/lib/systemd/system/haproxy.service; [b]disabled[/b]; preset: [b]disabled[/b])
     Active: [b]active (running)[/b] since Sat 2024-03-02 02:08:08 CST; 7h ago
    Process: 2164 ExecStartPre=/usr/sbin/haproxy -f $CONFIG -f $CFGDIR -c -q $OPTIONS (code=exited, status=0/SUCCESS)
   Main PID: 2166 (haproxy)
      Tasks: 5 (limit: 22944)
     Memory: 10.3M
        CPU: 3.263s
     CGroup: /system.slice/haproxy.service
             ├─2166 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -f /etc/haproxy/conf.d -p /run/haproxy.pid
             └─2168 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -f /etc/haproxy/conf.d -p /run/haproxy.pid

Mar 02 02:08:08 localhost.localdomain systemd[1]: Starting HAProxy Load Balancer...
Mar 02 02:08:08 localhost.localdomain haproxy[2166]: [NOTICE]   (2166) : New worker #1 (2168) forked
Mar 02 02:08:08 localhost.localdomain systemd[1]: Started HAProxy Load Balancer.
Mar 02 02:08:08 localhost.localdomain haproxy[2168]: [WARNING]  (2168) : Server jellyfin_backend/jellyfin is DOWN, reason: Layer4 connection problem, info: "General socket error (Permission denied)", check duratio>
Mar 02 02:08:08 localhost.localdomain haproxy[2168]: [NOTICE]   (2168) : haproxy version is 2.4.22-f8e3218
Mar 02 02:08:08 localhost.localdomain haproxy[2168]: [NOTICE]   (2168) : path to executable is /usr/sbin/haproxy
Mar 02 02:08:08 localhost.localdomain haproxy[2168]: [ALERT]    (2168) : sendmsg()/writev() failed in logger #2: No such file or directory (errno=2)
Mar 02 02:08:08 localhost.localdomain haproxy[2168]: [ALERT]    (2168) : backend 'jellyfin_backend' has no server available!

Code:

#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    option http-server-close
    option forwardfor       except 127.0.0.0/8
    option                  redispatch
    retries                 3
    timeout http-request    10s
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           10s
    maxconn                 3000

#---------------------------------------------------------------------
# main frontend which proxys to the backends
#---------------------------------------------------------------------
frontend jellyfin_proxy
    bind *:443 ssl crt /etc/letsencrypt/live/jellyfin.mydomain.com/fullchain.pem alpn h2,http/1.1
    redirect scheme https if !{ ssl_fc }
    
    option forwardfor
  
    acl letsencrypt_auth path_beg /.well-known/acme-challenge/

    acl is_jellyfin hdr(host) -i jellyfin.henrilogon.com

    use_backend jellyfin_backend if is_jellyfin

#---------------------------------------------------------------------
# static backend for serving up images, stylesheets and such
#---------------------------------------------------------------------
backend jellyfin_backend
    option httpchk
    option forwardfor
    http-check send meth GET uri /health
    http-check expect string Healthy
    server jellyfin 192.168.1.12:8096 check

backend letsencrypt
    server letsencrypt 127.0.0.1:8888

#---------------------------------------------------------------------


listen stats
bind *:9000
stats enable
stats uri /stats
stats refresh 10s
stats admin if TRUE

Code:

[root@localhost tim]# curl -v 192.168.1.12:8096
*   Trying 192.168.1.12:8096...
* Connected to 192.168.1.12 (192.168.1.12) port 8096 (#0)
> GET / HTTP/1.1
> Host: 192.168.1.12:8096
> User-Agent: curl/7.76.1
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 302 Found
< Content-Length: 0
< Date: Sat, 02 Mar 2024 16:26:27 GMT
< Server: Kestrel
< Location: /web/index.html
< 
* Connection #0 to host 192.168.1.12 left intact

Jellyfin Networking Settings:
Server Address Settings
    Local HTTP port number: 8096
    Enable HTTPS: Not Selected
    Local HTTPS port number: 8920
    Base URL: empty
    Bind to local network address: 192.168.1.12
    LAN networks: 192.168.1.0/24, 192.168.2.0/24
    Known proxies: 10.0.0.12
HTTPS Settings:
    *Disabled*
Remote Access Settings
    Allow remote connections to this server: checked
    Remote IP address filter: empty
    Remote IP address filter mode: Whitelist
    Enable automatic port mapping: unchecked
    Public HTTP port number: 8096
    Public HTTPS port number: 8920
IP Protocols
    *IPv4 Only*
Firewall and Proxy Settings
    jellyfin.mydomain.com


---- Edit ----
I did finally find the following log

Code:

Mar  2 10:51:50 localhost setroubleshoot[2096]: SELinux is preventing /usr/sbin/haproxy from name_connect 
access on the tcp_socket port 8096.#012#012*****  Plugin connect_ports (85.9 confidence) suggests   
*********************#012#012If you want to allow /usr/sbin/haproxy to connect to network port 8096#012
Then you need to modify the port type.#012Do#012# semanage port -a -t PORT_TYPE -p tcp 8096#012 
where PORT_TYPE is one of the following: commplex_link_port_t, commplex_main_port_t, dns_port_t, 
dnssec_port_t, fmpro_internal_port_t, http_cache_port_t, http_port_t, kerberos_port_t, ocsp_port_t, 
rtp_media_port_t.#012#012*****  Plugin catchall_boolean (7.33 confidence) suggests   ******************#012#012
If you want to allow nis to enabled#012Then you must tell SELinux about this by enabling the 'nis_enabled' boolean.
#012#012Do#012setsebool -P nis_enabled 1#012#012*****  Plugin catchall_boolean (7.33 confidence) suggests   
******************#012#012If you want to allow haproxy to connect any#012Then you must tell SELinux about this by 
enabling the 'haproxy_connect_any' boolean.#012#012Do#012setsebool -P haproxy_connect_any 1#012#012*****  
Plugin catchall (1.35 confidence) suggests   **************************#012#012If you believe that haproxy should be 
allowed name_connect access on the port 8096 tcp_socket by default.#012Then you should report this as a bug.#012
You can generate a local policy module to allow this access.#012Do#012allow this access for now by executing:#012#
ausearch -c 'haproxy' --raw | audit2allow -M my-haproxy#012# semodule -X 300 -i my-haproxy.pp#012