SOLVED: Intermittent Extremely Slow Stream Speeds on LAN only

[St. Glocktopus]

Okay so I'm absolutely stumped on this one. I have spent many days and too many hours trying to identify this issue and I'm hoping someone has experienced similar or has knowledge on how to find/fix. I'm a sysadmin for a living but networking is not my strong suite.

Setup details simplified breakdown:
=====

My setup is a SuperMicro SuperServer (specs available if needed) running Ubuntu 22.04.2 LTS with all the latest updates for all apps applied. I have Jellyfin running directly on the machine (no docker) using Apache2, Letsencrypt/certbot, and a domain provided by my router from Tp-Link and DDNS to point to it. I have a static Public IP from my ISP as well.

All of my devices are hard-wired to my network via Cat6 cables and 3 unmanaged switches rated for 1GB throughput each. I have fiber internet with 250/250mbps. I have verified the speeds are consistent to beta.speedtest.net while hardwired with every ethernet cable in use by connecting my laptop and verifying speeds are consistent for 3x tests on each cable, each port, each switch (absolutely a pointless test but I had to ensure my hardware wasn't the issue).

====

On to the issue ~

While attempting to stream shows/movies from my Jellyfin server to my local devices I have very random drops in speed from 250+mbps to less than 2mbps for no apparent reason and does not speed back up to 250+mbps until I close/reopen/restart/etc. I am using the latest updated version of Infuse Pro on my up-to-date iPhone (iPhone 14 Pro) and Apple TV (ethernet - model A2843 running tvOS 16.6 build 20M73) to test the speeds via the "share" speed-test function. Shows/movies will stutter hard and fail to play or load in small chunks, play for a few minutes/seconds and then buffer again for long periods of time or if I'm lucky the entire show/movie will load in one go and I'll have no issues until I attempt to fast-forward/rewind or stop the show/movie and attempt to resume it.

For the tests I would select a show/movie and press "start test" - it would begin the test reporting speeds of 250mbps+ the entire test no problem. If I stop and/or start a new test again it would either be full speed at 250mbps+ or completely throttle down to <2mbps until I stop and restart the test again (several times depending on how it decides to behave). I have included screenshots of the tests with my phone, I can include more from my TV if needed. 

This issue only happens while on my local network. If I am on cellular or another Wi-Fi network I have zero issues and full speeds 24/7. I share my server with several other users, as one does for their friends and family, and none of them have any issues with stuttering or lag at any time. My GF has the exact same setup that I do: iPhone, same apple TV up-to-date, etc., same router and switch with Cat6 cable connections, Infuse Pro, internet speeds of 100mbps via copper -- and has absolutely zero issues. I had her do the same tests I did to ensure the issue doesn't happen with speedtests. 

I have also tested this by using my browser on my laptop, desktop, the server itself, and a few other devices. It happens no matter what app I use (Swiftfin, Jellyfin browser, or Infuse) on any device and I can't figure it out at all. I have also ran 1000+ ping tests to make sure I wasn't having unexpected packet loss between my server and ISP, server and google DNS, server and computer/device, etc. I've tried setting my DNS to 1.1.1.1, 8.8.8.8, my router itself, my ISP's DNS, my sanity, etc. with no change.

I have gone through all my config files to ensure routing loops weren't causing the issue but again this only happens on local network and no one outside of my network has any issues. 

I have attempted to reboot all devices, hard shut-down by unplugging and plugging back in, the works. The full list of troubleshooting would be a novel by itself so I've spared the full troubleshooting breakdown from this post but can provide more details if needed.

I've attached my Jellyfin log and ffmpeg log for reference. 

Thank you for reading this very long post, I appreciate your time and hope I'm oblivious to a very obvious fix/issue.

[TheDreadPirate]

Where are you writing the transcodes to? Are any volumes filling up when it is transcoding?

[St. Glocktopus]

I am not sure where to check for where I'm writing transcodes to. I do not see any of my drive volumes filling with unexpected or large amounts of data either.

I did try toggling transcoding off in the dashboard per-user settings as some forums mentioned that worked for some users. It did not make a difference from what I can tell.

[TheDreadPirate]

I would not turn off transcoding.  Unless you have a known set of devices, with known capabilities, and your library strictly adheres to those known limitations.

Jellyfin will always direct play when possible.

Your transcoding directory is in Settings > Dashboard > Playback   (About halfway down)

Since the ffmpeg log you attached was copying the video and audio into a new container (no reencoding) if the transcode directory is on a small volume it would fill up pretty quickly.  I'm saying that is what is happening.  Just grasping at straws.

What does your storage look like?  RAID?  ZFS?  Hardware specs, just for giggles?  Does it have a GPU?

[jimmyjammy]

It certainly sounds like an issue with your home network. Am not very familiar with using Domains, is it possible to direct your client software to the local ethernet ip address of your server & see if the problem persists ?

[St. Glocktopus]

@TheDreadPirate

I went looking for the transcode folder and found 658 items in it totaling up to 1.1GB of disk usage. I wasn't sure if there were corrupt or partial files in it so I deleted all of them and restarted the server to "refresh" everything. I did re-enable transcoding as per your advice.

The drive the transcodes are sitting on is an 870 Evo with 250+GB of space remaining.

I have my OS and programs running off of my SSD (870 Evo) and it has plenty of room storage wise, it's also a brand new drive I purchased and installed about 2 months ago when I rebuilt the entire server migrating it from a desktop machine.

The rest of my data is stored on multiple HDDs that I had laying around from my job, it's a mix of sizes with minimum speeds of 7200rpm and each drive has a cache of 32MB minimum with my largest having a 256MB cache. Nothing impressive, but free is free.

In total I have 6 drives currently with 1 being the SSD running my OS and software. No RAID or ZFS configuration at the moment as my drives are mismatched in size; plans for it in the future though.

After wiping the data from the transcode dir last night I also changed the transcode path to a new directory on my highest cache HDD but realized that was a dumb idea as it would read/write much slower from any other drive and I also didn't configure Jellyfin to have the rwx permissions so thus I changed it back to the original file path. Today I haven't had a chance to test further but will report back when I am off work this afternoon.

The server does not have a dedicated GPU but does have a build-in processor from a company I've never heard of called Aspeed. Here are the full server specifications:

2x Intel Xeon E5-2630 v3 @2.4GHz for a total of 32 cores
64GB ECC DDR4 @2666MHz
Hardware RAID Controller (not in use as RAID)
GPU processor Aspeed (unknown model #, can provide later)
6x storage drives

I appreciate your time and interest in this weird situation.
====

@jimmyjammy

I also believe the issue resides with my local network. I believe my router may not support NAT loopback, but googling the model number has forums claiming it does with very little information to trust, and is thus sending my requests out to my ISP and back. But, even if that were the case I don't understand why I'm having issues only locally as all external streams are flawless and full speed.

I cannot connect to the server locally as my apache config redirects all traffic to the plain IP address or domain to my https://FQDN instead. I can connect to the server locally via a browser by typing in https://server.IP:443; however, this does not work on any other applications from my knowledge. I too thought maybe there is a configuration issue in my apache config but I can't quite put my finger on it if so. If you have have any experience with apache or use it I would love to compare config files for sites-available/enabled as that's where my suspicions lie with reverse-proxy.

Thank you for your input! I appreciate your time as well.

[TheDreadPirate]

I did see some weird "external address" logs, so I think I am also starting to this the issue is your router.

Is your TP Link router supported by any of the 3rd party router firmware projects?  Most of them likely support NAT Loopback.  I know Fresh Tomato does (my Netgear R7000 runs this and NAT loopback works).

Do all of your devices connect via the DDNS address? What happens when you use the local IP + port 8096?

[St. Glocktopus]

I saw those too and that's what lead me down the NAT loopback rabbit-hole.

The Tp-Link router is not supported by any 3rd party firmware, I also looked into that and was sorely disappointed. I do have a Sonicwall TZ-240 hardware firewall sitting around that I've been considering implementing to replace the Tp-Link as primary but that would break my DDNS and domain forcing me to recreate the whole setup I think.

All my devices are connected via the DDNS address, I cannot connect to it using the local IP address and port 8096 as I get an error with failure to connect or in a browser it just fails to load and says the site does not exist or is not available.

I can post my apache config file for my 000-default-le later today as I was a dummy and configured both Ombi and Jellyfin to run in the same config file.

[TheDreadPirate]

Does your server's firewall not allow 8096 from external addresses? Or from different subnets? Something along those lines. I'm just trying to isolate this being a DDNS/NAT loopback routing issue and wanted to test if local accesses behave the properly or exhibit the same behavior.

[St. Glocktopus]

The software firewall is configured to allow 8096/tpc and udp on the server so I don't know why I can't connect to it via IP:port. I can't reach it via localhost:8096 even directly on the server itself via the browser, that's why I was thinking my Apache config may be interfering with access.

I can access is via the local IP and port 443 which doesn't seem to auto-redirect to my FQDN but any attempts to use just the plain IP address will redirect and any attempts to specify any port other than 443 (or ports to my other programs I have running such a radarr, sonarr, prowlarr, ombi, and qbitt which I can reach via the local IP + their respective port(s)).

I've been trying to get it to work off of local IP:8096 for further testing of speeds locally but can't figure out why it refuses connections to that IP:port. I did experience an oddity when I restarted my router and got full 1Gbps speeds during the router boot process with a speed test that quickly calmed back down to the usual 250mbps after the router finished booting fully. Which brings me back to thinking the router is sending all traffic out and back in for routing to the FQDN.