Solution: After inspecting my network, I have found the problem. Wireguard, where I was pushing all my trafic through before it hit me or the internet, was throttling my upload speeds on the server. Disabling Wireguard has increased my speeds to that of no buffering now.
