Jellyfin stuttering with direct play on everything but host device

[Caffeine]

Server is 10.11.1
I really don't know why this is happening, it never happened when the server was hosted on my NAS and it didn't happen when I first reinstalled on my laptop either. Media plays just fine watching off jellyfin on my laptop but on any other platform the entire episode freezes every 2 seconds and buffers, but when transcoding this issue NEVER comes up. Not sure what to provide for logs either because it seems jellyfin doesn't produce any if you're direct playing

[Tone]

Which client are you using?
I have a similar problem, but just on one fire stick (2nd gen)