Large files & remuxes buffer while streaming in the same network

[Sash]

I installed Jellyfin MPV Shim and did the configuration that was mentioned in this post

1. I can play all formats on MPV Shim. 
2. When I cast videos using MPV Shim they are very stuttery, but when I use mpv.exe directly there is no stuttering or lag.

I disabled the transcoding in Jellyfin MPV Shim by right clicking on the taskbar tray icon, and then selecting "application menu" but it still caching and transcoding. 

Here are some videos of the issue

mpv: https://streamable.com/78qkx7
jellyfin shim cast: https://streamable.com/woucij

What could be causing the stutter?