Hmmm. Neither my main Linux Jellyfin server and my test Windows Jellyfin server showed any appreciable change in memory usage when direct playing. Tried both 1080P and 4K HDR content.
The memory usage in Windows is from running Destiny, btw. Not Jellyfin server.
Jellyfin 10.10.1 (Docker)
Ubuntu 24.04 LTS
Intel i3 12100
Intel Arc A380
OS drive - SK Hynix P41 1TB
Storage
3x WD Red Pro 6TB CMR in RAIDZ1 (JF Library)