Thinking of switching to windows or truenas for a new jellyfin build

[TheDreadPirate]

I suspect a lot of the QSV issues you had were related to non-free Intel driver packages not being installed. Not every distro includes them by default. And with your 11700, there are issues with older kernels not supporting the iGPU in 11th gen+ Intel CPUs (needs 6.17+, IIRC). And OMV is a bit slower with pushing out newer kernel versions. And when you added docker, you also need to pass the GPU into the container.

1. Any OS is stable enough these days if you don't care about rebooting for updates. Also, ZFS is natively supported in Linux. And ZFS has great software RAID support and has things like snapshots and bit rot protection (needs parity based RAID).
2. Quick sync requires additional setup in some Linux distros. But popular distros like Ubuntu are pretty much zero setup. You do have to be careful that you use newer kernels depending on which Intel GPU you have. With Windows 10 you need to install Intel's drivers, not just the ones Windows automatically installs. So you aren't really saving any steps here with Windows.
3. TrueNAS Scale (based on Debian Linux), lags behind a bit with what kernel version they push out. But you can always manually install a newer kernel. Getting QSV to work on TrueNAS Scale is about the same as plain Debian.

IMO, Windows 10 is a pain to work with when using it as a server. Even server editions of Windows aren't very good servers. In my experience as both a Windows and Linux admin, the only thing that Windows does better as a server is be an Active Directory Domain Controller. Yes, using the command line is a pretty significant learning curve. But once you are comfortable with it, you will come to appreciate the level of control you have.

Also, if Jellyfin was your only container you should consider just doing a bare metal install. In Ubuntu/Debian there is an official script to automate installation. And it saves you the extra steps of configuring Docker, making sure to pass in the GPU, etc.