SOLVED: Why is the transcoding bitrate limited

[bagno]

I run Jellyfin in my homelab on one of my nodes with 2x Xeon E5-2670 v3.
All my hot media (accessible to Jellyfin) is stored on a RAID0 array of 8 SAS drives (sequential reads 1.1GB/s, random reads ~600 MB/s).
My network within my homelab and household has a 2.5 Gbps capacity.

I set my transcode CRF for H264 and H265 to 16. Bandwidth is not a concern. Everything is local, nothing is over the internet.
Hardware acceleration is disabled. The transcode profile is set to medium and I'm managing around 1.6x transcode speed with around 60% utilization on both CPUs.

My source for this particular case is a 30.8 Mbps H.265 HDR10 video. I have tonemapping enabled, as my target client doesn't do HDR very well.

The target client is MS Edge running on Xbox Series X.

So what's wrong?
1. Well, it appears that Jellyfin will forcefully limit the transcode bitrate to that of the source - which absolutely annihilates the quality:

2. Despite having the right transcode speed for smooth playback and having paused for 10 minutes to get a significant bump in transcode progress - the playback is choppy as heck and drops frames left and right.

My question:
Would it be possible to absolutely REMOVE all the bitrate restrictions for all transcoded media? I want to be able to run at around 16 CRF for video and 512 kbps for transcoded AAC audio.
If not, can someone give me a hint where in the Jellyfin source code can I find the aforementioned restrictions, fork it, modify it and run my own custom build? I'd do it sooner but Jellyfin is a fairly large project and I can't seem to find where this restriction might be.

Suggestion:
Perhaps a checkbox that toggles "LAN mode" would be a really nice feature for people in my boat. You know, people who have a lot of compute and run entirely on LAN.

[TheDreadPirate]

30Mbps H264 should be plenty, especially if the resulting output resolution is 1080P. The setting that is probably killing your quality is the "veryfast" preset.

[bagno]

Ah, my apologies. I forgot the mention that the source/target is 4K, not 1080p.
I tried setting the preset to medium again, and now I'm left with only the second problem - see the video below (recorded before the change, but the symptom is the same):
https://www.youtube.com/watch?v=i5Xaic49ZsI

[TheDreadPirate]

I don't know what kind of device that is, but the TV resolution is 1080P but you are still sending a 4K file to it. Depending on the specs of the client device, it may just be struggling to play it.

Settings (not dashboard) > Playback > Maximum Allowed Video Transcoding Resolution. Set that to 1080P.

[bagno]

I think you might be right, but for another reason.

Theoretically Xbox Series X should do 4K video just fine and it should output 4K no issue.
The TV is definitely 4K, but it appears that I'm running into a limitation of what MS Edge can do on the Xbox.
For some odd reason UWP apps seem to be restricted to 1080p and seem to have much less resources available to them.

I hooked up a laptop to the TV and streamed via Chrome, smooth playback absolutely no issues whatsoever.
Seems like I need to re-think my living room setup for streaming, perhaps build a dedicated couch PC.

Really appreciate your willingness to help out, despite the fact the issue is clearly on the client side for me.

Thank you!