Bad (capped) transcode quality even in the best conditions

[tokki]

Hey ! I'm trying to use correctly the transcoder, as every film I host has different resolutions, codecs, bitrates, color range, etc
But I'm stuck on something, it's the maximum transcoding quality.

To put it simple, no matter how I try to tune the transcode quality, it will be capped at around 10-11 Mbps (or around 1.5x the original bitrate), which is not enough, I can see artifacts and blocking. Not so much that I can't use the result at all, but still visible.
Let's take an example on this film

Viewer dimensions:
1872x1004
Video resolution :
3840x2076

Transcoding info :
Video codec :
H264
Codec audio :
AAC (direct)
Bitrate:
11.6 Mbps
Transcode fps :
232 fps
Why is it transcoding :
The video's range type is not supported

Original file info
Container :
mkv
Size :
5.4 GiB
Bitrate :
8.1 Mbps
Video Codec :
HEVC Main 10
Video Bitrate :
6.6 Mbps
Video range type:
HDR10
Codec audio :
AAC LC
Bitrate audio :
768 kbps

I'm viewing with the 4k - 120Mbps setting
I'm on localhost, so there is no connection bottleneck 
I'm aware that the original file is not top-notch quality, BUT, the encoding is good and when directly viewing it with VLC, I don't see any blocking or artifacts. As I'm in the best conditions possibles (localhost, only user, hardware acceleration), I would aim to have a similar transcoded quality perception, even at the price of a big bitrate. (I mean, the parameter says 120Mbps, right ?)

So here, the transcoding happens because my screen can't display HDR, which is the case, so all good.
But as you can see, the transcoded bitrate is low, (and so is the perceived quality).

I have hardware acceleration enabled (RTX 2080 Super), and I don't know what could be causing this problem
I tried to change the encoding CRF to absurd values just for testing purposes, I see 0 differences, not in the slightest, I don't know if these values or used at all (or if I misconfigured something)
   
I didn't try to change the encoding preset as in my understanding of things it affects quality for a given bitrate (or at least it changes the speed or encoding more than the bitrate itself). I will still try as I have some performance headroom, but it wouldn't be a real solution

So I'm here asking for help
If this is expected behavior, can you explain to me why ?
And if not, how to correct this ? Thanks a lot !

[TheDreadPirate]

IIRC, the 1.5X change in bit rate is hard coded when going from HEVC to H264. And changing the CRF only affects the quality when using CPU transcoding. If you are using NVENC that setting won't do anything.

What OS are you using? What driver version? Can you share your ffmpeg logs via pastebin?

The 120Mbps setting in your client only tells it what the max allowed bit rate is. If you set the max bit rate to 6Mbps and your video is 10Mbps it will trigger a transcode and will lower the bit rate to 6Mbps.

[gnattu]

Bitrate been too conservative is a known issue in 10.8 and the limitation has being slightly lifted in 10.9, which should looks better.

[tokki]

CRF only affects the quality when using CPU transcoding. If you are using NVENC that setting won't do anything.

Oh I see, this explains why
I'm on windows, with the Nvidia STUDIO driver 551.23 (I'm going to update soon to 552.22) and Jellyfin version 10.8.13
Here is the FFMPEG log
https://pastebin.com/embed_iframe/Gy24U6Kr?theme=dark

Bitrate been too conservative is a known issue in 10.8 and the limitation has being slightly lifted in 10.9, which should looks better.

Yeah if I remember correctly on my tests made one the 10.8, the bitrate was set to more or less exactly the same bitrate as the original one.
The recent update apparently changed that by 1.5x, that is better but not enough, imo this should be better if linked with the user set parameter (no need to clamp at 1.5x if the user is asking up to 120Mbps)

So if I understand correctly, there is not much I can do if this is hard coded ?

[TheDreadPirate]

Correct. As gnattu said, this is changed in 10.9, which is right around the corner. So you shouldn't have to deal with this for too long.

It looks like your screen is 1080P based on what you posted in the opening. Most clients there is a setting to limit the transcode resolution. If you set that explicitly to 1080P that should look much better at 11Mbps.