Transcoding bitrate limited to 12.8 Mbps

[TheDreadPirate]

When transcoding Jellyfin has static multipliers for codec to codec conversion. Something like 1.4x when going from HEVC to H264 to account for the lower encoding efficiency of H264. So a HEVC video with a bit rate of 10Mbps will get a bit rate of 14Mbps when transcoded to H264. And the multiplier varies depending on the original vs new codec used. But the resulting transcode's bit rate will be the original video bitrate times that multiplier. And that cannot be changed.

Regarding the low quality you are experiencing, it depends on which GPU you are using. AMD's H264 hardware encoder is trash. Period. Their HEVC encoder is ok. As is their AV1 encoder. The issue isn't bit rate. You can throw more bits at an AMD GPU encoding to H264 and the quality will still be trash.

The same bit rate on modern Nvidia and Intel GPUs would result is much higher quality, regardless of codec.