Transcoding bitrate limited to 12.8 Mbps

[esmorun]

I've recently noticed that when transcoding, the bitrate won't go higher than 12.9 Mbps, resulting in poor image quality. It will go lower if I set it to a lower value in the web UI, but setting it to a higher setting or automatic has no effect on bitrate. I've tested on two systems:

1:
Ubuntu 24.04
Jellyfin 10.10.6
ffmpeg 6.0.1-Jellyfin
Intel GPU

2:
Docker (on Ubuntu 24.10)
Jellyfin 10.10.6
ffmpeg 7.0.2-Jellyfin
AMD GPU

The ffmpeg log confirms the low bitrate (12685428 bps = 12.685428 mbps), this is from the Intel system:

Code:

/usr/lib/jellyfin-ffmpeg/ffmpeg -analyzeduration 200M -probesize 1G -f matroska -init_hw_device drm=dr:/dev/dri/renderD129 -init_hw_device vaapi=va@dr -init_hw_device vulkan=vk@dr -filter_hw_device vk -hwaccel vaapi -hwaccel_output_format vaapi -noautorotate -i file:"/media/Movies/Dune.mkv" -noautoscale -map_metadata -1 -map_chapters -1 -threads 0 -map 0:0 -map 0:1 -map -0:s -codec:v:0 h264_vaapi -rc_mode VBR -b:v [color=#f012be]12685428[/color] -maxrate [color=#ff30dc]12685428[/color] -bufsize 25370856 -profile:v:0 high -sei -a53_cc -force_key_frames:0 "expr:gte(t,n_forced*3)" -vf "setparams=color_primaries=bt2020:color_trc=smpte2084:colorspace=bt2020nc,hwmap=derive_device=drm,format=drm_prime,libplacebo=upscaler=none:downscaler=none:format=bgra:tonemapping=bt.2390:peak_detect=0:color_primaries=bt709:color_trc=bt709:colorspace=bt709,format=vulkan,hwmap=derive_device=vaapi,format=vaapi,scale_vaapi=format=nv12" -codec:a:0 libfdk_aac -ac 2 -ab 256000 -af "volume=2" -copyts -avoid_negative_ts disabled -max_muxing_queue_size 2048 -f hls -max_delay 5000000 -hls_time 3 -hls_segment_type fmp4 -hls_fmp4_init_filename "eeabbdabff582c442236fd1167776255-1.mp4" -start_number 0 -hls_segment_filename "/cache/transcodes/eeabbdabff582c442236fd1167776255%d.mp4" -hls_playlist_type vod -hls_list_size 0 -y "/cache/transcodes/eeabbdabff582c442236fd1167776255.m3u8"

I assumed it was related to HW acceleration at first but it is also limited when I do software encoding, I've set the crf value to 20 in the settings but it seems to be overruled by a maxrate. Here's an example:

Code:

/usr/lib/jellyfin-ffmpeg/ffmpeg -analyzeduration 200M -probesize 1G -ss 00:01:18.000 -noaccurate_seek  -i file:"/media/Movies/Dune.mkv" -map_metadata -1 -map_chapters -1 -threads 0 -map 0:0 -map 0:1 -map -0:s -codec:v:0 libx264 -preset veryfast -crf 20 -maxrate [color=#d900a7]12685428[/color] -bufsize 25370856 -profile:v:0 high -level 51 -x264opts:0 subme=0:me_range=16:rc_lookahead=10:me=hex:open_gop=0 -force_key_frames:0 "expr:gte(t,n_forced*3)" -sc_threshold:v:0 0 -vf "setparams=color_primaries=bt2020:color_trc=smpte2084:colorspace=bt2020nc,scale=trunc(min(max(iw\,ih*a)\,min(3840\,1606*a))/2)*2:trunc(min(max(iw/a\,ih)\,min(3840/a\,1606))/2)*2,tonemapx=tonemap=bt2390:desat=0:peak=100:t=bt709:m=bt709:p=bt709:format=yuv420p" -codec:a:0 libfdk_aac -ac 2 -ab 256000 -af "volume=2" -copyts -avoid_negative_ts disabled -max_muxing_queue_size 2048 -f hls -max_delay 5000000 -hls_time 3 -hls_segment_type fmp4 -hls_fmp4_init_filename "2a6b9453dfbbab8c7ca25d25e1e46aac-1.mp4" -start_number 26 -hls_segment_filename "/cache/transcodes/2a6b9453dfbbab8c7ca25d25e1e46aac%d.mp4" -hls_playlist_type vod -hls_list_size 0 -y "/cache/transcodes/2a6b9453dfbbab8c7ca25d25e1e46aac.m3u8"

In the past, my experience has been that Jellyfin will max out at the same bitrate as the source file when doing hevc --> hevc, our double the bitrate if doing hevc --> h264.

Is there a reason for this change? Is it a bug, or am I missing something obvious?

[esmorun]

Looks like it's just me being stupid, the videos I tested was much lower bitrate than I realized, so what I was seeing was about double the bitrate... The quality of the transcode was much worse than the original though, would have been nice to have a setting to allow it to be more than twice of the source file

[TheDreadPirate]

When transcoding Jellyfin has static multipliers for codec to codec conversion. Something like 1.4x when going from HEVC to H264 to account for the lower encoding efficiency of H264. So a HEVC video with a bit rate of 10Mbps will get a bit rate of 14Mbps when transcoded to H264. And the multiplier varies depending on the original vs new codec used. But the resulting transcode's bit rate will be the original video bitrate times that multiplier. And that cannot be changed.

Regarding the low quality you are experiencing, it depends on which GPU you are using. AMD's H264 hardware encoder is trash. Period. Their HEVC encoder is ok. As is their AV1 encoder. The issue isn't bit rate. You can throw more bits at an AMD GPU encoding to H264 and the quality will still be trash.

The same bit rate on modern Nvidia and Intel GPUs would result is much higher quality, regardless of codec.

[Efficient_Good_5784]

That's not exactly how transcoding works.

You cannot make a video look better with simple transcoding.
A transcode will always look worse than the original as you're throwing out data to make the new transcode.
You can increase the bitrate to compensate, but it's wasteful and you will still be losing data in the end.

Jellyfin will only show available bitrates to transcode to that are no larger than the video's actual bitrate.
That said, it will show an option that's larger than the video bitrate because the original bitrate could be just enough past one limit and into the next.

As an example, if you have a video that's 11Mbps, Jellyfin may show you an option to transcode up to 20Mbps.
Jellyfin would not stop at 10Mbps because the original is past it, so in an attempt to allow you to transcode with the original bitrate, it gives you an option of 20Mbps.
If the original bitrate was 10Mbps or less, it would cap the options to 10Mbps and lower.