Transcoding bitrate limited to 12.8 Mbps

[esmorun]

I've recently noticed that when transcoding, the bitrate won't go higher than 12.9 Mbps, resulting in poor image quality. It will go lower if I set it to a lower value in the web UI, but setting it to a higher setting or automatic has no effect on bitrate. I've tested on two systems:

1:
Ubuntu 24.04
Jellyfin 10.10.6
ffmpeg 6.0.1-Jellyfin
Intel GPU

2:
Docker (on Ubuntu 24.10)
Jellyfin 10.10.6
ffmpeg 7.0.2-Jellyfin
AMD GPU

The ffmpeg log confirms the low bitrate (12685428 bps = 12.685428 mbps), this is from the Intel system:

Code:

/usr/lib/jellyfin-ffmpeg/ffmpeg -analyzeduration 200M -probesize 1G -f matroska -init_hw_device drm=dr:/dev/dri/renderD129 -init_hw_device vaapi=va@dr -init_hw_device vulkan=vk@dr -filter_hw_device vk -hwaccel vaapi -hwaccel_output_format vaapi -noautorotate -i file:"/media/Movies/Dune.mkv" -noautoscale -map_metadata -1 -map_chapters -1 -threads 0 -map 0:0 -map 0:1 -map -0:s -codec:v:0 h264_vaapi -rc_mode VBR -b:v [color=#f012be]12685428[/color] -maxrate [color=#ff30dc]12685428[/color] -bufsize 25370856 -profile:v:0 high -sei -a53_cc -force_key_frames:0 "expr:gte(t,n_forced*3)" -vf "setparams=color_primaries=bt2020:color_trc=smpte2084:colorspace=bt2020nc,hwmap=derive_device=drm,format=drm_prime,libplacebo=upscaler=none:downscaler=none:format=bgra:tonemapping=bt.2390:peak_detect=0:color_primaries=bt709:color_trc=bt709:colorspace=bt709,format=vulkan,hwmap=derive_device=vaapi,format=vaapi,scale_vaapi=format=nv12" -codec:a:0 libfdk_aac -ac 2 -ab 256000 -af "volume=2" -copyts -avoid_negative_ts disabled -max_muxing_queue_size 2048 -f hls -max_delay 5000000 -hls_time 3 -hls_segment_type fmp4 -hls_fmp4_init_filename "eeabbdabff582c442236fd1167776255-1.mp4" -start_number 0 -hls_segment_filename "/cache/transcodes/eeabbdabff582c442236fd1167776255%d.mp4" -hls_playlist_type vod -hls_list_size 0 -y "/cache/transcodes/eeabbdabff582c442236fd1167776255.m3u8"

I assumed it was related to HW acceleration at first but it is also limited when I do software encoding, I've set the crf value to 20 in the settings but it seems to be overruled by a maxrate. Here's an example:

Code:

/usr/lib/jellyfin-ffmpeg/ffmpeg -analyzeduration 200M -probesize 1G -ss 00:01:18.000 -noaccurate_seek  -i file:"/media/Movies/Dune.mkv" -map_metadata -1 -map_chapters -1 -threads 0 -map 0:0 -map 0:1 -map -0:s -codec:v:0 libx264 -preset veryfast -crf 20 -maxrate [color=#d900a7]12685428[/color] -bufsize 25370856 -profile:v:0 high -level 51 -x264opts:0 subme=0:me_range=16:rc_lookahead=10:me=hex:open_gop=0 -force_key_frames:0 "expr:gte(t,n_forced*3)" -sc_threshold:v:0 0 -vf "setparams=color_primaries=bt2020:color_trc=smpte2084:colorspace=bt2020nc,scale=trunc(min(max(iw\,ih*a)\,min(3840\,1606*a))/2)*2:trunc(min(max(iw/a\,ih)\,min(3840/a\,1606))/2)*2,tonemapx=tonemap=bt2390:desat=0:peak=100:t=bt709:m=bt709:p=bt709:format=yuv420p" -codec:a:0 libfdk_aac -ac 2 -ab 256000 -af "volume=2" -copyts -avoid_negative_ts disabled -max_muxing_queue_size 2048 -f hls -max_delay 5000000 -hls_time 3 -hls_segment_type fmp4 -hls_fmp4_init_filename "2a6b9453dfbbab8c7ca25d25e1e46aac-1.mp4" -start_number 26 -hls_segment_filename "/cache/transcodes/2a6b9453dfbbab8c7ca25d25e1e46aac%d.mp4" -hls_playlist_type vod -hls_list_size 0 -y "/cache/transcodes/2a6b9453dfbbab8c7ca25d25e1e46aac.m3u8"

In the past, my experience has been that Jellyfin will max out at the same bitrate as the source file when doing hevc --> hevc, our double the bitrate if doing hevc --> h264.

Is there a reason for this change? Is it a bug, or am I missing something obvious?