Low bitrate for rare frames during transcoding

[vessd]

Hello

I'm using Jellyfin with an AMD Radeon RX 6600 for transcoding on Arch Linux with VAAPI for my Android TV

jellyfin-ffmpeg 1:6.0.1p1-2
jellyfin-server 10.8.13-1
jellyfin-androidtv 0.16.7

Most of the time the picture quality is good, but sometimes there are low quality frames.
Rare
   
Most of the time
   

It doesn't look like I'm running out of resources for transcoding, the system is almost idle
       

Are there any settings I could try to change? As I understand it, the CRF parameter does not affect the quality of the VAAPI encoder
Transcoding settings
   
FFmpeg log
  FFmpeg.Transcode-2024-03-25_21-53-30.txt (Size: 91.62 KB / Downloads: 33)

[nyanmisaka]

AMD's rate control in encoding H264 and HEVC is terrible. I don't think they can encode a decent 1080p HEVC stream at 4000kbps. VA-API is even worse than Windows AMF.
Especially when the content of the picture changes drastically. We should probably increase the default bitrate multiplier for AMD.

You can try playing around with these parameters in the ffmpeg command line and give your feedback. -rc_mode VBR -b:v 4359887 -maxrate 4359887 -bufsize 8719774

[Fate]

AMD's rate control in encoding H264 and HEVC is terrible. I don't think they can encode a decent 1080p HEVC stream at 4000kbps. VA-API is even worse than Windows AMF.
Especially when the content of the picture changes drastically. We should probably increase the default bitrate multiplier for AMD.

You can try playing around with these parameters in the ffmpeg command line and give your feedback. -rc_mode VBR -b:v 4359887 -maxrate 4359887 -bufsize 8719774

I have to agree here somewhat.
AMDs ratecontrol is really struggling with ratecontrol in VBR/VBV.
Increasing bufsize should help a bit.

[vessd]

Thanks, I tried increasing the buffer and it fixed this case.
It looks like right now the buffer size is the bitrate x2, I increased it to x3 (-rc_mode VBR -b:v 4359887 -maxrate 4359887 -bufsize 13079661)

Before
   
File size: 970 KiB
Duration: 2 s 961 ms
Overall bit rate mode: Variable
Overall bit rate: 2 616 kb/s

After
   
File size: 1.66 MiB
Duration: 2 s 961 ms
Overall bit rate mode: Variable
Overall bit rate: 4 511 kb/s

In general, I did not notice an increase in resource consumption, only transcoded files began to take up a little more space.

Before

Code:

$ du -sh /media/transcoding
802M /media/transcoding

After

Code:

$ du -sh /media/transcoding
838M /media/transcoding

Should I look for a minimum buffer size that will fix this case? 

P.S. Code block formatting on this forum is a joke