SOLVED: Transcodes to HEVC stutters while H.264 does not

[dekomoon]

Thanks for the help previously, but I am now encountering another problem...

Transcoding to H.264 works perfectly, the resulting bitrate is 35mbps and video playback is smooth on the Jellyfin Media Player on my M1 MBA. However, video quality is really really poor. I can not figure out how to give the transcoded H.264 more bitrate than 35mbps. I am guessing 35mbps is a hard limit because the source video is also 35mbps? Is there a way to go above that limit since H.264 is less efficient than HEVC and AV1?

I also tried HEVC, but for some reason the video stutters even though transcoding fps is above 60 (I have provided a log of the ffmpeg transcode below, and also my docker-compose.yml file) on both my linux server and Mac's Jellyfin Media Player. It only stutters when HEVC is playing at a bitrate of 35mbps, when I limit the video to 1080p 20mbps, no stuttering happens. Why is the stuttering / frame drop happening for videos transcoded to HEVC above 20mbps?

Thanks!

Server
- OS: PikaOS (Ubuntu-based)
- Jellyfin Build: Podman, LinuxServer Docker image w/ latest tag & AMD VA-API mod.
- Install Method: podman-compose as root / superuser

[TheDreadPirate]

Regarding the quality issue. It is likely the AMD GPU. Even Jellyfin's documentation says transcoding to H264 is not great with AMD.

https://jellyfin.org/docs/general/admini...nd-quality

I have an Intel CPU, so I am using QSV instead of VAAPI. My media is mostly 1080p HEVC and not 4k60, but when it gets transcoded to lower bit rate H264 (not at home + cellular) the quality is shockingly close to the 10-15Mbps HEVC source.

So the fact that 35Mbps H264 the quality is "really really poor" tells me it's the AMD GPU. And my recent experience with AMD GPUs (RX6800) and encoding via Handbrake reinforces what the Jellyfin link above states about AMD GPU based encoding. So much so that I opted to use my old PC with a GTX 1070 as my HEVC encoding box.

[dekomoon]

I kind of regret getting this gpu by now, but hopefully there is a way to increase the output bitrate for h.264, or remove the stuttering for HEVC since the HEVC playback looks good in terms of quality

[Venson]

@dekomoon

Recently i heard some good things about AMD-VAAPI, you might want to try that out.

[dekomoon]

Hey thanks, but I was using AMD VAAPI in my container. I have decided to buy an RTX 3050 and save myself the trouble of dealing with AMD.

[nyanmisaka]

AMD's H.264 encoder has not been improved over the years. Better to use HEVC and AV1 encoder instead.

As for the stuttering issue, I think software decoding is used on the client side, which leads to the frame dropping(stuttering) in 4k@60fps.


Stream #0:0: Video: hevc (Main) (hvc1 / 0x3163766, vaapi(tv, bt709, progressive), 3840x2160 [SAR 1:1 DAR 16:9], q=2-31, 35884 kb/s, 60 fps, 90k tbn
Metadata:
encoder : Lavc59.37.100 hevc_vaapi


[dekomoon]

AMD's H.264 encoder has not been improved over the years. Better to use HEVC and AV1 encoder instead.

As for the stuttering issue, I think software decoding is used on the client side, which leads to the frame dropping(stuttering) in 4k@60fps.


  Stream #0:0: Video: hevc (Main) (hvc1 / 0x3163766, vaapi(tv, bt709, progressive), 3840x2160 [SAR 1:1 DAR 16:9], q=2-31, 35884 kb/s, 60 fps, 90k tbn
    Metadata:
      encoder        : Lavc59.37.100 hevc_vaapi


Thanks, if it is a client issue that is pretty odd considering the cpu on my client maintains below 10% usage. I'll try to use H.264 instead, but is there anyway to increase the resulting transcode from 35mbps to say 50mbps or higher?