Jellyfin Forum
Ahead-of-time x265 transcode looks much worse than Jellyfin QSV transcode - Printable Version

+- Jellyfin Forum (https://forum.jellyfin.org)
+-- Forum: Off Topic (https://forum.jellyfin.org/f-off-topic)
+--- Forum: Media (https://forum.jellyfin.org/f-media)
+--- Thread: Ahead-of-time x265 transcode looks much worse than Jellyfin QSV transcode (/t-ahead-of-time-x265-transcode-looks-much-worse-than-jellyfin-qsv-transcode)

Ahead-of-time x265 transcode looks much worse than Jellyfin QSV transcode - StaticUnit97 - 2025-01-03

Not sure if this is the best place for this, since it's not really a Jellyfin issue.

One user of my Jellyfin server has to stream at bitrates ~8mbps due to connection limitations at times. For the movies and shows they enjoy, I was going to transcode these files ahead of time using Handbrake with x265 to ideally preserve as much of the quality as possible.

As a test, I took an existing movie encoded at 1080p 40 mbps with H.264 and re-encoded it to H.265 using the x265 10-bit encoder. I used RF 24, slow preset, and no tune, and the resulting average bitrate was 6.8 mbps which was perfect.

However, the resulting quality of the video looks much worse than just using Jellyfin's on-the-fly transcoding using H.265 QSV and forcing a bitrate of 5mbps.
How can I improve the H.265 x265 transcode? I assume that the settings Jellyfin uses with its version of ffmpeg have been pretty well tuned, but I was not expecting such a drastic difference in quality. Any help is appreciated!

RE: Ahead-of-time x265 transcode looks much worse than Jellyfin QSV transcode - Efficient_Good_5784 - 2025-01-03

From my own personal experience and bias, using the GPU to transcode almost always results in worse video quality per bitrate used when compared to using the CPU.

What you consider better may just be something you're used to that feels better to you. It would be helpful if you could share PNG screenshot examples of both encodes to have us better understand what you're experiencing.

Also as a note that I want to add. Unlike H264 where going with a slower preset lowers the final output size, going with slower presets for H265 actually increases your final video size. Slower presets in H265 use more complex algorithms to better capture movement detail and other things which use more data.

That said, if you care about quality, the slow preset is recommended to be used as a minimum like you are.

RE: Ahead-of-time x265 transcode looks much worse than Jellyfin QSV transcode - bitmap - 2025-01-03

This might belong better in the off-topic ffmpeg megathread that TDP and I resurrect on occasion, but there are a TON of x265 params that you can set with ffmpeg, so playing with some of those might be the answer. I'm much more familiar with the QSV encoders or libsvtav1. For example, here's the output of ffmpeg -h encoder=libx265:

Code:
Encoder libx265 [libx265 H.265 / HEVC]:
    General capabilities: dr1 delay threads
    Threading capabilities: other
    Supported pixel formats: yuv420p yuvj420p yuv422p yuvj422p yuv444p yuvj444p gbrp yuv420p10le yuv422p10le yuv444p10le gbrp10le yuv420p12le yuv422p12le yuv444p12le gbrp12le gray gray10le gray12le
libx265 AVOptions:
  -crf              <float>      E..V....... set the x265 crf (from -1 to FLT_MAX) (default -1)
  -qp                <int>        E..V....... set the x265 qp (from -1 to INT_MAX) (default -1)
  -forced-idr        <boolean>    E..V....... if forcing keyframes, force them as IDR frames (default false)
  -preset            <string>    E..V....... set the x265 preset
  -tune              <string>    E..V....... set the x265 tune parameter
  -profile          <string>    E..V....... set the x265 profile
  -udu_sei          <boolean>    E..V....... Use user data unregistered SEI if available (default false)
  -a53cc            <boolean>    E..V....... Use A53 Closed Captions (if available) (default true)
  -x265-params      <dictionary> E..V....... set the x265 configuration using a :-separated list of key=value parameters

The key here is the x265-params option, which allows you to utilize the libx265 command line parameters which provide a lot of flexibility. I'll try to be as general as possible below:
  • Don't encode the whole thing, pick a chunk that is awful and re-encode that as part of trial and error.
  • I would try encoding with hevc_qsv if you can -- which it sounds like you're able to do in real-time. You may have to learn some ffmpeg, but it's not as daunting as you'd think. I'm pretty sure Handbrake supports the QSV encoders as well (maybe not AV1, but should support hevc_qsv).
  • Preset matters. The default for ffmpeg is 4 (medium), which I've found provides very poor results. I heard that this matters less (or not at all) for QSV encodes, but if you're using libx265, it matters quite a bit. Bump it down to slow and see what you get. If not satisfied, go with slower and see what you get.
  • You might even go with libsvtav1 if you can stand longer encode times, depending on your hardware. I have a template that provides good visual clarity as well as flexibility with bit rates around 2000-6000 kbps which seems to be what you're aiming for.
  • If you're aiming for a specific bit rate, you should probably consider running two-pass encoding for these files. It will take twice as long, but you can set target bit rate as well as max bit rate for the encode. These options will have the encoder run a first pass to get more information about the media to try and maximize quality with the bit rate you've allocated. The quality is generally still worse than CRF or QP encodes, however, it will be better than single-pass encoding if you're targeting bit rate.

I think part of this depends on the route you want to go: Handbrake or ffmpeg? HEVC or AV1? Hardware-accelerated or not? What sort of hardware are you running these encodes on?

RE: Ahead-of-time x265 transcode looks much worse than Jellyfin QSV transcode - StaticUnit97 - 2025-01-03

(2025-01-03, 06:52 PM)bitmap Wrote: I think part of this depends on the route you want to go: Handbrake or ffmpeg? HEVC or AV1? Hardware-accelerated or not? What sort of hardware are you running these encodes on?

Thanks for all the good info! Some replies to your final questions:

  1. No preference on Handbrake vs. ffmpeg. I'm very comfortable with CLI tools, I just have only used handbrake up to this point.
  2. Again, no preference. I'm interested in AV1 from a quality per bitrate standpoint, so I'd definitely be interested in that script you mentioned.
  3. I would imagine not HW accelerated? The goal of this is to maximize quality for a given bitrate, and it seems like software gives the best. The server is obviously using hardware acceleration when doing the real-time transcoding.
  4. I'm running this on a 5700X3D, so 8C/16T, on Windows. I'd probably use ffmpeg in WSL. The Jellyfin server has a i5-9500T, which is 6C/6T.

Thanks!

RE: Ahead-of-time x265 transcode looks much worse than Jellyfin QSV transcode - Efficient_Good_5784 - 2025-01-03

(2025-01-03, 07:56 PM)StaticUnit97 Wrote: I'm running this on a 5700X3D, so 8C/16T, on Windows. I'd probably use ffmpeg in WSL. The Jellyfin server has a i5-9500T, which is 6C/6T.
You can look at ffmpeg's website to find Windows builds of ffmpeg: https://ffmpeg.org//download.html

You just get that, unpack it, then all you need to do is navigate to it using command prompt and run it there.

RE: Ahead-of-time x265 transcode looks much worse than Jellyfin QSV transcode - bitmap - 2025-01-03

(2025-01-03, 10:21 PM)Efficient_Good_5784 Wrote: You can look at ffmpeg's website to find Windows builds of ffmpeg: https://ffmpeg.org//download.html

You just get that, unpack it, then all you need to do is navigate to it using command prompt and run it there.

I might recommend using jellyfin-ffmpeg. There's a win64 version in a ZIP file that should install similarly to vanilla ffmpeg. The reason for this is all of the included extras pre-compiled. Here is my "script" for libsvtav1 encodes which, fortunately, has an svtav1-params list that uses the same syntax as libx265 if you decide to experiment there.

Code:
MEDIA="MEDIA" && INPUT="INPUT" && \
cd "/mnt/media/DIRECTORY" && \ # change to source dir
mkdir -p "/mnt/media/encoded/${MEDIA}" && \ # change to destination dir
echo "Calculating crop values..." && \ # below is optional autocropping which can save bit rate and removes letterboxing
cropval=$(ffmpeg -ss 120 -i "${INPUT}" -t 9:00 -filter:v fps=1/2,cropdetect -f null - 2>&1 | awk '/crop/ { print $NF }' | tr ' ' '\n' | sort | uniq -c | sort -n | tail -1 | awk '{ print $NF }') && \
ffmpeg -analyzeduration 200M -probesize 1G \ # beginning of encoding
  -i "${INPUT}" \
  -map 0:v:0 -map 0:a:0 -map 0:s:0 -map 0:s:1 \ # map desired streams | ffmpeg default is first video + audio | replace w/ -map 0 for all streams
  -c copy -c:v:0 libsvtav1 -pix_fmt yuv420p10le \ # remove pix_fmt if 10-bit not desired
  -svtav1-params "preset=4:crf=23:tune=2:enable-variance-boost=1:film-grain=5:film-grain-denoise=1:lookahead=40" \ # recommend changing film-grain based on media ≤ 8 max
  -filter:v "${cropval}" \ # remove if not autocropping
  -c:a libopus -b:a 256k -ac 6 \ # change to desired audio codec, bit rate, channels
  # all metadata set below is optional
  -metadata title="${MEDIA}" -metadata:s:v:0 title='SVT-AV1 P4 CRF23 FGS5 FGD' \ # sets overall title + video stream title
  -metadata:s:a:0 title='5.1 Surround' -metadata:s:a:0 language='eng' -disposition:a:0 default \ # sets first mapped stream title + default
  -metadata:s:s:0 title='English' -metadata:s:s:0 language='eng' -disposition:s:0 0 \ # info for first mapped sub stream
  -metadata:s:s:1 title='English (SDH)' -metadata:s:s:1 language='eng' -disposition:s:1 0 \ # info for second mapped sub stream
  "/mnt/media/encoded/${MEDIA}/${MEDIA} [Bluray-1080p AV1 OPUS 5.1][EN][EN]-RLSGRP.mkv" && \ # output file

Customize to your needs and remove anything you don't want or will not use. Here is documentation on the libsvtav1 params. Keep in mind a few things:
  • If the client is not compatible with the AV1 codec, JF will still attempt to transcode the file. Test and/or research the client(s) being used.
  • Same goes for audio codec. Safest bet is AAC ≤ 6 channels (i.e., 5.1 surround). Other popular codecs include FLAC, AC3 (Dolby Digital), E-AC3 (Dolby Digital+).
  • Avoid re-encoding lossy audio or video -- that is already-encoded video or non-lossy audio formats. (Easier said than done...)
  • I would not recommend going lower than preset 3 for libsvtav1. I have noticed very little improvement and ~50% performance impact decreasing to preset 3.
  • Values above CRF 24 tend to have artifacting or macro-blocking but can save additional bit rate.

Remove all my comments before running, remove or replace all identified placeholders or parameters. Software encoding in AV1 is an extremely slow process. I have an i7-13700k and my speed is ~0.3-0.7x for most applications. Adding additional filters (e.g., smartblur, yadif) will impact performance. 

Ask any questions you have, I can try to answer. I know that nyanmisaka is our resident ffmpeg expert and others may have different opinions, but I'm quite happy with the simple setup above.