Ahead-of-time x265 transcode looks much worse than Jellyfin QSV transcode

[bitmap]

This might belong better in the off-topic ffmpeg megathread that TDP and I resurrect on occasion, but there are a TON of x265 params that you can set with ffmpeg, so playing with some of those might be the answer. I'm much more familiar with the QSV encoders or libsvtav1. For example, here's the output of ffmpeg -h encoder=libx265:

Code:

Encoder libx265 [libx265 H.265 / HEVC]:
    General capabilities: dr1 delay threads
    Threading capabilities: other
    Supported pixel formats: yuv420p yuvj420p yuv422p yuvj422p yuv444p yuvj444p gbrp yuv420p10le yuv422p10le yuv444p10le gbrp10le yuv420p12le yuv422p12le yuv444p12le gbrp12le gray gray10le gray12le
libx265 AVOptions:
  -crf              <float>      E..V....... set the x265 crf (from -1 to FLT_MAX) (default -1)
  -qp                <int>        E..V....... set the x265 qp (from -1 to INT_MAX) (default -1)
  -forced-idr        <boolean>    E..V....... if forcing keyframes, force them as IDR frames (default false)
  -preset            <string>    E..V....... set the x265 preset
  -tune              <string>    E..V....... set the x265 tune parameter
  -profile          <string>    E..V....... set the x265 profile
  -udu_sei          <boolean>    E..V....... Use user data unregistered SEI if available (default false)
  -a53cc            <boolean>    E..V....... Use A53 Closed Captions (if available) (default true)
  -x265-params      <dictionary> E..V....... set the x265 configuration using a :-separated list of key=value parameters

The key here is the x265-params option, which allows you to utilize the libx265 command line parameters which provide a lot of flexibility. I'll try to be as general as possible below:

• Don't encode the whole thing, pick a chunk that is awful and re-encode that as part of trial and error.
  
• I would try encoding with hevc_qsv if you can -- which it sounds like you're able to do in real-time. You may have to learn some ffmpeg, but it's not as daunting as you'd think. I'm pretty sure Handbrake supports the QSV encoders as well (maybe not AV1, but should support hevc_qsv).
  
• Preset matters. The default for ffmpeg is 4 (medium), which I've found provides very poor results. I heard that this matters less (or not at all) for QSV encodes, but if you're using libx265, it matters quite a bit. Bump it down to slow and see what you get. If not satisfied, go with slower and see what you get.
  
• You might even go with libsvtav1 if you can stand longer encode times, depending on your hardware. I have a template that provides good visual clarity as well as flexibility with bit rates around 2000-6000 kbps which seems to be what you're aiming for.
  
• If you're aiming for a specific bit rate, you should probably consider running two-pass encoding for these files. It will take twice as long, but you can set target bit rate as well as max bit rate for the encode. These options will have the encoder run a first pass to get more information about the media to try and maximize quality with the bit rate you've allocated. The quality is generally still worse than CRF or QP encodes, however, it will be better than single-pass encoding if you're targeting bit rate.
  

I think part of this depends on the route you want to go: Handbrake or ffmpeg? HEVC or AV1? Hardware-accelerated or not? What sort of hardware are you running these encodes on?