Ahead-of-time x265 transcode looks much worse than Jellyfin QSV transcode

[bitmap]

You can look at ffmpeg's website to find Windows builds of ffmpeg: https://ffmpeg.org//download.html

You just get that, unpack it, then all you need to do is navigate to it using command prompt and run it there.

I might recommend using jellyfin-ffmpeg. There's a win64 version in a ZIP file that should install similarly to vanilla ffmpeg. The reason for this is all of the included extras pre-compiled. Here is my "script" for libsvtav1 encodes which, fortunately, has an svtav1-params list that uses the same syntax as libx265 if you decide to experiment there.

Code:

MEDIA="MEDIA" && INPUT="INPUT" && \
cd "/mnt/media/DIRECTORY" && \ # change to source dir
mkdir -p "/mnt/media/encoded/${MEDIA}" && \ # change to destination dir
echo "Calculating crop values..." && \ # below is optional autocropping which can save bit rate and removes letterboxing
cropval=$(ffmpeg -ss 120 -i "${INPUT}" -t 9:00 -filter:v fps=1/2,cropdetect -f null - 2>&1 | awk '/crop/ { print $NF }' | tr ' ' '\n' | sort | uniq -c | sort -n | tail -1 | awk '{ print $NF }') && \
ffmpeg -analyzeduration 200M -probesize 1G \ # beginning of encoding
  -i "${INPUT}" \
  -map 0:v:0 -map 0:a:0 -map 0:s:0 -map 0:s:1 \ # map desired streams | ffmpeg default is first video + audio | replace w/ -map 0 for all streams
  -c copy -c:v:0 libsvtav1 -pix_fmt yuv420p10le \ # remove pix_fmt if 10-bit not desired
  -svtav1-params "preset=4:crf=23:tune=2:enable-variance-boost=1:film-grain=5:film-grain-denoise=1:lookahead=40" \ # recommend changing film-grain based on media ≤ 8 max
  -filter:v "${cropval}" \ # remove if not autocropping
  -c:a libopus -b:a 256k -ac 6 \ # change to desired audio codec, bit rate, channels
  # all metadata set below is optional
  -metadata title="${MEDIA}" -metadata:s:v:0 title='SVT-AV1 P4 CRF23 FGS5 FGD' \ # sets overall title + video stream title
  -metadata:s:a:0 title='5.1 Surround' -metadata:s:a:0 language='eng' -disposition:a:0 default \ # sets first mapped stream title + default
  -metadata:s:s:0 title='English' -metadata:s:s:0 language='eng' -disposition:s:0 0 \ # info for first mapped sub stream
  -metadata:s:s:1 title='English (SDH)' -metadata:s:s:1 language='eng' -disposition:s:1 0 \ # info for second mapped sub stream
  "/mnt/media/encoded/${MEDIA}/${MEDIA} [Bluray-1080p AV1 OPUS 5.1][EN][EN]-RLSGRP.mkv" && \ # output file

Customize to your needs and remove anything you don't want or will not use. Here is documentation on the libsvtav1 params. Keep in mind a few things:

• If the client is not compatible with the AV1 codec, JF will still attempt to transcode the file. Test and/or research the client(s) being used.
  
• Same goes for audio codec. Safest bet is AAC ≤ 6 channels (i.e., 5.1 surround). Other popular codecs include FLAC, AC3 (Dolby Digital), E-AC3 (Dolby Digital+).
  
• Avoid re-encoding lossy audio or video -- that is already-encoded video or non-lossy audio formats. (Easier said than done...)
  
• I would not recommend going lower than preset 3 for libsvtav1. I have noticed very little improvement and ~50% performance impact decreasing to preset 3.
  
• Values above CRF 24 tend to have artifacting or macro-blocking but can save additional bit rate.
  

Remove all my comments before running, remove or replace all identified placeholders or parameters. Software encoding in AV1 is an extremely slow process. I have an i7-13700k and my speed is ~0.3-0.7x for most applications. Adding additional filters (e.g., smartblur, yadif) will impact performance. 

Ask any questions you have, I can try to answer. I know that nyanmisaka is our resident ffmpeg expert and others may have different opinions, but I'm quite happy with the simple setup above.