Encoding Discussion Megathread (ffmpeg, Handbrake, AV1, etc...)

[bitmap]

Okay, here we go @TheDreadPirate...

Made a few improvements and a lot of mistakes along the way. Testing consisted of six files, ranked from A-F based on bitrate over time and resolution (extremely subjective). Essentially: would I be satisfied with this media were I to have it in my collection. Spoilers, I do and, for the most part, the grades aren't good. I chose a variety of files to show how the new script works. *This test was completed using the hevc_qsv codec, CBR 4500 kbps (x2 for maxrate, x4 for bufsize), plus adaptive turned on.

Code:

user@server:encoding$ ls source_files/mixed_media/
total 7.7G
drwxrwxr-x 2 user group 4.0K Sep 21 10:52 .
drwxrwxr-x 3 user group 4.0K Sep 21 10:52 ..
-rw-rw-r-- 1 user group 4.1G Sep 21 02:03 A.27100kbps.AVC.21m40s.24FPS-1920x1080.mkv
-rw-r--r-- 1 user group 2.5G Sep 21 02:08 B.6385kbps.VP9.55m11s.30FPS-3840x2160.mkv
-rw-r--r-- 1 user group 484M Sep 21 02:08 C.1236kbps.VP9.54m40s.30FPS-1280x720.mkv
-rw-rw-r-- 1 user group 274M Sep 21 02:08 D.707kbps.AVC.54m5s.30FPS-640x480.mp4
-rw-r--r-- 1 user group 119M Sep 21 02:08 F.304kbps.VP9.54m29s.30FPS-468x360.mkv
-rw-rw-r-- 1 user group 194M Sep 21 02:00 F.482kbps.HEVC.56m2s.24FPS-640x480.mkv

They go from very high bitrate to very low bitrate. The revised script handles files in this way:

1. Set your options. Easily available user-set options are: adaptive bitrate, QSV device, audio codec (encode), video codec (encode), bitrate reduction factor (percentage), bitrate floor (kbps), paths, streams, desired CBR, output file extension, primary audio stream bitrate, and video encoder preset. Note that many other options can be set, but some may cause unexpected behavior (e.g., CLEAN or AFLTR) and others may break the script entirely (e.g., setting audio or sub streams that do not exist in target file sets). Many of the parameters are optional, but until I get a way to un-scriptify this project, you just have a delete what you don't want.
   
   
2. Determine if adaptive bit rate is enabled. If so, use ffprobe (must be in PATH) to determine bitrate ($FFVR bps --> $CKVR kbps). Note that this works with MOST files. The ffprobe command used here works even with files that store this in the main header of the file (e.g., older HEVC MKVs or poorly encoded files), but many will return the overall bitrate, not just video bitrate. I recommend taking this into account when deciding on $VMIN as well as $REDX and accounting for the possibility of audio being included in the value returned.
   
   
3. If the media's bitrate falls below $VMIN, set the adaptive bitrate to the measured bitrate of the original media. If the media's bitrate exceeds the user-specified maximum desired video bitrate ($VRATE), set the bitrate for
   encoding to $VRATE. You can do large batches, ensuring crap-quality files are adapted and kept near original bitrate while larger files are reduced to your desired size/quality profile based on the guard rails that you set.
   
   
4. Solidify the encoding options and run through ffmpeg with the first pass using ONLY video (you don't need two-pass audio). Verify that existing options map or refer to the correct data in the correct order, uncomment any mappings for streams needed in your encode, and remove any lines, maps, or metadata not necessary to complete the encoding process. Understand that the wider net you cast, the more opportunity for error presents itself, particularly when running batch operations. This is also where you should set any metadata you want present. You can review metadata with ffprobe and/or mediainfo.
   
   
5. Pass one feeds into pass two, which maps all the other streams requested and finishes the process, then loops back until no more media is found.
   

Here's the script in action. I can't figure out how to suppress the libva output...if anybody knows how, let me know...

Code:

Attempting to find adaptive bit rate for mixed media...
Bit rate over specified limit, reverting to 4500 kbps...
Beginning first encoding pass for A.27100kbps.AVC.21m40s.24FPS-1920x1080. Starting file size: 4402MB...
[...]
    frame=31187 fps=179 q=-0.0 Lsize=  603953kB time=00:21:40.59 bitrate=3804.1kbits/s speed=7.47x   
First pass complete, beginning second encoding pass...
[...]
    frame=31187 fps=163 q=-0.0 Lsize=  631850kB time=00:21:40.75 bitrate=3979.3kbits/s speed=6.78x   
Finished encoding A.27100kbps.AVC.21m40s.24FPS-1920x1080. Final file size: 648MB
***
Attempting to find adaptive bit rate for mixed media...
Bit rate over specified limit, reverting to 4500 kbps...
Beginning first encoding pass for B.6385kbps.VP9.55m11s.30FPS-3840x2160. Starting file size: 2644MB...
[...]
    frame=99354 fps= 69 q=-0.0 Lsize= 1786053kB time=00:55:11.66 bitrate=4418.1kbits/s speed=2.31x   
First pass complete, beginning second encoding pass...
[...]
    frame=99354 fps= 66 q=-0.0 Lsize= 1865920kB time=00:55:11.82 bitrate=4615.5kbits/s speed=2.19x   
Finished encoding B.6385kbps.VP9.55m11s.30FPS-3840x2160. Final file size: 1911MB
***
Attempting to find adaptive bit rate for mixed media...
Bit rate will be reduced to 90% of original...
Bit rate set for encoding: 1112.62 kbps...
Beginning first encoding pass for C.1236kbps.VP9.54m40s.30FPS-1280x720. Starting file size: 507MB...
[...]
    frame=98308 fps=387 q=-0.0 Lsize=  429457kB time=00:54:40.07 bitrate=1072.6kbits/s speed=12.9x   
First pass complete, beginning second encoding pass...
[...]
    frame=98308 fps=359 q=-0.0 Lsize=  479925kB time=00:54:40.20 bitrate=1198.6kbits/s speed=  12x   
Finished encoding C.1236kbps.VP9.54m40s.30FPS-1280x720. Final file size: 492MB
***
Attempting to find adaptive bit rate for mixed media...
Bit rate will be reduced to 90% of original...
Bit rate set for encoding: 636.508 kbps...
Beginning first encoding pass for D.707kbps.AVC.54m5s.30FPS-640x480. Starting file size: 287MB...
[...]
    frame=97269 fps=711 q=-0.0 Lsize=  242347kB time=00:54:05.40 bitrate= 611.7kbits/s speed=23.7x   
First pass complete, beginning second encoding pass...
[...]
    frame=97269 fps=629 q=-0.0 Lsize=  293114kB time=00:54:05.63 bitrate= 739.8kbits/s speed=  21x   
Finished encoding D.707kbps.AVC.54m5s.30FPS-640x480. Final file size: 301MB
***
Attempting to find adaptive bit rate for mixed media...
Bit rate too low, not reducing further...
Bit rate set for encoding: 303.592 kbps...
Beginning first encoding pass for F.304kbps.VP9.54m29s.30FPS-468x360. Starting file size: 125MB...
[...]
    frame=97994 fps=988 q=-0.0 Lsize=  114260kB time=00:54:29.59 bitrate= 286.3kbits/s speed=  33x   
First pass complete, beginning second encoding pass...
[...]
    frame=97994 fps=774 q=-0.0 Lsize=  166591kB time=00:54:29.74 bitrate= 417.4kbits/s speed=25.8x   
Finished encoding F.304kbps.VP9.54m29s.30FPS-468x360. Final file size: 171MB
***
Attempting to find adaptive bit rate for mixed media...
Bit rate too low, not reducing further...
Bit rate set for encoding: 482.033 kbps...
Beginning first encoding pass for F.482kbps.HEVC.56m2s.24FPS-640x480. Starting file size: 203MB...
[...]
    frame=78235 fps=641 q=-0.0 Lsize=  194150kB time=00:56:02.08 bitrate= 473.1kbits/s speed=27.6x   
First pass complete, beginning second encoding pass...
[...]
    frame=78235 fps=562 q=-0.0 Lsize=  237283kB time=00:56:02.08 bitrate= 578.2kbits/s speed=24.1x   
Finished encoding F.482kbps.HEVC.56m2s.24FPS-640x480. Final file size: 243MB
***

Minimal informational output that delays slightly for readability. Here's the end result of the same files as above:

Code:

user@server:encoding$ ls batch_output/hevc_qsv/
total 3.6G
drwxrwxr-x 2 user group 4.0K Sep 21 12:07  .
drwxrwxr-x 3 user group 4.0K Sep 21 03:01  ..
-rw-rw-r-- 1 user group 618M Sep 21 11:00 'A.27100kbps.AVC.21m40s.24FPS-1920x1080-[AV1.OPUS.2.0].mkv'
-rw-rw-r-- 1 user group 1.8G Sep 21 11:49 'B.6385kbps.VP9.55m11s.30FPS-3840x2160-[AV1.OPUS.2.0].mkv'
-rw-rw-r-- 1 user group 469M Sep 21 11:58 'C.1236kbps.VP9.54m40s.30FPS-1280x720-[AV1.OPUS.2.0].mkv'
-rw-rw-r-- 1 user group 287M Sep 21 12:03 'D.707kbps.AVC.54m5s.30FPS-640x480-[AV1.OPUS.2.0].mkv'
-rw-rw-r-- 1 user group 163M Sep 21 12:07 'F.304kbps.VP9.54m29s.30FPS-468x360-[AV1.OPUS.2.0].mkv'
-rw-rw-r-- 1 user group 232M Sep 21 12:11 'F.482kbps.HEVC.56m2s.24FPS-640x480-[AV1.OPUS.2.0].mkv'

And now the script. I'm probably close to the text limit...

Code:

# Adaptive two-pass for HEVC_QSV

ADPT=1 && \ # set whether to use adaptive bit rate
DEV="/dev/dri/renderD129" && \
ACODEC="libopus" && \
VCODEC="hevc_qsv" && \
REDX="90" && \ # percentage of original
VMIN=500 && \ # minimum acceptable video bitrate (do not go below)
CLEAN=".m[kp][4v]" && \ # removes extensions and other extraneous filename info
IPATH="/home/bitmap/testbed/encoding/source_files/mixed_media" && \ # input path
OPATH="/home/bitmap/testbed/encoding/batch_output/hevc_qsv" && \ # output path
PRAUD="0" && \ # primary audio stream index
SEAUD="" && \
PRSUB="0?" && \ # primary subtitle stream index
SESUB="" && \
TRSUB="" && \
VRATE="4500" && \ # desired video bitrate
VMAX="9000" && \ # maximum desired video bitrate
VBUF="18000" && \ # video buffer size
PRESET="slower" && \ # encoder preset
ABIT="192k" && \ # default audio bitrate
AFLTR="pan=stereo|FL<FC+0.30*FL+0.30*BL|FR<FC+0.30*FR+0.30*BR" && \
EXT="mkv" && \ # output file extension
QUAL="WEBRip" && \ # quality -- only used for file naming
shopt -s globstar && \
clear && \
# start it up -- use extreme caution editing below this comment
BV="${VRATE}k" && \ # set fallbacks
MRATE="${VMAX}k" && \
BSIZE="${VBUF}k" && \
for MEDIA in "${IPATH}"/**/*.m[kp][4v]; do
    # reset flags
    ENCD=0
    ROVER=0
    RUNDER=0
    if [[ $ADPT == 1 ]]; then
        printf "Attempting to find adaptive bit rate for mixed media" && \
        printf "." && sleep 0.5 && printf "." && sleep 0.5 && printf "." && \
        sleep 0.5 && printf "\n" && \
        # ffprobe to check existing bitrate
        FFVR=$(ffprobe -v error -select_streams v:0 -show_entries format=bit_rate -of default=nw=1:nk=1 "${MEDIA}") && \
        if [[ $FFVR != "N/A" ]]; then
            KBVR=$(awk "BEGIN { print $FFVR*0.001 }") && \ # extract bitrate as kbps
            ROVER=$(awk "BEGIN { print ($KBVR > $VRATE) }") && \ # check if bitrate over desired
            RUNDR=$(awk "BEGIN { print ($KBVR < $VMIN) }") # check if bitrate under min
            if [[ $RUNDER == 1 ]]; then
                RDXD=0.001 && \ # set encode bitrate to 100% if under min
                ENCD=1 # go ahead with modified encoding settings
                printf "Bit rate too low, not reducing further" && \
                printf "." && sleep 0.5 && printf "." && sleep 0.5 && printf "." && \
                sleep 0.5 && printf "\n"
            elif [[ $ROVER == 1 ]]; then
                printf "Bit rate over specified limit, reverting to %s kbps" "${VRATE}" && \
                printf "." && sleep 0.5 && printf "." && sleep 0.5 && printf "." && \
                sleep 0.5 && printf "\n"
            else
                RDXD=$(awk "BEGIN { print $REDX/100000 }") && \
                ENCD=1 && \
                printf "Bit rate will be reduced to %s%% of original" "${REDX}" && \
                printf "." && sleep 0.5 && printf "." && sleep 0.5 && printf "." && \
                sleep 0.5 && printf "\n"
            fi
            INVR=$(awk "BEGIN { print $FFVR*$RDXD }") && \
            if [[ $ENCD == 1 ]]; then
            printf "Bit rate set for encoding: %s kbps" "${INVR}" && \
                printf "." && sleep 0.5 && printf "." && sleep 0.5 && printf "." && \
                sleep 0.5 && printf "\n"
            MCON=$(awk "BEGIN { print $INVR*2 }") && \
            BCON=$(awk "BEGIN { print $INVR*4 }") && \
            BV="${INVR}k" && \
            MRATE="${MCON}k" && \
            BSIZE="${BCON}k" && \
            ABIT="128k"
            fi
        else
            printf "FFprobe could not identify a bit rate, falling back to %s kbps" "${VRATE}" && \
            printf "." && sleep 0.5 && printf "." && sleep 0.5 && printf "." && \
            sleep 0.5 && printf "\n" && \
        fi
    else
        echo "Video bit rate for all session files locked at %s kbps" "${VRATE}" && \
        printf "." && sleep 0.5 && printf "." && sleep 0.5 && printf "." && \
        sleep 0.5 && printf "\n" && \
    fi
    ISIZE=$(du -BMB "${MEDIA}" | cut -f -1) && \
    FNAME=$(basename "${MEDIA%${CLEAN}}") && \
    OFILE="${OPATH}/${FNAME}-[AV1.OPUS.2.0].${EXT}" && \
    printf "Beginning first encoding pass for %s. Starting file size: %s" "${FNAME}" "${ISIZE}" && \
    printf "." && sleep 0.5 && printf "." && sleep 0.5 && printf "." && \
    sleep 0.5 && printf "\n" && \
    ffmpeg -y -hide_banner -loglevel warning -v quiet -stats -qsv_device "${DEV}" \
    -i "${MEDIA}" -pass 1 \
    -map_metadata 0 -map 0:v:0 \
    -c:v "${VCODEC}" -preset "${PRESET}" \
    -b:v "${BV}" -maxrate:v "${MRATE}" -bufsize:v "${BSIZE}" \
    -an -sn \
    "${OFILE}" && \
    printf "First pass complete, beginning second encoding pass" && \
    printf "." && sleep 0.5 && printf "." && sleep 0.5 && printf "." && \
    sleep 0.5 && printf "\n" && \
    ffmpeg -y -hide_banner -loglevel warning -v quiet -stats -qsv_device "${DEV}" \
    -i "${MEDIA}" -pass 2 \
    -map_metadata 0 -map 0:v:0 \
    -map 0:a:"${PRAUD}" \
    #-map 0:a:"${PRAUD}" \
    #-map 0:a:"${SEAUD}" \
    #-map 0:a \
    -map 0:s:"${PRSUB}" \
    #-map 0:s:"${SESUB}" \
    #-map 0:s:"${TRSUB}" \
    #-map 0:s \
    #-map 0:t? \
    -c:v "${VCODEC}" -preset "${PRESET}" \
    #-extbrc 1 -look_ahead 1 -look_ahead_depth 100 \ # should work, haven't tested
    -b:v "${BV}" -maxrate:v "${MRATE}" -bufsize:v "${BSIZE}" \
-c:a "${ACODEC}" -ac:a:0 2 -b:a:0 "${ABIT}" -filter:a:0 "${AFLTR}" \
#-ac:a:1 6 -b:a:1 256k \
  -c:s copy \
  -metadata title="${FNAME} | ${GRP}" \
  -metadata:s:v:0 title='Fill in the Blank' \
    -metadata:s:a:0 title='English OPUS Stereo' \
    -metadata:s:s:0 title='English' \
    #-metadata:s:s:1 title='English (SDH)' \
    #-metadata:s:s:2 title='English (PGS)' \
  "${OFILE}"  && \
  ESIZE=$(du -BMB "${OFILE}" | cut -f -1) && \
  printf "Finished encoding %s. Final file size: %s" "${FNAME}" "${ESIZE}"; done

Hope this helps along the way!

EDIT: Missed a bug. Bitrate was not calculated correctly because I was relying on an unset variable. Caught that in a new terminal today. Fixed it in the above. Have better ideas of how to handle a few things as well. Will start working on an actual script and program version soon. This is absolutely just hacked together to paste into your terminal.