Encoding Discussion Megathread (ffmpeg, Handbrake, AV1, etc...)

[bitmap]

Okay so I did something I think is cool? Maybe it's stupid, but I can see some pretty interesting applications for just converting older media, which I have a lot of at this point.

I created a script, which I'm transforming into a program at the moment, that does adaptive video bitrate deduction using ffprobe to match the bitrate of your video and sacrifice as little quality as possible during encoding. I was trying to throw together something that could scrape HUGE series I have (I'm archiving some educational content) and encode them all at once. I made this great script that was working and when I finally threw it against one of my series, I was turning 250 MB episodes into 900 MB bloated nightmares and dropping thousands of frames every which way.

It sucks because I'm stuck using constant bitrate and the compromise is to run two-pass encodes. So I thought...can I get the existing bitrate? Surely ffprobe can give me that information and I can steal it in a usable way? Well...seven hours later, here I am, running this script with reasonable results that I think are within a) expectations, b) margins of error, and c) some bloat expected from likely poor quality audio.

Things that do work:
* Set a fallback/ceiling video bitrate
* If video's bitrate exceeds ceiling, fallback bitrate is used
* Script hunts for all video within subdirectories
* Copies chapters (but not explicitly...)
* Turn on/off adaptive video bitrate
* All audio currently set to OPUS, easily changed
* Ability to map up to two audio streams
* Duplicate primary audio into a downmixed stereo stream
* Specify bitrate for primary stereo downmix
* Change your container (but why?)
* Change your preset
* Cleaning of filenames to set metadata and rename output files (regex-like bash parameter expansion)
* Set basic metadata


Things I don't have implemented, but want to:
* Video filters (de-interlacing, scaling, maybe others)
* Support for more than just bitrate control (I hate it)
* Easier options -- building the program it's pretty monstrous, but I'm not certain how to make it simpler
* Pixel formats (i.e., setting for 10-bit encoding, as AV1 doesn't allow for this)
* Choose your own encoder -- this is harder since they're all so different
* Pick your codecs and bit rates more easily
* Pick your streams more easily
* Interactive terminal choices (take a sample file from a dir, simplify ffprobe/mediainfo, offer choices, select best options, go)

Anyway, I'm having fun. Maybe this will go somewhere, maybe not. At the very least, somebody can steal the ffprobe and awk functions I'm throwing around. Here are a few results I've gotten:

Code:

Attempting to find adaptive bit rate for mixed media...
Bit rate to be used with ffmpeg: 994.119 kbps...
Beginning first encoding pass for XYZ (1900) - S00E00 - The Great Great Great Great. Starting file size: 429MB
[...]
    frame=98258 fps=1372 q=-0.0 Lsize=  372212kB time=00:54:38.54 bitrate= 930.0kbits/s speed=45.8x    
First pass complete, beginning second encoding pass...
[...]
    frame=98258 fps=1169 q=-0.0 Lsize=  414444kB time=00:54:38.58 bitrate=1035.5kbits/s speed=  39x    
Finished encoding XYZ (1900) - S00E00 - The Great Great Great Great. Final file size: 425MB

The idea wasn't really to minify these episodes, but to convert them to AV1 with as little quality loss as possible for older episodes, with space savings coming for the higher quality episodes. Here's another:

Code:

Attempting to find adaptive bit rate for mixed media...
Bit rate to be used with ffmpeg: 588.209 kbps...
Beginning first encoding pass for XYZ (1900) - S00E00 - Radio Radio Radio. Starting file size: 270MB
[...]
    frame=104452 fps=1456 q=-0.0 Lsize=  229741kB time=00:58:01.80 bitrate= 540.5kbits/s speed=48.5x    
First pass complete, beginning second encoding pass...
[...]
    frame=104452 fps=1232 q=-0.0 Lsize=  270798kB time=00:58:01.90 bitrate= 637.1kbits/s speed=41.1x    
Finished encoding XYZ (1900) - S00E00 - Radio Radio Radio. Final file size: 278MB

Okay, so enough of my BS. Here's the code. It's not commented yet as I just finished putting it together in working fashion. I'm still working on the program version of it. Once I have anything more than just something you can copy and paste into a terminal, I'll throw it on Github.

Code:

# Adaptive AV1_QSV script for use in the terminal with entire series within subfolders

ADAPT=0 && \
ACODEC="libopus" && \
IPATH="" && \
OPATH="" && \
CLEAN="\ \[*\]-*.m[kop4][gv4]" && \
PRAUD="0" && \
SEAUD="" && \
PRSUB="0" && \
SESUB="" && \
TRSUB="" && \
VRATE="4500" && \
VMAX="9000" && \
VBUF="18000" && \
PRESET="slower" && \
ABIT="192k" && \
AFLTR="pan=stereo|FL<FC+0.30*FL+0.30*BL|FR<FC+0.30*FR+0.30*BR" && \
EXT="mkv" && \
QUAL="WEBRip" && \
RES="1080p" && \
LNG="EN" && \
SUBS="EN" && \
GRP="bitmap" && \
shopt -s globstar && \
mkdir -p "${OPATH}/import" && \
clear && \
BV="${VRATE}" && \
MRATE=$"{VMAX}" && \
BSIZE="${VBUF}" && \
for MEDIA in "${IPATH}"/**/*.m[kp][4v]; do
    if [[ $ADAPT -eq 1 ]]; then
        echo "Attempting to find adaptive bit rate for mixed media..." && \
        FFP=$(ffprobe -v error -select_streams v:0 -show_entries format=bit_rate -of default=nw=1:nk=1 "${MEDIA}") && \
        if [[ $FFP != "N/A" ]]; then
            VCON=$(awk "BEGIN { print $FFP*0.00095 }") && \
            SLCT=$(awk "BEGIN { print ($VCON < $VRATE) }") && \
            if [[ $SLCT == 1 ]]; then
                echo "Bit rate to be used with ffmpeg: $VCON kbps..."
                MCON=$(awk "BEGIN { print $VCON*2 }") && \
                BCON=$(awk "BEGIN { print $VCON*4 }") && \
                BV="${VCON}k" && \
                MRATE="${MCON}k" && \
                BSIZE="${BCON}k" && \
                ABIT="128k"
            else
                echo "Identified rate exceeds ${VRATE} kbps, falling back..."
            fi
        else
            echo "FFprobe could not identify a bit rate, falling back to ${VRATE} kbps."
        fi
    else
        echo "Global bit rate for session locked at ${VRATE} kbps..."
    fi
    ISIZE=$(du -BMB "${MEDIA}" | cut -f -1) && \
    FNAME=$(basename "${MEDIA%${CLEAN}}") && \
    OFILE="${OPATH}/${FNAME}-[${QUAL}-${RES}.AV1.OPUS.2.0][${LNG}][${SUBS}]-${GRP}.${EXT}" && \
    echo "Beginning first encoding pass for ${FNAME}. Starting file size: ${ISIZE}" && \
    ffmpeg -y -hide_banner -loglevel warning -v quiet -stats \
    -hwaccel qsv -hwaccel_output_format qsv -qsv_device /dev/dri/renderD129 \
    -i "${MEDIA}" -pass 1 -map_metadata 0 -map 0:v:0 \
    -c:v av1_qsv -preset "${PRESET}" -look_ahead 1 -look_ahead_depth 100 \
    -b:v "${BV}" -maxrate:v "${MRATE}" -bufsize:v "${BSIZE}" \
    -an -sn \
    "${OFILE}" && \
    echo "First pass complete, beginning second encoding pass..." && \
    ffmpeg -y -hide_banner -loglevel warning -v quiet -stats \
    -hwaccel qsv -hwaccel_output_format qsv -qsv_device /dev/dri/renderD129 \
    -i "${MEDIA}" -pass 2 -map_metadata 0 -map 0:v:0 \
    -map 0:a:"${PRAUD}" \
    #-map 0:a:"${PRAUD}" \
    #-map 0:a:"${SEAUD}" \
    #-map 0:a \
    -map 0:s:"${PRSUB}" \
    #-map 0:s:"${SESUB}" \
    #-map 0:s:"${TRSUB}" \
    #-map 0:s \
    #-map 0:t? \
    -c:v av1_qsv -preset "${PRESET}" -look_ahead 1 -look_ahead_depth 100 \
    -b:v "${BV}" -maxrate:v "${MRATE}" -bufsize:v "${BSIZE}" \
    -c:a "${ACODEC}" -ac:a:0 2 -b:a:0 "${ABIT}" -filter:a:0 "${AFLTR}" \
    #-ac:a:1 6 -b:a:1 256k \
       -c:s copy \
       -metadata title="${FNAME} | ${GRP}" \
       -metadata:s:v:0 title='AV1 encoding by nAV1s' \
    -metadata:s:a:0 title='English OPUS Stereo' \
    -metadata:s:s:0 title='English' \
    #-metadata:s:s:1 title='English (SDH)' \
    #-metadata:s:s:2 title='English (PGS)' \
       "${OFILE}"  && \
       ESIZE=$(du -BMB "${OFILE}" | cut -f -1) && \
       echo "Finished encoding ${FNAME}. Final file size: ${ESIZE}"; done