Apple VideoToolBox encoding failure

[grips]

Hello,

I am having some transcoding issues with certain files, and it seems to be related to hardware encoding with Apple VideoToolBox.

I have Jellyfin installed on an Intel-based Mac Mini using the v10.8.13 dmg.

The ffmpeg log looks like:

ffmpeg -analyzeduration 200M -init_hw_device videotoolbox=vt -hwaccel videotoolbox -autorotate 0 -i file:"/Volumes/video/Series/MacGyver/Season 1/01 Pilot-C.avi" -autoscale 0 -map_metadata -1 -map_chapters -1 -threads 0 -map 0:0 -map 0:1 -map -0:s -codec:v:0 h264_videotoolbox -b:v 3203195 -maxrate 3203195 -bufsize 6406390 -force_key_frames:0 "expr:gte(t,0+n_forced*3)" -g:v:0 72 -keyint_min:v:0 72 -vf "setparams=color_primaries=bt709:color_trc=bt709:colorspace=bt709,scale=trunc(min(max(iw\,ih*a)\,min(512\,384*a))/2)*2:trunc(min(max(iw/a\,ih)\,min(512/a\,384))/2)*2,format=nv12" -codec:a:0 aac -ac 2 -ab 256000 -copyts -avoid_negative_ts disabled -max_muxing_queue_size 2048 -f hls -max_delay 5000000 -hls_time 3 -hls_segment_type mpegts -start_number 0 -hls_segment_filename "/Users/xxx/.local/share/jellyfin/transcodes/7736832436128392456bd5ce87ea24d7%d.ts" -hls_playlist_type vod -hls_list_size 0 -y "/Users/xxx/.local/share/jellyfin/transcodes/7736832436128392456bd5ce87ea24d7.m3u8"


ffmpeg version 5.1.2-tessus Copyright © 2000-2022 the FFmpeg developers
  built with Apple clang version 11.0.0 (clang-1100.0.33.17)
  configuration: --cc=/usr/bin/clang --prefix=/opt/ffmpeg --extra-version=tessus --enable-avisynth --enable-fontconfig --enable-gpl --enable-libaom --enable-libass --enable-libbluray --enable-libdav1d --enable-libfreetype --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-libmysofa --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenh264 --enable-libopenjpeg --enable-libopus --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvmaf --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxavs --enable-libxvid --enable-libzimg --enable-libzmq --enable-libzvbi --enable-version3 --pkg-config-flags=--static --disable-ffplay
  libavutil      57. 28.100 / 57. 28.100
  libavcodec    59. 37.100 / 59. 37.100
  libavformat    59. 27.100 / 59. 27.100
  libavdevice    59.  7.100 / 59.  7.100
  libavfilter    8. 44.100 /  8. 44.100
  libswscale      6.  7.100 /  6.  7.100
  libswresample  4.  7.100 /  4.  7.100
  libpostproc    56.  6.100 / 56.  6.100
Input #0, avi, from 'file:/Volumes/video/Series/MacGyver/Season 1/01 Pilot-C.avi':
  Metadata:
    software        : Lavf52.32.0
  Duration: 00:45:40.62, start: 0.000000, bitrate: 618 kb/s
  Stream #0:0: Video: mpeg4 (Simple Profile) (xvid / 0x64697678), yuv420p, 512x384 [SAR 1:1 DAR 4:3], 512 kb/s, 23.98 fps, 23.98 tbr, 23.98 tbn
  Stream #0:1: Audio: ac3 ([0] [0][0] / 0x2000), 44100 Hz, stereo, fltp, 96 kb/s
Stream mapping:
  Stream #0:0 -> #0:0 (mpeg4 (native) -> h264 (h264_videotoolbox))
  Stream #0:1 -> #0:1 (ac3 (native) -> aac (native))
Press [q] to stop, [?] for help
[mpeg4 @ 0x7f80a600a980] Failed setup for format videotoolbox_vld: hwaccel initialisation returned error.
[h264_videotoolbox @ 0x7f80a60085c0] Error: cannot create compression session: -12903
[h264_videotoolbox @ 0x7f80a60085c0] Try -allow_sw 1. The hardware encoder may be busy, or not supported.
Error initializing output stream 0:0 -- Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height
[aac @ 0x7f80a6009840] Qavg: 623.956
[aac @ 0x7f80a6009840] 2 frames left in the queue on closing
Conversion failed!

It does work properly with other video files. I'm looking for ideas on resolving this. Let me know if I can provide any extra details to help. 

Thank you

[TheDreadPirate]

What model CPU? And can you share a screenshot of the codec boxes in Dashboard > Playback?

[grips]

  Model Name: Mac mini
  Model Identifier: Macmini8,1
  Processor Name: 6-Core Intel Core i7
  Processor Speed: 3.2 GHz
  Number of Processors: 1
  Total Number of Cores: 6
  L2 Cache (per Core): 256 KB
  L3 Cache: 12 MB

Is this enough detail on the CPU?

Edit: looks to be a Core i7 (I7-8700B)

[TheDreadPirate]

Core i7 is not enough. I need the specific model like "Core i3 12100". Based on what I can find for "macmini8,1" it looks like an 8th generation Intel CPU? Is that correct?

If so, uncheck MPEG4 and save. AFAIK, Intel iGPUs cannot decode that codec and your video uses MPEG4. Unchecking the box just means the CPU will decode the video.

[gnattu]

Your ffmpeg is not configured with videotoolbox enabled, and current 10.8.13 dmg has a bug that cannot use the correct ffmpeg that comes with jellyfin.

A workaround is to manually download the jellyfin's ffmpeg from the repo: https://repo.jellyfin.org/?path=/ffmpeg/...6.x/x86_64

and then manually edit the encoding.xml to let jellyfin find that ffmpeg.

Alternatively you can use jellyfin's ffmpeg to replace your system's ffmpeg.

You do not have to do these in the following 10.9 release.

[grips]

Your ffmpeg is not configured with videotoolbox enabled, and current 10.8.13 dmg has a bug that cannot use the correct ffmpeg that comes with jellyfin.

A workaround is to manually download the jellyfin's ffmpeg from the repo: https://repo.jellyfin.org/?path=/ffmpeg/...6.x/x86_64

and then manually edit the encoding.xml to let jellyfin find that ffmpeg.

Alternatively you can use jellyfin's ffmpeg to replace your system's ffmpeg.

You do not have to do these in the following 10.9 release.

These errors are generated with jellfyin-ffmpeg.

Code:

xxx@Singularity bin % ll|grep ffmpeg
lrwxr-xr-x    1 xxx  admin    73 Apr 16 11:54 ffmpeg -> /Users/xxx/Downloads/jellyfin-ffmpeg_6.0.1-5_portable_mac64-gpl/ffmpeg
lrwxr-xr-x    1 xxx  admin    74 Apr 16 11:54 ffprobe -> /Users/xxx/Downloads/jellyfin-ffmpeg_6.0.1-5_portable_mac64-gpl/ffprobe
xxx@Singularity bin % pwd
/usr/local/bin
xxx@Singularity bin %

Code:

  <HardwareAccelerationType>videotoolbox</HardwareAccelerationType>
  <EncoderAppPath>/usr/local/bin/ffmpeg</EncoderAppPath>
  <EncoderAppPathDisplay>/usr/local/bin/ffmpeg</EncoderAppPathDisplay>

Core i7 is not enough. I need the specific model like "Core i3 12100". Based on what I can find for "macmini8,1" it looks like an 8th generation Intel CPU? Is that correct?

If so, uncheck MPEG4 and save. AFAIK, Intel iGPUs cannot decode that codec and your video uses MPEG4. Unchecking the box just means the CPU will decode the video.

Sorry, I edited the earlier post with the details instead of responding. It seems to be a Core i7 (I7-8700B). I tried unchecking the MPEG4 and it still failed, but the message is actually slightly different.

Input #0, avi, from 'fileVolumes/video/Series/MacGyver/Season 1/01 Pilot-C.avi':
  Metadata:
    software        : Lavf52.32.0
  Duration: 00:45:40.62, start: 0.000000, bitrate: 618 kb/s
  Stream #0:0: Video: mpeg4 (Simple Profile) (xvid / 0x6469767, yuv420p, 512x384 [SAR 1:1 DAR 4:3], 512 kb/s, 23.98 fps, 23.98 tbr, 23.98 tbn
  Stream #0:1: Audio: ac3 ([0] [0][0] / 0x2000), 44100 Hz, stereo, fltp, 96 kb/s
Stream mapping:
  Stream #0:0 -> #0:0 (mpeg4 (native) -> h264 (h264_videotoolbox))
  Stream #0:1 -> #0:1 (ac3 (native) -> aac (native))
Press [q] to stop, [?] for help
[h264_videotoolbox @ 0x7fd3a6908d80] Error: cannot create compression session: -12903
[h264_videotoolbox @ 0x7fd3a6908d80] Try -allow_sw 1. The hardware encoder may be busy, or not supported.
[vost#0:0/h264_videotoolbox @ 0x7fd3a6908a80] Error initializing output stream: Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height
[aac @ 0x7fd3a6909ec0] Qavg: 921.578
[aac @ 0x7fd3a6909ec0] 2 frames left in the queue on closing
Conversion failed!

[gnattu]

These errors are generated with jellfyin-ffmpeg.

Code:

xxx@Singularity bin % ll|grep ffmpeg
lrwxr-xr-x    1 xxx  admin    73 Apr 16 11:54 ffmpeg -> /Users/xxx/Downloads/jellyfin-ffmpeg_6.0.1-5_portable_mac64-gpl/ffmpeg
lrwxr-xr-x    1 xxx  admin    74 Apr 16 11:54 ffprobe -> /Users/xxx/Downloads/jellyfin-ffmpeg_6.0.1-5_portable_mac64-gpl/ffprobe
xxx@Singularity bin % pwd
/usr/local/bin
xxx@Singularity bin %

Code:

  <HardwareAccelerationType>videotoolbox</HardwareAccelerationType>
  <EncoderAppPath>/usr/local/bin/ffmpeg</EncoderAppPath>
  <EncoderAppPathDisplay>/usr/local/bin/ffmpeg</EncoderAppPathDisplay>

Input #0, avi, from 'fileVolumes/video/Series/MacGyver/Season 1/01 Pilot-C.avi':
  Metadata:
    software        : Lavf52.32.0
  Duration: 00:45:40.62, start: 0.000000, bitrate: 618 kb/s
  Stream #0:0: Video: mpeg4 (Simple Profile) (xvid / 0x6469767, yuv420p, 512x384 [SAR 1:1 DAR 4:3], 512 kb/s, 23.98 fps, 23.98 tbr, 23.98 tbn
  Stream #0:1: Audio: ac3 ([0] [0][0] / 0x2000), 44100 Hz, stereo, fltp, 96 kb/s
Stream mapping:
  Stream #0:0 -> #0:0 (mpeg4 (native) -> h264 (h264_videotoolbox))
  Stream #0:1 -> #0:1 (ac3 (native) -> aac (native))
Press [q] to stop, [?] for help
[h264_videotoolbox @ 0x7fd3a6908d80] Error: cannot create compression session: -12903
[h264_videotoolbox @ 0x7fd3a6908d80] Try -allow_sw 1. The hardware encoder may be busy, or not supported.
[vost#0:0/h264_videotoolbox @ 0x7fd3a6908a80] Error initializing output stream: Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height
[aac @ 0x7fd3a6909ec0] Qavg: 921.578
[aac @ 0x7fd3a6909ec0] 2 frames left in the queue on closing
Conversion failed!

I need to see the *full log*. Please please please don't cut off the log as the part that you *think* is relevant may not be helpful.

[grips]

Sure thing. Here is the full FFmpeg log with MPEG4 unchecked. Let me know if anything else would help.

/videos/52f05471-6b03-6882-790c-f8bcebbe8c1b/hls1/main/0.ts

{"Protocol":0,"Id":"52f054716b036882790cf8bcebbe8c1b","Path":"/Volumes/video/Series/MacGyver/Season 1/01 Pilot-C.avi","EncoderPath":null,"EncoderProtocol":null,"Type":0,"Container":"avi","Size":211960960,"Name":"01 Pilot-C","IsRemote":false,"ETag":"77d983fec07f6590e5fe5dc0067f0c52","RunTimeTicks":27406155776,"ReadAtNativeFramerate":false,"IgnoreDts":false,"IgnoreIndex":false,"GenPtsInput":false,"SupportsTranscoding":true,"SupportsDirectStream":true,"SupportsDirectPlay":true,"IsInfiniteStream":false,"RequiresOpening":false,"OpenToken":null,"RequiresClosing":false,"LiveStreamId":null,"BufferMs":null,"RequiresLooping":false,"SupportsProbing":true,"VideoType":0,"IsoType":null,"Video3DFormat":null,"MediaStreams":[{"Codec":"mpeg4","CodecTag":"xvid","Language":null,"ColorRange":null,"ColorSpace":null,"ColorTransfer":null,"ColorPrimaries":null,"DvVersionMajor":null,"DvVersionMinor":null,"DvProfile":null,"DvLevel":null,"RpuPresentFlag":null,"ElPresentFlag":null,"BlPresentFlag":null,"DvBlSignalCompatibilityId":null,"Comment":null,"TimeBase":"125/2997","CodecTimeBase":null,"Title":null,"VideoRange":"SDR","VideoRangeType":"SDR","VideoDoViTitle":null,"LocalizedUndefined":null,"LocalizedDefault":null,"LocalizedForced":null,"LocalizedExternal":null,"DisplayTitle":"384p MPEG4 SDR","NalLengthSize":null,"IsInterlaced":false,"IsAVC":null,"ChannelLayout":null,"BitRate":512511,"BitDepth":8,"RefFrames":1,"PacketLength":null,"Channels":null,"SampleRate":null,"IsDefault":false,"IsForced":false,"Height":384,"Width":512,"AverageFrameRate":23.976,"RealFrameRate":23.976,"Profile":"Simple Profile","Type":1,"AspectRatio":"4:3","Index":0,"Score":null,"IsExternal":false,"DeliveryMethod":null,"DeliveryUrl":null,"IsExternalUrl":null,"IsTextSubtitleStream":false,"SupportsExternalStream":false,"Path":null,"PixelFormat":"yuv420p","Level":3,"IsAnamorphic":null},{"Codec":"ac3","CodecTag":null,"Language":null,"ColorRange":null,"ColorSpace":null,"ColorTransfer":null,"ColorPrimaries":null,"DvVersionMajor":null,"DvVersionMinor":null,"DvProfile":null,"DvLevel":null,"RpuPresentFlag":null,"ElPresentFlag":null,"BlPresentFlag":null,"DvBlSignalCompatibilityId":null,"Comment":null,"TimeBase":"128/3675","CodecTimeBase":null,"Title":null,"VideoRange":null,"VideoRangeType":null,"VideoDoViTitle":null,"LocalizedUndefined":null,"LocalizedDefault":null,"LocalizedForced":null,"LocalizedExternal":null,"DisplayTitle":"Dolby Digital - Stereo","NalLengthSize":null,"IsInterlaced":false,"IsAVC":null,"ChannelLayout":"stereo","BitRate":96000,"BitDepth":null,"RefFrames":null,"PacketLength":null,"Channels":2,"SampleRate":44100,"IsDefault":false,"IsForced":false,"Height":null,"Width":null,"AverageFrameRate":null,"RealFrameRate":null,"Profile":null,"Type":0,"AspectRatio":null,"Index":1,"Score":null,"IsExternal":false,"DeliveryMethod":null,"DeliveryUrl":null,"IsExternalUrl":null,"IsTextSubtitleStream":false,"SupportsExternalStream":false,"Path":null,"PixelFormat":null,"Level":0,"IsAnamorphic":null}],"MediaAttachments":[],"Formats":[],"Bitrate":618725,"Timestamp":null,"RequiredHttpHeaders":{},"TranscodingUrl":null,"TranscodingSubProtocol":null,"TranscodingContainer":null,"AnalyzeDurationMs":null,"DefaultAudioStreamIndex":null,"DefaultSubtitleStreamIndex":null}

/usr/local/bin/ffmpeg -analyzeduration 200M -init_hw_device videotoolbox=vt -autorotate 0 -i file:"/Volumes/video/Series/MacGyver/Season 1/01 Pilot-C.avi" -map_metadata -1 -map_chapters -1 -threads 0 -map 0:0 -map 0:1 -map -0:s -codec:v:0 h264_videotoolbox -b:v 3203195 -maxrate 3203195 -bufsize 6406390 -force_key_frames:0 "expr:gte(t,0+n_forced*3)" -g:v:0 72 -keyint_min:v:0 72 -vf "setparams=color_primaries=bt709:color_trc=bt709:colorspace=bt709,scale=trunc(min(max(iw\,ih*a)\,min(512\,384*a))/2)*2:trunc(min(max(iw/a\,ih)\,min(512/a\,384))/2)*2,format=yuv420p" -codec:a:0 aac -ac 2 -ab 256000 -copyts -avoid_negative_ts disabled -max_muxing_queue_size 2048 -f hls -max_delay 5000000 -hls_time 3 -hls_segment_type mpegts -start_number 0 -hls_segment_filename "/Users/xxx/.local/share/jellyfin/transcodes/11f680a00e47b9fe31944789a117e0aa%d.ts" -hls_playlist_type vod -hls_list_size 0 -y "/Users/xxx/.local/share/jellyfin/transcodes/11f680a00e47b9fe31944789a117e0aa.m3u8"


ffmpeg version 6.0.1-Jellyfin Copyright © 2000-2023 the FFmpeg developers
  built with Apple clang version 14.0.0 (clang-1400.0.29.202)
  configuration: --prefix=/ffbuild/prefix --disable-shared --enable-static --pkg-config-flags=--static --enable-pthreads --cc=clang --host-cflags='-I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -I/opt/ffbuild/prefix/include' --host-ldflags='-L/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/lib -L/opt/ffbuild/prefix/lib' --extra-version=Jellyfin --extra-cflags=-I/opt/ffbuild/prefix/include --extra-cxxflags= --extra-ldflags=-L/opt/ffbuild/prefix/lib --extra-ldexeflags= --extra-libs= --enable-gpl --enable-version3 --enable-lto --disable-ffplay --disable-debug --disable-doc --disable-ptx-compression --disable-sdl2 --disable-libjack --disable-indev=jack --enable-neon --enable-runtime-cpudetect --enable-audiotoolbox --enable-videotoolbox --enable-iconv --enable-libxml2 --enable-zlib --enable-libfreetype --enable-libfribidi --enable-gmp --enable-openssl --enable-lzma --enable-fontconfig --enable-libvorbis --enable-opencl --disable-amf --enable-chromaprint --enable-libdav1d --disable-libfdk-aac --disable-ffnvcodec --disable-cuda --disable-cuda-llvm --disable-cuvid --disable-nvdec --disable-nvenc --enable-libass --enable-libbluray --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvpx --enable-libwebp --enable-libopenmpt --enable-libsrt --enable-libsvtav1 --enable-libx264 --enable-libx265 --enable-libzimg --enable-libzvbi
  libavutil      58.  2.100 / 58.  2.100
  libavcodec    60.  3.100 / 60.  3.100
  libavformat    60.  3.100 / 60.  3.100
  libavdevice    60.  1.100 / 60.  1.100
  libavfilter    9.  3.100 /  9.  3.100
  libswscale      7.  1.100 /  7.  1.100
  libswresample  4. 10.100 /  4. 10.100
  libpostproc    57.  1.100 / 57.  1.100
Input #0, avi, from 'fileVolumes/video/Series/MacGyver/Season 1/01 Pilot-C.avi':
  Metadata:
    software        : Lavf52.32.0
  Duration: 00:45:40.62, start: 0.000000, bitrate: 618 kb/s
  Stream #0:0: Video: mpeg4 (Simple Profile) (xvid / 0x6469767, yuv420p, 512x384 [SAR 1:1 DAR 4:3], 512 kb/s, 23.98 fps, 23.98 tbr, 23.98 tbn
  Stream #0:1: Audio: ac3 ([0] [0][0] / 0x2000), 44100 Hz, stereo, fltp, 96 kb/s
Stream mapping:
  Stream #0:0 -> #0:0 (mpeg4 (native) -> h264 (h264_videotoolbox))
  Stream #0:1 -> #0:1 (ac3 (native) -> aac (native))
Press [q] to stop, [?] for help
[h264_videotoolbox @ 0x7fcb4ff09840] Error: cannot create compression session: -12903
[h264_videotoolbox @ 0x7fcb4ff09840] Try -allow_sw 1. The hardware encoder may be busy, or not supported.
[vost#0:0/h264_videotoolbox @ 0x7fcb4ff09400] Error initializing output stream: Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height
[aac @ 0x7fcb4ff0a780] Qavg: 921.578
[aac @ 0x7fcb4ff0a780] 2 frames left in the queue on closing
Conversion failed!

[gnattu]

This really should not happen and I tried the same command and I cannot reproduce this behavior. What is your macOS version? Just in case it is too old to be supported.

To verify if your system works with hwaccel in general, you can test with:


./ffmpeg -f lavfi -i testsrc=size=512x384:rate=23.98 -t 30 -c:v h264_videotoolbox test.mp4


in the path contains jellyfin-ffmpeg. If it still errors out for this probably minimal use case, then it is not really a jellyfin issue, but rather some problem with:

- macOS on your machine, a not compatible version or a broken installation

- or even worse, a broken Mac (less likely)

[grips]

Crazy, that command does fail. I guess it's something with my setup, then. I'm running the latest MacOS.

Other formats seem to work without any issues. I found this old, closed github issue that looks similar: https://github.com/jellyfin/jellyfin/issues/9507

Anyway, I appreciate the time and info. I will try to look into ffmpeg more and see if I can't resolve something.