+- Jellyfin Forum (https://forum.jellyfin.org)
+-- Forum: Support (https://forum.jellyfin.org/f-support)
+--- Forum: Troubleshooting (https://forum.jellyfin.org/f-troubleshooting)
+--- Thread: Apple VideoToolBox encoding failure (/t-apple-videotoolbox-encoding-failure)
Apple VideoToolBox encoding failure - grips - 2024-04-16
Hello,
I am having some transcoding issues with certain files, and it seems to be related to hardware encoding with Apple VideoToolBox.
I have Jellyfin installed on an Intel-based Mac Mini using the v10.8.13 dmg.
The ffmpeg log looks like:
Quote:ffmpeg -analyzeduration 200M -init_hw_device videotoolbox=vt -hwaccel videotoolbox -autorotate 0 -i file:"/Volumes/video/Series/MacGyver/Season 1/01 Pilot-C.avi" -autoscale 0 -map_metadata -1 -map_chapters -1 -threads 0 -map 0:0 -map 0:1 -map -0:s -codec:v:0 h264_videotoolbox -b:v 3203195 -maxrate 3203195 -bufsize 6406390 -force_key_frames:0 "expr:gte(t,0+n_forced*3)" -g:v:0 72 -keyint_min:v:0 72 -vf "setparams=color_primaries=bt709:color_trc=bt709:colorspace=bt709,scale=trunc(min(max(iw\,ih*a)\,min(512\,384*a))/2)*2:trunc(min(max(iw/a\,ih)\,min(512/a\,384))/2)*2,format=nv12" -codec:a:0 aac -ac 2 -ab 256000 -copyts -avoid_negative_ts disabled -max_muxing_queue_size 2048 -f hls -max_delay 5000000 -hls_time 3 -hls_segment_type mpegts -start_number 0 -hls_segment_filename "/Users/xxx/.local/share/jellyfin/transcodes/7736832436128392456bd5ce87ea24d7%d.ts" -hls_playlist_type vod -hls_list_size 0 -y "/Users/xxx/.local/share/jellyfin/transcodes/7736832436128392456bd5ce87ea24d7.m3u8"
ffmpeg version 5.1.2-tessus Copyright © 2000-2022 the FFmpeg developers
built with Apple clang version 11.0.0 (clang-1100.0.33.17)
configuration: --cc=/usr/bin/clang --prefix=/opt/ffmpeg --extra-version=tessus --enable-avisynth --enable-fontconfig --enable-gpl --enable-libaom --enable-libass --enable-libbluray --enable-libdav1d --enable-libfreetype --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-libmysofa --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenh264 --enable-libopenjpeg --enable-libopus --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvmaf --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxavs --enable-libxvid --enable-libzimg --enable-libzmq --enable-libzvbi --enable-version3 --pkg-config-flags=--static --disable-ffplay
libavutil 57. 28.100 / 57. 28.100
libavcodec 59. 37.100 / 59. 37.100
libavformat 59. 27.100 / 59. 27.100
libavdevice 59. 7.100 / 59. 7.100
libavfilter 8. 44.100 / 8. 44.100
libswscale 6. 7.100 / 6. 7.100
libswresample 4. 7.100 / 4. 7.100
libpostproc 56. 6.100 / 56. 6.100
Input #0, avi, from 'file:/Volumes/video/Series/MacGyver/Season 1/01 Pilot-C.avi':
Metadata:
software : Lavf52.32.0
Duration: 00:45:40.62, start: 0.000000, bitrate: 618 kb/s
Stream #0:0: Video: mpeg4 (Simple Profile) (xvid / 0x64697678), yuv420p, 512x384 [SAR 1:1 DAR 4:3], 512 kb/s, 23.98 fps, 23.98 tbr, 23.98 tbn
Stream #0:1: Audio: ac3 ([0] [0][0] / 0x2000), 44100 Hz, stereo, fltp, 96 kb/s
Stream mapping:
Stream #0:0 -> #0:0 (mpeg4 (native) -> h264 (h264_videotoolbox))
Stream #0:1 -> #0:1 (ac3 (native) -> aac (native))
Press [q] to stop, [?] for help
[mpeg4 @ 0x7f80a600a980] Failed setup for format videotoolbox_vld: hwaccel initialisation returned error.
[h264_videotoolbox @ 0x7f80a60085c0] Error: cannot create compression session: -12903
[h264_videotoolbox @ 0x7f80a60085c0] Try -allow_sw 1. The hardware encoder may be busy, or not supported.
Error initializing output stream 0:0 -- Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height
[aac @ 0x7f80a6009840] Qavg: 623.956
[aac @ 0x7f80a6009840] 2 frames left in the queue on closing
Conversion failed!
It does work properly with other video files. I'm looking for ideas on resolving this. Let me know if I can provide any extra details to help.
Thank you
RE: Apple VideoToolBox encoding failure - TheDreadPirate - 2024-04-16
What model CPU? And can you share a screenshot of the codec boxes in Dashboard > Playback?
RE: Apple VideoToolBox encoding failure - grips - 2024-04-16
Quote: Model Name: Mac mini
Model Identifier: Macmini8,1
Processor Name: 6-Core Intel Core i7
Processor Speed: 3.2 GHz
Number of Processors: 1
Total Number of Cores: 6
L2 Cache (per Core): 256 KB
L3 Cache: 12 MB
Is this enough detail on the CPU?
Edit: looks to be a Core i7 (I7-8700B)
![[Image: Screenshot-2024-04-16-at-12-08-12-PM.png]](https://i.ibb.co/YdM90hW/Screenshot-2024-04-16-at-12-08-12-PM.png)
RE: Apple VideoToolBox encoding failure - TheDreadPirate - 2024-04-16
Core i7 is not enough. I need the specific model like "Core i3 12100". Based on what I can find for "macmini8,1" it looks like an 8th generation Intel CPU? Is that correct?
If so, uncheck MPEG4 and save. AFAIK, Intel iGPUs cannot decode that codec and your video uses MPEG4. Unchecking the box just means the CPU will decode the video.
RE: Apple VideoToolBox encoding failure - gnattu - 2024-04-16
Your ffmpeg is not configured with videotoolbox enabled, and current 10.8.13 dmg has a bug that cannot use the correct ffmpeg that comes with jellyfin.
A workaround is to manually download the jellyfin's ffmpeg from the repo: https://repo.jellyfin.org/?path=/ffmpeg/macos/latest-6.x/x86_64
and then manually edit the
encoding.xml to let jellyfin find that ffmpeg.Alternatively you can use jellyfin's ffmpeg to replace your system's ffmpeg.
You do not have to do these in the following 10.9 release.
RE: Apple VideoToolBox encoding failure - grips - 2024-04-16
(2024-04-16, 05:30 PM)gnattu Wrote: Your ffmpeg is not configured with videotoolbox enabled, and current 10.8.13 dmg has a bug that cannot use the correct ffmpeg that comes with jellyfin.
A workaround is to manually download the jellyfin's ffmpeg from the repo: https://repo.jellyfin.org/?path=/ffmpeg/macos/latest-6.x/x86_64
and then manually edit theencoding.xmlto let jellyfin find that ffmpeg.
Alternatively you can use jellyfin's ffmpeg to replace your system's ffmpeg.
You do not have to do these in the following 10.9 release.
These errors are generated with jellfyin-ffmpeg.
Code:
xxx@Singularity bin % ll|grep ffmpeg
lrwxr-xr-x 1 xxx admin 73 Apr 16 11:54 ffmpeg -> /Users/xxx/Downloads/jellyfin-ffmpeg_6.0.1-5_portable_mac64-gpl/ffmpeg
lrwxr-xr-x 1 xxx admin 74 Apr 16 11:54 ffprobe -> /Users/xxx/Downloads/jellyfin-ffmpeg_6.0.1-5_portable_mac64-gpl/ffprobe
xxx@Singularity bin % pwd
/usr/local/bin
xxx@Singularity bin %Code:
<HardwareAccelerationType>videotoolbox</HardwareAccelerationType>
<EncoderAppPath>/usr/local/bin/ffmpeg</EncoderAppPath>
<EncoderAppPathDisplay>/usr/local/bin/ffmpeg</EncoderAppPathDisplay>Quote:Core i7 is not enough. I need the specific model like "Core i3 12100". Based on what I can find for "macmini8,1" it looks like an 8th generation Intel CPU? Is that correct?
If so, uncheck MPEG4 and save. AFAIK, Intel iGPUs cannot decode that codec and your video uses MPEG4. Unchecking the box just means the CPU will decode the video.
Sorry, I edited the earlier post with the details instead of responding. It seems to be a Core i7 (I7-8700B). I tried unchecking the MPEG4 and it still failed, but the message is actually slightly different.
Quote:Input #0, avi, from 'fileVolumes/video/Series/MacGyver/Season 1/01 Pilot-C.avi':
Metadata:
software : Lavf52.32.0
Duration: 00:45:40.62, start: 0.000000, bitrate: 618 kb/s
Stream #0:0: Video: mpeg4 (Simple Profile) (xvid / 0x6469767, yuv420p, 512x384 [SAR 1:1 DAR 4:3], 512 kb/s, 23.98 fps, 23.98 tbr, 23.98 tbn
Stream #0:1: Audio: ac3 ([0] [0][0] / 0x2000), 44100 Hz, stereo, fltp, 96 kb/s
Stream mapping:
Stream #0:0 -> #0:0 (mpeg4 (native) -> h264 (h264_videotoolbox))
Stream #0:1 -> #0:1 (ac3 (native) -> aac (native))
Press [q] to stop, [?] for help
[h264_videotoolbox @ 0x7fd3a6908d80] Error: cannot create compression session: -12903
[h264_videotoolbox @ 0x7fd3a6908d80] Try -allow_sw 1. The hardware encoder may be busy, or not supported.
[vost#0:0/h264_videotoolbox @ 0x7fd3a6908a80] Error initializing output stream: Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height
[aac @ 0x7fd3a6909ec0] Qavg: 921.578
[aac @ 0x7fd3a6909ec0] 2 frames left in the queue on closing
Conversion failed!
RE: Apple VideoToolBox encoding failure - gnattu - 2024-04-16
(2024-04-16, 05:44 PM)grips Wrote: These errors are generated with jellfyin-ffmpeg.
Code:xxx@Singularity bin % ll|grep ffmpeg
lrwxr-xr-x 1 xxx admin 73 Apr 16 11:54 ffmpeg -> /Users/xxx/Downloads/jellyfin-ffmpeg_6.0.1-5_portable_mac64-gpl/ffmpeg
lrwxr-xr-x 1 xxx admin 74 Apr 16 11:54 ffprobe -> /Users/xxx/Downloads/jellyfin-ffmpeg_6.0.1-5_portable_mac64-gpl/ffprobe
xxx@Singularity bin % pwd
/usr/local/bin
xxx@Singularity bin %
Code:<HardwareAccelerationType>videotoolbox</HardwareAccelerationType>
<EncoderAppPath>/usr/local/bin/ffmpeg</EncoderAppPath>
<EncoderAppPathDisplay>/usr/local/bin/ffmpeg</EncoderAppPathDisplay>
Quote:Input #0, avi, from 'file
Metadata:
software : Lavf52.32.0
Duration: 00:45:40.62, start: 0.000000, bitrate: 618 kb/s
Stream #0:0: Video: mpeg4 (Simple Profile) (xvid / 0x6469767
Stream #0:1: Audio: ac3 ([0] [0][0] / 0x2000), 44100 Hz, stereo, fltp, 96 kb/s
Stream mapping:
Stream #0:0 -> #0:0 (mpeg4 (native) -> h264 (h264_videotoolbox))
Stream #0:1 -> #0:1 (ac3 (native) -> aac (native))
Press [q] to stop, [?] for help
[h264_videotoolbox @ 0x7fd3a6908d80] Error: cannot create compression session: -12903
[h264_videotoolbox @ 0x7fd3a6908d80] Try -allow_sw 1. The hardware encoder may be busy, or not supported.
[vost#0:0/h264_videotoolbox @ 0x7fd3a6908a80] Error initializing output stream: Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height
[aac @ 0x7fd3a6909ec0] Qavg: 921.578
[aac @ 0x7fd3a6909ec0] 2 frames left in the queue on closing
Conversion failed!
I need to see the *full log*. Please please please don't cut off the log as the part that you *think* is relevant may not be helpful.
RE: Apple VideoToolBox encoding failure - grips - 2024-04-16
Sure thing. Here is the full FFmpeg log with MPEG4 unchecked. Let me know if anything else would help.
Quote:/videos/52f05471-6b03-6882-790c-f8bcebbe8c1b/hls1/main/0.ts
{"Protocol":0,"Id":"52f054716b036882790cf8bcebbe8c1b","Path":"/Volumes/video/Series/MacGyver/Season 1/01 Pilot-C.avi","EncoderPath":null,"EncoderProtocol":null,"Type":0,"Container":"avi","Size":211960960,"Name":"01 Pilot-C","IsRemote":false,"ETag":"77d983fec07f6590e5fe5dc0067f0c52","RunTimeTicks":27406155776,"ReadAtNativeFramerate":false,"IgnoreDts":false,"IgnoreIndex":false,"GenPtsInput":false,"SupportsTranscoding":true,"SupportsDirectStream":true,"SupportsDirectPlay":true,"IsInfiniteStream":false,"RequiresOpening":false,"OpenToken":null,"RequiresClosing":false,"LiveStreamId":null,"BufferMs":null,"RequiresLooping":false,"SupportsProbing":true,"VideoType":0,"IsoType":null,"Video3DFormat":null,"MediaStreams":[{"Codec":"mpeg4","CodecTag":"xvid","Language":null,"ColorRange":null,"ColorSpace":null,"ColorTransfer":null,"ColorPrimaries":null,"DvVersionMajor":null,"DvVersionMinor":null,"DvProfile":null,"DvLevel":null,"RpuPresentFlag":null,"ElPresentFlag":null,"BlPresentFlag":null,"DvBlSignalCompatibilityId":null,"Comment":null,"TimeBase":"125/2997","CodecTimeBase":null,"Title":null,"VideoRange":"SDR","VideoRangeType":"SDR","VideoDoViTitle":null,"LocalizedUndefined":null,"LocalizedDefault":null,"LocalizedForced":null,"LocalizedExternal":null,"DisplayTitle":"384p MPEG4 SDR","NalLengthSize":null,"IsInterlaced":false,"IsAVC":null,"ChannelLayout":null,"BitRate":512511,"BitDepth":8,"RefFrames":1,"PacketLength":null,"Channels":null,"SampleRate":null,"IsDefault":false,"IsForced":false,"Height":384,"Width":512,"AverageFrameRate":23.976,"RealFrameRate":23.976,"Profile":"Simple Profile","Type":1,"AspectRatio":"4:3","Index":0,"Score":null,"IsExternal":false,"DeliveryMethod":null,"DeliveryUrl":null,"IsExternalUrl":null,"IsTextSubtitleStream":false,"SupportsExternalStream":false,"Path":null,"PixelFormat":"yuv420p","Level":3,"IsAnamorphic":null},{"Codec":"ac3","CodecTag":null,"Language":null,"ColorRange":null,"ColorSpace":null,"ColorTransfer":null,"ColorPrimaries":null,"DvVersionMajor":null,"DvVersionMinor":null,"DvProfile":null,"DvLevel":null,"RpuPresentFlag":null,"ElPresentFlag":null,"BlPresentFlag":null,"DvBlSignalCompatibilityId":null,"Comment":null,"TimeBase":"128/3675","CodecTimeBase":null,"Title":null,"VideoRange":null,"VideoRangeType":null,"VideoDoViTitle":null,"LocalizedUndefined":null,"LocalizedDefault":null,"LocalizedForced":null,"LocalizedExternal":null,"DisplayTitle":"Dolby Digital - Stereo","NalLengthSize":null,"IsInterlaced":false,"IsAVC":null,"ChannelLayout":"stereo","BitRate":96000,"BitDepth":null,"RefFrames":null,"PacketLength":null,"Channels":2,"SampleRate":44100,"IsDefault":false,"IsForced":false,"Height":null,"Width":null,"AverageFrameRate":null,"RealFrameRate":null,"Profile":null,"Type":0,"AspectRatio":null,"Index":1,"Score":null,"IsExternal":false,"DeliveryMethod":null,"DeliveryUrl":null,"IsExternalUrl":null,"IsTextSubtitleStream":false,"SupportsExternalStream":false,"Path":null,"PixelFormat":null,"Level":0,"IsAnamorphic":null}],"MediaAttachments":[],"Formats":[],"Bitrate":618725,"Timestamp":null,"RequiredHttpHeaders":{},"TranscodingUrl":null,"TranscodingSubProtocol":null,"TranscodingContainer":null,"AnalyzeDurationMs":null,"DefaultAudioStreamIndex":null,"DefaultSubtitleStreamIndex":null}
/usr/local/bin/ffmpeg -analyzeduration 200M -init_hw_device videotoolbox=vt -autorotate 0 -i file:"/Volumes/video/Series/MacGyver/Season 1/01 Pilot-C.avi" -map_metadata -1 -map_chapters -1 -threads 0 -map 0:0 -map 0:1 -map -0:s -codec:v:0 h264_videotoolbox -b:v 3203195 -maxrate 3203195 -bufsize 6406390 -force_key_frames:0 "expr:gte(t,0+n_forced*3)" -g:v:0 72 -keyint_min:v:0 72 -vf "setparams=color_primaries=bt709:color_trc=bt709:colorspace=bt709,scale=trunc(min(max(iw\,ih*a)\,min(512\,384*a))/2)*2:trunc(min(max(iw/a\,ih)\,min(512/a\,384))/2)*2,format=yuv420p" -codec:a:0 aac -ac 2 -ab 256000 -copyts -avoid_negative_ts disabled -max_muxing_queue_size 2048 -f hls -max_delay 5000000 -hls_time 3 -hls_segment_type mpegts -start_number 0 -hls_segment_filename "/Users/xxx/.local/share/jellyfin/transcodes/11f680a00e47b9fe31944789a117e0aa%d.ts" -hls_playlist_type vod -hls_list_size 0 -y "/Users/xxx/.local/share/jellyfin/transcodes/11f680a00e47b9fe31944789a117e0aa.m3u8"
ffmpeg version 6.0.1-Jellyfin Copyright © 2000-2023 the FFmpeg developers
built with Apple clang version 14.0.0 (clang-1400.0.29.202)
configuration: --prefix=/ffbuild/prefix --disable-shared --enable-static --pkg-config-flags=--static --enable-pthreads --cc=clang --host-cflags='-I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -I/opt/ffbuild/prefix/include' --host-ldflags='-L/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/lib -L/opt/ffbuild/prefix/lib' --extra-version=Jellyfin --extra-cflags=-I/opt/ffbuild/prefix/include --extra-cxxflags= --extra-ldflags=-L/opt/ffbuild/prefix/lib --extra-ldexeflags= --extra-libs= --enable-gpl --enable-version3 --enable-lto --disable-ffplay --disable-debug --disable-doc --disable-ptx-compression --disable-sdl2 --disable-libjack --disable-indev=jack --enable-neon --enable-runtime-cpudetect --enable-audiotoolbox --enable-videotoolbox --enable-iconv --enable-libxml2 --enable-zlib --enable-libfreetype --enable-libfribidi --enable-gmp --enable-openssl --enable-lzma --enable-fontconfig --enable-libvorbis --enable-opencl --disable-amf --enable-chromaprint --enable-libdav1d --disable-libfdk-aac --disable-ffnvcodec --disable-cuda --disable-cuda-llvm --disable-cuvid --disable-nvdec --disable-nvenc --enable-libass --enable-libbluray --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvpx --enable-libwebp --enable-libopenmpt --enable-libsrt --enable-libsvtav1 --enable-libx264 --enable-libx265 --enable-libzimg --enable-libzvbi
libavutil 58. 2.100 / 58. 2.100
libavcodec 60. 3.100 / 60. 3.100
libavformat 60. 3.100 / 60. 3.100
libavdevice 60. 1.100 / 60. 1.100
libavfilter 9. 3.100 / 9. 3.100
libswscale 7. 1.100 / 7. 1.100
libswresample 4. 10.100 / 4. 10.100
libpostproc 57. 1.100 / 57. 1.100
Input #0, avi, from 'file
Metadata:
software : Lavf52.32.0
Duration: 00:45:40.62, start: 0.000000, bitrate: 618 kb/s
Stream #0:0: Video: mpeg4 (Simple Profile) (xvid / 0x6469767
Stream #0:1: Audio: ac3 ([0] [0][0] / 0x2000), 44100 Hz, stereo, fltp, 96 kb/s
Stream mapping:
Stream #0:0 -> #0:0 (mpeg4 (native) -> h264 (h264_videotoolbox))
Stream #0:1 -> #0:1 (ac3 (native) -> aac (native))
Press [q] to stop, [?] for help
[h264_videotoolbox @ 0x7fcb4ff09840] Error: cannot create compression session: -12903
[h264_videotoolbox @ 0x7fcb4ff09840] Try -allow_sw 1. The hardware encoder may be busy, or not supported.
[vost#0:0/h264_videotoolbox @ 0x7fcb4ff09400] Error initializing output stream: Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height
[aac @ 0x7fcb4ff0a780] Qavg: 921.578
[aac @ 0x7fcb4ff0a780] 2 frames left in the queue on closing
Conversion failed!
RE: Apple VideoToolBox encoding failure - gnattu - 2024-04-16
This really should not happen and I tried the same command and I cannot reproduce this behavior. What is your macOS version? Just in case it is too old to be supported.
To verify if your system works with hwaccel in general, you can test with:
./ffmpeg -f lavfi -i testsrc=size=512x384:rate=23.98 -t 30 -c:v h264_videotoolbox test.mp4
in the path contains jellyfin-ffmpeg. If it still errors out for this probably minimal use case, then it is not really a jellyfin issue, but rather some problem with:
- macOS on your machine, a not compatible version or a broken installation
- or even worse, a broken Mac (less likely)
RE: Apple VideoToolBox encoding failure - grips - 2024-04-16
Crazy, that command does fail. I guess it's something with my setup, then. I'm running the latest MacOS.
Other formats seem to work without any issues. I found this old, closed github issue that looks similar: https://github.com/jellyfin/jellyfin/issues/9507
Anyway, I appreciate the time and info. I will try to look into ffmpeg more and see if I can't resolve something.