SOLVED: Failed to transcode to HEVC on mac mini

[hczxpaul]

Hello forum, I have a mac mini with m2 chip, and I use the Jellyfin App(the .dmg) for mac V10.8.13. I turned on hardware transcoding and selected "Apple VideoToolBox" with all option's checked(HEVC、VP9、H264, etc). However it failed when transcoding HEVC to HEVC format(hevc to h.264 works fine). ffmepg log is as uploaded to the attachment. Please help and check what is happening, thanks in advance!

the ending of the log is as follows：
Stream mapping:
  Stream #0:0 -> #0:0 (hevc (native) -> hevc (hevc_videotoolbox))
  Stream #0:1 -> #0:1 (truehd (native) -> aac (native))
Press [q] to stop, [?] for help
[hevc_videotoolbox @ 0x7fa676717000] Error: cannot create compression session: -12908
[hevc_videotoolbox @ 0x7fa676717000] Try -allow_sw 1. The hardware encoder may be busy, or not supported.
Error initializing output stream 0:0 -- Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height
[aac @ 0x7fa6767163c0] Qavg: 65536.000
[aac @ 0x7fa6767163c0] 2 frames left in the queue on closing
Conversion failed!

[TheDreadPirate]

Are you using Vargol's ffmpeg build?

[hczxpaul]

Are you using Vargol's ffmpeg build?

I did not install ffmpeg seperately, I think Jellyfin comes with it's own ffmpeg which is what I am using. It also wouldn't let me change the path of ffmpeg(plz see the screenshot attached).

[TheDreadPirate]

Are you using Docker?

You can still change the ffmpeg path in a config file.

/config/config/encoder.xml is the container path.

AFAIK, you need to use the Vargol FFMPEG build for Apple Silicon.

[hczxpaul]

[attachment=2606]

Are you using Docker?

You can still change the ffmpeg path in a config file.

/config/config/encoder.xml is the container path.

AFAIK, you need to use the Vargol FFMPEG build for Apple Silicon.

I am using the .dmg installer. I tried the docker version before, it seems like docker version cannot transcode via hardware accl on apple silicon at all.

I tried to build Vargol ffmpeg but failed every time, I ended up download a ffmpeg bianry for apple silicon here:https://www.osxexperts.net/.

The next thing I did I go to jellyfin config folder, and find encoding.xml(no encoder.xml found) and change

Code:

  <EncoderAppPathDisplay>ffmpeg</EncoderAppPathDisplay>

to  

Code:

<EncoderAppPathDisplay>/usr/local/lib/ffmpeg</EncoderAppPathDisplay>

However the encoding.xml seems to be reset by the jellyfin server every time it starts up, so I think path of ffmpeg cannot be changed by modifing the encoding.xml, please correct me if I did something wrong.  PEGVargol FFMPEGVargol FFMPEG

[nardisowhat]

I don’t want to hijack this thread but wanted to say that I tried jellyfin 10.8.13 on my M1 Mac mini a few weeks ago. I am trying to move away from Plex and getting a Plex pass for hardware transcoding. Plex macOS App is Apple Silicon native.

Since jellyfin 10.8.13 isn’t Apple Silicon native yet, I have low expectations. 10.9 is supposed to be Apple Silicon native but there’s no info when 10.9 stable is due out. Software transcoding is ok on Jellyfin 10.8.13 for macOS using Rosetta. I do see a lot of high iGPU and CPU usage when transcoding HEVC movies for jellyfin clients that aren’t capable of playing it directly. Also when scanning libraries, it hangs.

I am hoping when 10.9 comes out as Apple Silicon native this will be corrected.

Out of curiosity, last week I picked up a miniPC n100 for $179 last week on Amazon and tried out jellyfin on it in just windows 11 pro and it’s amazing and efficient at hardware transcoding HEVC. I got the miniPC to use proxmox, pfsense, pi-hole but was curious how Jellyfin would operate. Ideally, I will use the miniPC for proxmox, pfsense and pi-hole solely and use Jellyfin on my M1 Mac mini when 10.9 comes out.

[hczxpaul]

I don’t want to hijack this thread but wanted to say that I tried jellyfin 10.8.13 on my M1 Mac mini a few weeks ago.  I am trying to move away from Plex and getting a Plex pass for hardware transcoding.  Plex macOS App is Apple Silicon native. 

Since jellyfin 10.8.13 isn’t Apple Silicon native yet,  I have low expectations.  10.9 is supposed to be Apple Silicon native but there’s no info when 10.9 stable is due out.    Software transcoding is ok on Jellyfin 10.8.13 for macOS using Rosetta.  I do see a lot of high iGPU and CPU usage when transcoding HEVC movies for jellyfin clients that aren’t capable of playing it directly.    Also when scanning libraries, it hangs.

I am hoping when 10.9 comes out as Apple Silicon native this will be corrected.

Out of curiosity, last week I picked up a miniPC n100 for $179 last week on Amazon and tried out jellyfin on it in just windows 11 pro and it’s amazing and efficient at hardware transcoding HEVC.  I got the miniPC to use proxmox, pfsense, pi-hole but was curious how Jellyfin would operate.  Ideally, I will use the miniPC for proxmox, pfsense and pi-hole solely  and use Jellyfin on my M1 Mac mini when 10.9 comes out.

I tend to belive that its not an issue with Rosseta since everything other than harware transcoding on HEVC, seems to work fine. I personally did not encounter problem with library scaning(about 3TB library size). 

Not like yours, I didn't see iGPU usage for ffmpeg at all(almost 0% at all time), without hardware transcoding CPU usage is like full 800%, and with hardware transcoding on, CPU
usage is around 200%.

So I think the problem now is that I cannot modify the path of ffmpeg via editing the "encoding.xml" in config file, so I cannot confirm if it is the problem the ffmpeg build. Is there a way to change the ffmpeg path in Jellyfin 10.8.13?

[nyanmisaka]

Last week our team welcomed a MacOS developer @gnattu who is familiar with FFmpeg, which allows us to effectively improve the transcoding performance of Jellyfin Server on MacOS, and change the current status of Jellyfin HWA only works great on Linux and Windows. I'll pass this issue to him and see if he has any ideas.

[gnattu]

Your system complained about that it does not find an hardware encoder. Most likely you are using an x86 ffmpeg and programs running with rosetta will have problems find the correct VT encoder.

You can use file $(which ffmpeg) in console to see if it is the case.

For now, your best option is to install homebrew via https://brew.sh, and use brew install ffmpeg to use their ffmpeg instead. Theirs will have correct hardware acceleration behavior.

[gnattu]

By the way the <EncoderAppPathDisplay>ffmpeg</EncoderAppPathDisplay> is only how the path is displayed, you need to modify <EncoderAppPath>ffmpeg</EncoderAppPath> as well.

If you are using homebrew's ffmpeg, then it will look like thie:


<EncoderAppPath>/opt/homebrew/bin/ffmpeg</EncoderAppPath>
<EncoderAppPathDisplay>/opt/homebrew/bin/ffmpeg</EncoderAppPathDisplay>