SOLVED: Jellyfin transcodes as AV1 even though it's not allowed

[pitfallfritz]

Hi,

so I just migrated my Jellyfin server from Linux/amd64 to a Mac with an M1. I've enabled hardware acceleration in the transcoding settings. Everything works, except that incompatible videos (say, an AVI with a H263 track) will be transcoded to AV1. That's a problem because the M1 doesn't support AV1 encoding yet and software encoding is slower than 1x, even on the ridiculously low resolution of the source.

Transcoding settings enable hardware decoding for everything. Hardware encoding is enabled. Under Encoding format options, I have only selected "Allow encoding in HEVC format". Still, in Chromium, playback will start after a short delay and then constantly stutter. Playback Info reveals that it's transcoding to VP9 and OPUS, for some reason, at 0.79x.

I have checked the client's playback settings in Chromium, and manually set "Preferred transcode video codec" to HEVC, as well as "preferred transcode audio codec in video playback" to AAC. Even still, I always get VP9/OPUS.

In Firefox, it will transcode to H264 instead, which is not preferred, but at least it's fast enough.


I just tested it with the old instance on Linux/amd64, and it behaves the same, with the same settings, only it takes almost longer to even start. So it's not a problem with the Mac version or AppleVideoToolbox.

Is there any way to just disable VP9 transcoding? And is this a bug (it seems like it) or am I missing something? Ideally, it should also prefer HEVC if I want to and if the client is capable (and the client settings even prefer HEVC) – but that's another story.

[TheDreadPirate]

Can you share your logs via pastebin? I'm not able to reproduce on my system.

Regarding Firefox, it does not support HEVC, period.

[gnattu]

What you said does not really make sense:

- The VP9 encoder is removed from the server entirely in 10.10
- Even with software encoding, M1 can do more than 1x on at least "ridiculously low resolution of the source."

We need your server and ffmpeg log, and how you did the "migration"

[Fate]

Can you share your logs via pastebin?  I'm not able to reproduce on my system.

Regarding Firefox, it does not support HEVC, period.

Tech technically firefox can decode hevc when:
- media.wmf.hevc.enabled is set to 1
- a h265 decoder is available in windows (like ffmpeg or h265 from microsoft store)
- on windows only
- video is not 10bit

But i think even when that is the case jellyfin-web does not allow it and forces transcoding...

[pitfallfritz]

Thank you for your replies!

So, the migration to macOS only coincided with me discovering this behavior as I was testing different types of video. To remove this from the equation, I tested this with the original copy of my server running on Debian/amd64.

I selected a video in AVI/XviD/MP3 and played it in Chromium.
Transcode log — https://pastebin.com/WyM193wd
Jellyfin log  — https://pastebin.com/cQbJcRZr
Playback Info — https://imgur.com/a/kD2nL4H or attached

VP1 transcoding speed is abysmal at 7% of realtime, although this is a low-end passive-cooled Intel Atom. The M1 reached about 80% of realtime. For comparison, the same M1 Mac encodes the same video at about 3x realtime in Handbrake using VP9 (medium, CQ25, default settings). Most of all, it should not be allowed to be transcoded to VP9.

Re. Firefox and HEVC:
Yeah, you're right and I had thought so too. Jellyfin transcoded to x264 and I didn't even notice — before the migration HEVC videos in Jellyfin just refused to play in Firefox at all. My bad.

[gnattu]

Please don't hand pick the server log.... you are removing all useful info

It is really, really weird the server can even transcode as vp9 when we have removed the config entirely on the server side. Which chromium on what OS are you using? The browser must requested a very very weird profile.

My current guess is that your chromium could be some sort of the "liberal version" which intentionally removed h264 and hevc from the supported list, leaving only av1 and vp9, which force the server to use the only available codec.

[pitfallfritz]

I started it again and get this log output:
https://pastebin.com/bxvVF8vY

The Chromium is Version 130.0.6716.0 (Developer Build) (arm64) on macOS.
This doesn't happen in other browsers, so I guess I can live with Chromium being funky. Still very odd since VPN9 support is supposedly gone from Jellyfin, as you say.

I still wonder why it's not transcoding to HEVC when I play such a video on capable browsers such as Chrome (not Chromium), Edge or Safari. Is there a way to definitively prefer H265? Other than changing the client's playback settings to HEVC, which is ignored.

Maybe Jellyfin's implementation doesn't support this on AppleVideoToolbox, even if AppleVideoTOolbox supports hardware HEVC encoding.

[gnattu]

I started it again and get this log output:
https://pastebin.com/bxvVF8vY

The Chromium is Version 130.0.6716.0 (Developer Build) (arm64) on macOS.
This doesn't happen in other browsers, so I guess I can live with Chromium being funky. Still very odd since VPN9 support is supposedly gone from Jellyfin, as you say.

I still wonder why it's not transcoding to HEVC when I play such a video on capable browsers such as Chrome (not Chromium), Edge or Safari. Is there a way to definitively prefer H265? Other than changing the client's playback settings to HEVC, which is ignored.

It should transcode to HEVC with a compatible browser. Are you really sure it is transcoding to h264 even using safari on your Mac?

BTW, it looks like your build of chromium is not built with hevc and h264 support. I recommend this build on macOS if you hate google and still want decent chromium experience: https://formulae.brew.sh/cask/eloston-chromium

This one does support hardware accelerated HEVC.

Maybe Jellyfin's implementation doesn't support this on AppleVideoToolbox, even if AppleVideoTOolbox supports hardware HEVC encoding.

Please don’t say something like this because I am the exact person who implemented this in Jellyfin.

[pitfallfritz]

It should transcode to HEVC with a compatible browser. Are you really sure it is transcoding to h264 even using safari on your Mac?

Absolutely, I just made sure again. Each client's playback settings explicitely state HEVC/AAC as preferred codecs (just making sure – ideally I would want this handled by default for all users). Server settings still say HEVC is "allowed" and AV1 is not.

Safari – H264/MP3
Chromium – VP9/OPUS
Chrome – H264/MP3
Firefox – H264/MP3 (as expected)
Edge – H264/MP3

Since MP3 can be direct played on everything, that's fine.

BTW, it looks like your build of chromium is not built with hevc and h264 support. I recommend this build on macOS if you hate google and still want decent chromium experience: https://formulae.brew.sh/cask/eloston-chromium

This one does support hardware accelerated HEVC.

True enough, Eloston-Chromium behaves as any other browser. Thanks for the tip! The one I was using was just for testing purposes and installed via brew install --cask chromium.

Maybe Jellyfin's implementation doesn't support this on AppleVideoToolbox, even if AppleVideoToolbox supports hardware HEVC encoding.

Please don’t say something like this because I am the exact person who implemented this in Jellyfin.

Hahah, okay I'll take note. Thank you for your work, I really appreciate it!

So, apparently Jellyfin still has VP9 "support" hidden somewhere. Which is only a problem if I'm using an exotic browser without even H264 support. I guess I can live with that.
But, either my HEVC preference is getting ignored or I'm missing something fundamental.

(The cheap old Atom in my old server just doesn't support HEVC encoding, period. But it should work on the M1 with VideoToolBox.)

Do you have any other idea what I could have overlooked or what I could try? Make a proper bug report somewhere?

Thank you for your help!

[gnattu]

If you go to Settings->Playback, is the enable fmp4 enabled? This is required to have HEVC in browser and is a client side setting

   

It is still strange to me as enabling this is the default settings which means this should already be enabled.

But since you have mp3 direct playing I highly suspect you are using ts instead of fmp4, because most browsers does not play mp3 in fmp4 container.