SOLVED: Intel QSV HEVC Issues

[Jokertyf]

I'm running Jellyfin server 10.8.10.0 on a Ubuntu server with an Intel CPU (Gen 11, integrated graphics). On some specific files, I get an ffmpeg memory allocation bug when trying to do hardware acceleration via QSV on HEVC encoded videos. The error message always looks like this:



[Parsed_scale_qsv_4 @ 0x56365ccf2e00] Can't allocate a surface.
[Parsed_scale_qsv_4 @ 0x56365ccf2e00] Failed to query an output frame.
Error while filtering: Cannot allocate memory
Failed to inject frame into filter network: Cannot allocate memory
Error while processing the decoded data for stream #0:0
Conversion failed!


This happens no matter where I start playback from, but it always occurs 2-5 minutes before the end of the video. It's strange because when I run htop to see memory usage, I still have plenty of RAM available, so I'm not sure why ffmpeg would be running out of it. 

I don't think the files are corrupted because I can play them with no issues using VLC. Also, if I re-encode the video to use h264 then Jellyfin can handle it no problem. However, this only happens with certain HEVC files, and I can't tell what separates the ones that work from the ones that don't.

Has anyone seen an error like this before? I'd like to avoid re-encoding a bunch of files if possible.

[TheDreadPirate]

Try updating your motherboard's BIOS. Some other users with 11th Gen Intel GPUs had unexplained issues with transcoding fixed by a BIOS update.

https://forum.jellyfin.org/t-solved-hevc...w-decoding

[Jokertyf]

I upgraded to the latest BIOS, and the latest jellyfin-server as well because why not. I still seem to get the error though.

[TheDreadPirate]

What Ubuntu version? And what kernel are you running?

[Jokertyf]

Ubuntu 22.04.3 LTS, Linux 5.15.0-69-generic x86_64

[Jokertyf]

I've managed to trace this down a bit. It looks like the encoding will fail exactly when the subtitle track "runs out of data". That is, the video stream will be e.g. 25:00 long, and the subtitle stream will be 23:00 long. It will fail basically right at 23:00, as far as I can tell.

When I remove the subtitle track, the same thing will happen if the audio stream is shorter than the video stream. In that case however, the transcoding will stop, then skip to the end exactly 45 seconds after the audio stream ends.

With that in mind, has anyone seen behavior like this before?

[TheDreadPirate]

I have not. But we had a similar issue with Roku clients crashing when only a video client is present.

Submit an issue here.

https://github.com/jellyfin/jellyfin-android/issues

[Jokertyf]

I think I finally found it. Looks like it wasn't a QSV issue at all:

https://github.com/jellyfin/jellyfin/pull/10710

The fix isn't in the latest release, but because I'm so sure this is it, I'll mark as solved.