Transcoding VAAPI error 187

[simuz]

Hello everyone, I'm struggling on transcoding and HW acceleration. I'm kind of a noob about video codecs and HWA drivers, so excuse me in advance if I'm missing something obvious.

My setup

• CPU: INTEL Alder Lake n95
  
• SSD1: 512GB internal
  
• SSD2: 2TB internal
  
• OS: Debian 12
  
• Jellyfin running on Docker
  
• Jellyfin v10.10.6
  
• System and Docker on SSD1, media on SSD2
  

The issue
I had hardware acceleration (VAAPI) configured correctly for a while, and it was working fine. Then I tried watching a new series and started having issues—but only with the Windows app. I worked around it by opening Jellyfin in a browser (Google Chrome on Windows 10), and that worked.

Yesterday, for the first time, Chrome also failed (possibly after upgrading the client to Windows 11 that started triggering the transcoding? I'm not sure, but that seems to be the only difference). Now it looks like transcoding doesn't work at all anymore—not even for other media—and it always returns an error, and I do not think that this could be Win11 fault.

It still works on the TV app, but I checked and it's using direct play. Why doesn’t the native Windows app use direct play as well?

The video that originally caused issues was H264, but I'm currently having problems with HEVC as well.

No changes have been done to settings or configuration.

Drivers
/dev/dri contains card0 and renderD128

Transcoding settings
HWA: VAAPI
Device: /dev/dri/renderD128
Enabled for:

• H264
  
• HEVC
  
• MPEG2
  
• VC1
  
• VP8
  
• VP9
  

HW encoding:            Enabled
Low-Power encoding: Disabled (both H264 and HEVC)
HEVC encoding:         Enabled
AV1 encoding:           Enabled
FFMpeg path:            /usr/lib/jellyfin-ffmpeg/ffmpeg

Logs
Log: https://pastebin.com/prFY8egZ
Transcode log: https://pastebin.com/JBRbuQm4


I hope I’ve provided enough details—if not, I’ll be happy to add whatever is needed.

Thank you in advance!

[simuz]

It seems that I managed to solve the issue, with substantial help from ChatGPT. The first problem was that I hadn’t added the "render" group to the Docker Compose (in fact, I had to use the group ID because within the container it seems there is only "video," associated with card0, while renderD128 is associated with "render"). By doing this, transcoding started working with the apps.

To fix the issue with Chrome, I disabled AV1 encoding instead. Should I keep HEVC on?

I have no idea what changed since last week, when the groups weren’t set (not even "video") and things were still working fine.

[nyanmisaka]

Intel N95 does not have AV1 hw encoder. So you need to disable it. H264 and HEVC are both supported encoders.

Apart from that, it's a very classic permissions issue.