ffmpeg custom flags for VAAPI

[triarii]

VAAPI Encoding Filter Chain Issue on SR-IOV GPU Passthrough

CONTEXT
=======
I am testing Jellyfin on Proxmox VMs with an Intel Arc B50 using SR-IOV passthrough (the gpu is split up by the host proxmox system). I understand this is beyond typical use cases and very unusual, so I appreciate that this is an odd request. I wanted to report this because your FFmpeg build does work correctly here - it just needs different filter flags for SR-IOV GPUs.

I tried Quick Sync first but it fails with "Error creating a MFX session: -9" on SR-IOV - seems to be a limitation with Intel's implementation on virtual GPUs. The GPU is properly detected and the Xe driver is running fine.



ISSUE SUMMARY
=============
When using VAAPI hardware encoding on SR-IOV GPU passthrough devices (e.g., Intel Arc B50), Jellyfin's default video filter chain fails with exit code 187, causing playback transcoding to fail. The issue occurs because the filter chain is missing the hwupload filter required for proper hardware surface handling on SR-IOV GPUs.

TECHNICAL DETAILS
=================

Current Jellyfin VAAPI filter (fails):
-vf "setparams=color_primaries=bt709:color_trc=bt709:colorspace=bt709,scale=...,format=nv12,hwupload_vaapi"

Note: hwupload_vaapi does not exist as a valid filter.


Working FFmpeg command called from the VM using the installed (native) version of Jellyfin (Arch Cachy OS) with proper filter chain:

Code:

/usr/lib/jellyfin-ffmpeg/ffmpeg \
  -init_hw_device vaapi=va:/dev/dri/renderD128 \
  -f lavfi -i color=c=blue:s=1280x720:d=5 \
  -vf 'format=nv12,hwupload' \
  -c:v h264_vaapi \
  -f null -

Results: Encodes at 25x real-time speed (125 frames in 5 seconds)


REPRODUCTION STEPS
==================

(sorry that this is such a bizarre edge case)

1. Set up Jellyfin on a system with SR-IOV GPU passthrough (Intel Arc B50 tested)
2. Configure VAAPI in Jellyfin settings pointing to /dev/dri/renderD128
3. Attempt to play a video that requires transcoding
4. FFmpeg exits with code 187; playback fails



SR-IOV SETUP ON PROXMOX 9.1 for Intel Arc B50
===========================

(this is just for information I don't expect other people to do this)


1. Install Proxmox 9.1 and update
2. apt install sysfsutils
3. Add to /etc/sysfs.conf:
  #or whatever your specific device pcie address is
  devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:01.0/0000:03:00.0/sriov_numvfs = 3
4. Reboot - you should now have 3 virtual functions using the Xe driver
5. Create VM (tested with ubuntu server 25.10 and CachyOS)
6. Pass the VF to the VM - it should automatically detect and use the Xe driver


ENVIRONMENT
===========
Jellyfin: 10.11.5
FFmpeg: 7.1.3
GPU: Intel Arc B50 (SR-IOV passthrough)
OS: Ubuntu 25.10, CachyOS (kernel 6.17 / 6.18 )
Device: /dev/dri/renderD128


REQUEST
=======
Could the Jellyfin developers consider:

1. Some ability in the Transcoding Page to allow for custom flags like adding hwupload filter to the VAAPI filter chain

or

2. Making the VAAPI filter chain configurable via encoding.xml or environment variable, allowing users to override the default filters

The fix appears straightforward for the user to change the filter chain from:
  scale=...,format=nv12,hwupload_vaapi
to:
  scale_vaapi=...,format=nv12,hwupload

thanks for reading this, sorry that this is such an edge case.

[nyanmisaka]

> Error creating a MFX session: -9

This shouldn't happen, neither on the host nor in the VM. If it does, check the dmesg log to see if the firmwares required by the xe driver has been loaded correctly.

hwupload_vaapi should not appear in a normal transcoding pipeline; if it does, it means it's using compatibility mode. Restart the Jellyfin server and try again.

There's no reason to use VAAPI on such a new Intel graphics card， switch to QSV and try again.

[DomeShifter]

I run Jellyfin on Proxmox and had a Jellyfin VAAPI failure recently, but in my case VAAPI never initialised due to container passthrough issues.
In your case, VAAPI does initialise and the failure happens later in the FFmpeg filter chain, which makes this a Jellyfin VAAPI pipeline bug rather than a Proxmox or GPU passthrough issue.

It seems as though a lot of Jellyfin’s hardware acceleration logic assumes a fairly “normal” GPU setup. If you step outside that... containers, passthrough, SR-IOV, virtual GPUs, etc. you can end up in situations where everything looks okay (device present, VAAPI available, FFmpeg runs) but playback still fails because one of those assumptions no longer holds.

I hope you find a solution -  Jellyfin on Proxmox is a lovely setup.