SOLVED: Can't play handbrake

[onetoomany]

Playback failed due to a fatal player error.
I can play other transcoded H264 movies from WinX DVD in Jellyfin. I can play my trancoded handbrakes in VLC. I can't play them in Jellyfin unless I disable hardware acceleration.

I'm hosting on Proxmox as a container.
Playback settings:
Video Acceleration API (VAAPI)
/dev/dri/renderD128


The log files give me:
[AVHWDeviceContext @ 0x5b07bc82a4c0] No VA display found for any default device.
[AVFilterGraph @ 0x5b07bc822700] Error initializing filter 'hwupload_vaapi'
Error initializing complex filters.
Invalid argument

Movie details:

Code:

FailedMovie
Container: mp4
Path: /mnt/nas/movies/DVDs/FailedMovie.mp4
Size: 1200 MB

Video
Title: 480p H264 SDR
Codec: H264
Codec tag: avc1
AVC: Yes
Profile: Main
Level: 40
Resolution: 720x428
Aspect ratio: 2.40:1
Anamorphic: No
Interlaced: No
Framerate: 25
Bitrate: 684 kbps
Bit depth: 8 bit
Video range: SDR
Video range type: SDR
Colour space: smpte170m
Colour transfer: bt709
Colour primaries: bt470bg
Pixel format: yuv420p
Ref frames: 1
NAL: 4

Audio
Title: Stereo - English - AAC - Default
Language: eng
Codec: AAC
Codec tag: mp4a
AVC: No
Profile: LC
Layout: stereo
Channels: 2 ch
Bitrate: 161 kbps
Sample rate: 48000 Hz
Default: Yes
Forced: No
External: No

Subtitle
Title: Danish - Default - DVDSUB
Language: dan
Codec: DVDSUB
Codec tag: mp4s
AVC: No
Resolution: 720x576
Bitrate: 2 kbps
Default: Yes
Forced: No
External: No

Audio
Title: Visually Impaired - English - AAC - Stereo
Language: eng
Codec: AAC
Codec tag: mp4a
AVC: No
Profile: LC
Layout: stereo
Channels: 2 ch
Bitrate: 162 kbps
Sample rate: 48000 Hz
Default: No
Forced: No
External: No

Subtitle
Title: English - DVDSUB
Language: eng
Codec: DVDSUB
Codec tag: mp4s
AVC: No
Resolution: 720x576
Bitrate: 3 kbps
Default: No
Forced: No
External: No

Subtitle
Title: Finnish - DVDSUB
Language: fin
Codec: DVDSUB
Codec tag: mp4s
AVC: No
Resolution: 720x576
Bitrate: 2 kbps
Default: No
Forced: No
External: No

Subtitle
Title: Icelandic - DVDSUB
Language: isl
Codec: DVDSUB
Codec tag: mp4s
AVC: No
Resolution: 720x576
Bitrate: 2 kbps
Default: No
Forced: No
External: No

Subtitle
Title: Nor - DVDSUB
Language: nor
Codec: DVDSUB
Codec tag: mp4s
AVC: No
Resolution: 720x576
Bitrate: 2 kbps
Default: No
Forced: No
External: No

Subtitle
Title: Swedish - DVDSUB
Language: swe
Codec: DVDSUB
Codec tag: mp4s
AVC: No
Resolution: 720x576
Bitrate: 2 kbps
Default: No
Forced: No
External: No

[wenzelja]

Un-tick H.264 in both decode and encode in the transcode settings. Restart the server after saving, and try to play them.

[TheDreadPirate]

What Jellyfin version are you running? Looks like 10.8, maybe 10.9.

What are your hardware specs? Did you pass the GPU into the LXC? Is the LXC privileged or unprivileged?

[onetoomany]

Un-tick H.264 in both decode and encode in the transcode settings. Restart the server after saving, and try to play them.

Sadly, this didn't work - problem still remains.

[TheDreadPirate]

We still need your hardware specs and the other info I asked for.

[onetoomany]

What Jellyfin version are you running?  Looks like 10.8, maybe 10.9.

What are your hardware specs?  Did you pass the GPU into the LXC?  Is the LXC privileged or unprivileged?

I was ON 10.9.11, but have just upgaded to 10.10.3. Upgrade didn't solve the issue.
GPU should be passthrough to the LXC; 
Unprivileged container: No

Hardware specs - I've got an Intel Core i5 11400 11th Gen Rocket Lake 6 Processor, and a separate AMD Radeon R7 - the latter should be passed through.

LXC settings:

Code:

arch: amd64
cores: 2
features: nesting=1
hostname: jellyfin
memory: 2048
net0: name=eth0,bridge=vmbr0,gw=10.66.131.1,hwaddr=32:1A:00:92:53:34,ip=10.66.131.14/24,type=veth
onboot: 1
ostype: ubuntu
parent: vzdump
rootfs: local-lvm:vm-107-disk-1,size=8G
startup: order=10,up=120
swap: 512
lxc.cgroup2.devices.allow: c 226:0 rwm
lxc.cgroup2.devices.allow: c 226:128 rwm
lxc.cgroup2.devices.allow: c 29:0 rwm
lxc.mount.entry: /dev/fb0 dev/fb0 none bind,optional,create=file
lxc.mount.entry: /dev/dri dev/dri none bind,optional,create=dir
lxc.mount.entry: /dev/dri/renderD128 dev/renderD128 none bind,optional,create=file

With lshw from the jellyfin lxc console:

Code:

          *-display
                description: VGA compatible controller
                product: Oland PRO [Radeon R7 240/340]
                vendor: Advanced Micro Devices, Inc. [AMD/ATI]
                physical id: 0
                bus info: pci@0000:01:00.0
                version: 00
                width: 64 bits
                clock: 33MHz
                capabilities: pm pciexpress msi vga_controller cap_list rom
                configuration: driver=vfio-pci latency=0
                resources: irq:16 memory:50000000-5fffffff memory:61500000-6153ffff ioport:6000(size=256) memory:61540000-6155ffff

[TheDreadPirate]

Your R7 has no encoder. You Intel iGPU, on the other hand, has a MUCH MUCH better encoder. Shut down your PC, remove the AMD GPU, power it back on and pass in the Intel iGPU.

[onetoomany]

Thanks for this. You're spot on - it was an issue with passing through the AMD card.
I've changed my passthrough settings to give just the intel GPU, and this is now working. Thanks again.

Aargh - hit the wrong post for solution - should be #7 not #6. My bad thinking the button was at the top of the post, not the bottom.