SOLVED: Trouble passing through A380 to Proxmox LXC Container

[umaxtu]

Solution: So I noticed that the numbers in the 6th column of the ls -l output didn't match the numbers after the colon on the lxc.cgroup2.devices.allow lines in the container config file. I changed the numbers in the config file to match what I was seeing in the container (1 and 129 respectively) and it worked! I have no idea what those numbers represent, but it worked!

Problem:
I'm trying to pass through and Intel Arc A380 to a proxmox lxc container (debian bookworm container), but when I try to run /usr/lib/jellyfin-ffmpeg/vainfo --display drm --device /dev/dri/renderD128
I get:

Code:

Trying display: drm

Failed to open the given device!

I've also tried the non-jellyfin vainfo and only get:

Code:

Failed to open the given device!

Here is my config file for the container:

Code:

arch: amd64
cores: 2
hostname: airbus
memory: 2048
net0: name=eth0,bridge=vmbr0,firewall=1,hwaddr=BC:24:11:C2:07:79,ip=dhcp,type=veth
ostype: debian
rootfs: pool:subvol-102-disk-0,size=8G
swap: 512
lxc.cgroup2.devices.allow: c 226:0 rwm
lxc.cgroup2.devices.allow: c 226:128 rwm
lxc.mount.entry: /dev/dri/card1 dev/dri/card0 none bind,optional,create=file
lxc.mount.entry: /dev/dri/renderD129 dev/dri/renderD128 none bind,optional,create=file

output of ls -l /dev/dri on proxmox:

Code:

ls -l /dev/dri
total 0
drwxr-xr-x 2 root root        120 Dec 14 22:57 by-path
crw-rw---- 1 root video  226,  0 Dec 14 22:57 card0
crw-rw---- 1 root video  226,  1 Dec 14 22:57 card1
crw-rw---- 1 root render 226, 128 Dec 14 22:57 renderD128
crw-rw---- 1 root render 226, 129 Dec 14 22:57 renderD129

in the container:

Code:

ls -l /dev/dri
total 0
crw-rw---- 1 root video 226,  1 Dec 15 03:57 card0
crw-rw---- 1 root input 226, 129 Dec 15 03:57 renderD128

I'm pretty sure I've got the firmware setup correctly because I installed vainfo on proxmox itself and when I run vainfo --display drm --device /dev/dri/renderD129
It works, and I know its talking to the A380 vs the iGPU because I see AV1 listed as supported.

Another interesting tidbit is that I edit the container config to passthrough the iGPU, it works.

Hopefully I am missing something obvious.

[gbschenkel]

Hi, I just remade my jellyfin container, I use AMD but I think you can follow my instruction for Intel vga card.
I am assuming you are using unprivileged container.

First for using the render device, it always need be owned to a user or a group, then you need map your host(proxmox) with you container (lxc, in my case debian).

First we gonna check the info on host:

Code:

# ls -lha /dev/dri/
crw-rw----  1 root video  226,  0 Dec 18 18:16 card0
crw-rw----  1 root video  226,  1 Dec 18 18:17 card1
crw-rw----  1 root render 226, 128 Dec 18 18:16 renderD128

In my case is for card0 226:0 ; for card1 226:1 and for renderD128 226:128. Note the user/group is root/render and root/video.

Let list then the group on host(proxmox):

Code:

# cat /etc/group | grep video
render:x:44:
# cat /etc/group | grep render
render:x:103:

Do the same inside your container:

Code:

# cat /etc/group | grep video
render:x:44:
# cat /etc/group | grep render
render:x:106:

Okay with this info, we can add some lines on your container. Lets assume your jellyfin is using id 100.
First we gonna add the devices, you will use the values found on /dev/dri.

Code:

# vi /etc/pve/lxc/100.conf
lxc.cgroup2.devices.allow: c 226:0 rwm
lxc.cgroup2.devices.allow: c 226:128 rwm

Then we gonna added the mount entry as bind to the host:

Code:

lxc.mount.entry: /dev/dri/card0 dev/dri/card0 none bind,optional,create=file
lxc.mount.entry: /dev/dri/renderD128 dev/dri/renderD128 none bind,optional,create=file

Now you need map the host render group, to the container render group.
Is a bit awkward but try keep with me, you need map all 65536 ids else it not work from what I saw.
First you need set the user, and we gonna use the default info for that:

Code:

lxc.idmap: u 0 100000 65536

This line say, user 0 (root) on the container will map to the UID 100000 on host(proxmox), and plus 65536 IDs this way.
on container will be 0 to 65535, but on the host(proxmox) it will use 100000 to 165535.

If you understand that, now we will try mapping the groups.

Since we want use the host(proxmox) render group, and its GID is 44, we need fill the gaps, first using this:

Code:

lxc.idmap: g 0 100000 44

We start from 0, 10000, and increment 44 times, which stop on the GID 43, 100043. Then we add:

Code:

lxc.idmap: g 44 44 1

We mapped container gid 44(video) to the host gid 44 on Proxmox.
Now we fill the gaps between GID 44(video) and GID 106(render)

Code:

lxc.idmap: g 45 100045 61

The calc is the actual number you want, in case 106, less the next start ID, in this case 45. 106-45=61
Well now we gonna map the GID 106(video) from container to the GID 103 on the host(proxmox).

Code:

lxc.idmap: g 106 103 1

Then fill the gaps to the end using the same calc, 65536-107=65429

Code:

lxc.idmap: g 107 100107 65429

And its done, all input data used was:

Code:

lxc.cgroup2.devices.allow: c 226:0 rwm
lxc.cgroup2.devices.allow: c 226:128 rwm
lxc.mount.entry: /dev/dri/card0 dev/dri/card0 none bind,optional,create=file
lxc.mount.entry: /dev/dri/renderD128 dev/dri/renderD128 none bind,optional,create=file
lxc.idmap: u 0 100000 65536
lxc.idmap: g 0 100000 44
lxc.idmap: g 44 44 1
lxc.idmap: g 45 100045 61
lxc.idmap: g 106 103 1
lxc.idmap: g 107 100107 65429

PS: Since I used AMD, OpenCL need use AMD ROCm, and for that I need the /dev/kfd device, since it use render group, I don't need add any GID. The kfd use 510:0, then I needed two more lines on .conf:

Code:

lxc.cgroup2.devices.allow: c 510:0 rwm
lxc.mount.entry: /dev/kfd dev/kfd none bind,optional,create=file

Also I think is not needed the video device and group, but I didn't test it yet.
I hope this can help you. I will soon try to update the documentation for AMD GPU with this info.