SOLVED: Trouble passing through A380 to Proxmox LXC Container

[gbschenkel]

Hi, I just remade my jellyfin container, I use AMD but I think you can follow my instruction for Intel vga card.
I am assuming you are using unprivileged container.

First for using the render device, it always need be owned to a user or a group, then you need map your host(proxmox) with you container (lxc, in my case debian).

First we gonna check the info on host:

Code:

# ls -lha /dev/dri/
crw-rw----  1 root video  226,  0 Dec 18 18:16 card0
crw-rw----  1 root video  226,  1 Dec 18 18:17 card1
crw-rw----  1 root render 226, 128 Dec 18 18:16 renderD128

In my case is for card0 226:0 ; for card1 226:1 and for renderD128 226:128. Note the user/group is root/render and root/video.

Let list then the group on host(proxmox):

Code:

# cat /etc/group | grep video
render:x:44:
# cat /etc/group | grep render
render:x:103:

Do the same inside your container:

Code:

# cat /etc/group | grep video
render:x:44:
# cat /etc/group | grep render
render:x:106:

Okay with this info, we can add some lines on your container. Lets assume your jellyfin is using id 100.
First we gonna add the devices, you will use the values found on /dev/dri.

Code:

# vi /etc/pve/lxc/100.conf
lxc.cgroup2.devices.allow: c 226:0 rwm
lxc.cgroup2.devices.allow: c 226:128 rwm

Then we gonna added the mount entry as bind to the host:

Code:

lxc.mount.entry: /dev/dri/card0 dev/dri/card0 none bind,optional,create=file
lxc.mount.entry: /dev/dri/renderD128 dev/dri/renderD128 none bind,optional,create=file

Now you need map the host render group, to the container render group.
Is a bit awkward but try keep with me, you need map all 65536 ids else it not work from what I saw.
First you need set the user, and we gonna use the default info for that:

Code:

lxc.idmap: u 0 100000 65536

This line say, user 0 (root) on the container will map to the UID 100000 on host(proxmox), and plus 65536 IDs this way.
on container will be 0 to 65535, but on the host(proxmox) it will use 100000 to 165535.

If you understand that, now we will try mapping the groups.

Since we want use the host(proxmox) render group, and its GID is 44, we need fill the gaps, first using this:

Code:

lxc.idmap: g 0 100000 44

We start from 0, 10000, and increment 44 times, which stop on the GID 43, 100043. Then we add:

Code:

lxc.idmap: g 44 44 1

We mapped container gid 44(video) to the host gid 44 on Proxmox.
Now we fill the gaps between GID 44(video) and GID 106(render)

Code:

lxc.idmap: g 45 100045 61

The calc is the actual number you want, in case 106, less the next start ID, in this case 45. 106-45=61
Well now we gonna map the GID 106(video) from container to the GID 103 on the host(proxmox).

Code:

lxc.idmap: g 106 103 1

Then fill the gaps to the end using the same calc, 65536-107=65429

Code:

lxc.idmap: g 107 100107 65429

And its done, all input data used was:

Code:

lxc.cgroup2.devices.allow: c 226:0 rwm
lxc.cgroup2.devices.allow: c 226:128 rwm
lxc.mount.entry: /dev/dri/card0 dev/dri/card0 none bind,optional,create=file
lxc.mount.entry: /dev/dri/renderD128 dev/dri/renderD128 none bind,optional,create=file
lxc.idmap: u 0 100000 65536
lxc.idmap: g 0 100000 44
lxc.idmap: g 44 44 1
lxc.idmap: g 45 100045 61
lxc.idmap: g 106 103 1
lxc.idmap: g 107 100107 65429

PS: Since I used AMD, OpenCL need use AMD ROCm, and for that I need the /dev/kfd device, since it use render group, I don't need add any GID. The kfd use 510:0, then I needed two more lines on .conf:

Code:

lxc.cgroup2.devices.allow: c 510:0 rwm
lxc.mount.entry: /dev/kfd dev/kfd none bind,optional,create=file

Also I think is not needed the video device and group, but I didn't test it yet.
I hope this can help you. I will soon try to update the documentation for AMD GPU with this info.