Jellyfin Forum
Proxmox LXC with Nvidia Transcoding and Network Share - Printable Version

+- Jellyfin Forum (https://forum.jellyfin.org)
+-- Forum: Support (https://forum.jellyfin.org/f-support)
+--- Forum: Guides, Walkthroughs & Tutorials (https://forum.jellyfin.org/f-guides-walkthroughs-tutorials)
+--- Thread: Proxmox LXC with Nvidia Transcoding and Network Share (/t-proxmox-lxc-with-nvidia-transcoding-and-network-share)

Proxmox LXC with Nvidia Transcoding and Network Share - LordRatner - 2023-12-29

Hi there. I just wrote up a guide for getting Jellyfin to work on Proxmox in an LXC with GPU transcoding. You can see the whole thread here, in case this thread is missing the answers you seek: https://forum.proxmox.com/threads/jellyfin-lxc-with-nvidia-gpu-transcoding-and-network-storage.138873/

Sorry for the formatting, I couldn't figure out if this forum has inline code functionality. It looks cleaner on the ProxMox forum.

The Overview:

We're looking for an unprivileged LXC to serve as a Jellyfin server. We want to use an Nvidia GPU (in this case a GTX 1070Ti) that is also used by other LXCs for other services. The sharing of the GPU is why we are not considering using VMs. Additionally, this setup uses an NFS share for the video library, which has a couple other requirements.


Phase one: Installing Jellyfin
  1. We'll start by creating a Jellyfin LXC using on of the TTECK scripts available here: https://tteck.github.io/Proxmox/
    • Follow the instructions using the host shell in the Proxmox GUI
    • I recommend using the Advanced options, so you know what you're getting. I use:
      • 2 or 4 cores
      • 4096M RAM (two gb should suffice)
      • 16G Disk (this will need to be much larger if you plan on many simultaneous transcodes until Jellyfin 10.9 comes out with live transcode clearing)
      • Debian 12 (Ubuntu is fine if that's your preference)
      • Static IP address
      • IP6 Disabled
      • Root SSH enabled - You'll need this if you are rebuilding Jellyfin and want to save all your previous data and settings.
  2. Once created, open the console and stop the jellyfin service
    Code:
    systemctl stop jellyfin
  3. (Optional - Transfer old settings) If you are transferring from a previous install, you need to transfer the
    Code:
    /var/lib/jellyfin
    directory (ignoring the transcodes folder) from the old server to the new one.
    • WinSCP works well for this, if you remembered to enable root SSH access. If you forgot, open
      Code:
      /etc/ssh/sshd_config
      and change the
      Code:
      PermitRootLogin
      option to 'yes' and uncomment it if applicable.
      Code:
      systemctl restart sshd
      to make it active.
    • Transfer the directories
    • In the new Jellyfin LXC, navigate to
      Code:
      /var/lib/jellyfin
      and fix the ownership with
      chown -R jellyfin:jellyfin *
  4. (Optional - Shared Media folder) Create the mount folder where your shared media library is accessed on the Jellyfin LXC. In this example:
    Code:
    mkdir /mnt/theater
  5. Shutdown the LXC
  6. (Optional - Shared Media folder) We need to give the LXC access to your media library. I'm using NFS to do this, so you'll need to make sure whatever server is hosting your media is set up to accept NFS sharing. We are going to set up the NFS client access on the proxmox host, not the LXC. This will make migrating and snapshotting a bit easier. You can use CIFS, but you'll have to make sure the permissions and user/group settings allow for read and write access.
    • On all of your proxmox nodes, create a folder to mount to. In this example we are going to use
      Code:
      /mnt/lxc-share
      . It must be the same on all nodes.
    • Mount the folder to your shared media using the method of your choosing. Test to make sure your files are visible, and that you can add and edit files from the proxmox host. In my case NFS via autofs creates
      Code:
      /mnt/lxc-share/theater
    • In the proxmox host, open the LXC configuration file (using LXC 140 in this example)
      Code:
      nano /etc/pve/lxc/140.conf
    • Mount the media folder into the LXC using this line at the end of your configuration:
      Code:
      lxc.mount.entry: /mnt/lxc-share/theater mnt/theater none bind 0 0
      • We are using this mounting method because it allows for LXC snapshots.
      • By mounting the NFS share on the Proxmox host instead of the actual LXC, you have fewer share clients to create and setup. One connection per node, then as many LXCs as you want can access the share using the configuration entry.
  7. Start the LXC and confirm.
    • In the LXC console, navigate to your mounted media drive (Example:
      Code:
      /mnt/theater
      ) and confirm that you can see your media and add/edit files.
    • Code:
      systemctl start jellyfin
      Then check to see you are able to navigate to and access jellyfin. If this is a fresh install, you can start setting everything up through Jellyfin, or wait till after we've added the GPU. If this is a transfer, you should be able to log in using your old credentials, but it will ask you to set the admin password. Note, LDAP settings will transfer and work successfully.
  8. Take a snapshot of the LXC. In this example we are calling the snapshot "post-install."
  9. Shutdown the LXC
  10. Did you take a snapshot?
  11. Seriously, make sure you took a snapshot.

Phase Two: Adding the GPU.

This part is a pain. The problem is that Jellyfin requires two libraries, libnvcuvid1 and libnvidia-encode1, to work. You can read the documentation here: https://jellyfin.org/docs/general/administration/hardware-acceleration/nvidia#linux-setups

But installing these libraries forces whatever Nvidia drivers your current repositories consider current. This is a problem because the drivers you use on your host must match the drivers you use in the LXC. You have to install the GPU on the host first to be able to install it in the LXC, but you might not end up with the same drivers, so you end up having to go through this process. Do not start doing this yet, just read the bullets below to get an idea of the process:
  • Install drivers on the host. There is a whole world of how-tos explaining how to install an Nvidia GPU on Proxmox. Get to the point where you can run
    Code:
    nvidia-smi
    and see your card listed. I strongly recommend using this process for selecting the drivers: https://github.com/keylase/nvidia-patch#step-by-step-guide. Just the drivers for now, we will address the patch later. Pick some relatively newer drivers.
  • Make sure you snapshotted your Jellyfin LXC! Install the two libraries on the LXC following the Jellyfin Instructions: https://jellyfin.org/docs/general/administration/hardware-acceleration/nvidia#linux-setups This will download the drivers identified as required packages, and you will be able to see what drivers you need to install on the host. Make note of the driver version.
  • Rollback the LXC to the snapshot, removing the drivers and packages.
  • Back on the host, unsinstall the previous drivers
    Code:
    ./NVIDIA-Linux-x86_64-430.50.run --uninstall
    (edit for the version you have) and go through the directions for downloading and installing the needed version.
  • Repeat the steps from the jellyfin instructions to install the two libraries.
  • Test to make sure the card is transcoding.
Got it? It sucks, I know, but if you can figure out how to install the two required libraries using the Nvidia drivers of your choice, instead of the repository drivers, please let me know.

Here we go:
  1. Install the GPU on your Proxmox host(s). I won't cover this process, you can find the steps online. Use the driver download, not the repository, found here: https://github.com/keylase/nvidia-patch#step-by-step-guide . You will need to disable nuveau, and all sorts of other obnoxious steps. Google "proxmox lxc plex transcode" and you should find guides, just do the host steps, not the LXC steps.
  2. Shut down the LXC.
  3. On the host,
    Code:
    ls -l /dev/nvidia*
    . We have to pass these to the LXC. Take note of the two numbers separated by a comma, after the group and before the date. You can use both numbers, or just the first number with an asterisk. You should see the following lines, and we are going to pass them all:
    • /dev/nvidia0
    • /dev/nvidiactl
    • /dev/nvidia-modset
    • /dev/nvidia-uvm
    • /dev/nvidia-uvm-tools
    • /dev/nvidia-caps/nvidia-cap1
    • /dev/nvidia-caps/nvidia-cap12
  4. To pass these to the LXC you need the following lines in the configuration file
    Code:
    /etc/pve/lxc/140.conf
    . Note that you need to replace the bold/underlined numbers with the ones from your system:
    • lxc.cgroup2.devices.allow: c 195:* rwm
    • lxc.cgroup2.devices.allow: c 234:* rwm
    • lxc.cgroup2.devices.allow: c 235:* rwm
    • lxc.cgroup2.devices.allow: c 238:* rwm
    • lxc.mount.entry: /dev/nvidia0 dev/nvidia0 none bind,optional,create=file
    • lxc.mount.entry: /dev/nvidiactl dev/nvidiactl none bind,optional,create=file
    • lxc.mount.entry: /dev/nvidia-uvm dev/nvidia-uvm none bind,optional,create=file
    • lxc.mount.entry: /dev/nvidia-modeset dev/nvidia-modeset none bind,optional,create=file
    • lxc.mount.entry: /dev/nvidia-uvm-tools dev/nvidia-uvm-tools none bind,optional,create=file
    • lxc.mount.entry: /dev/nvidia-caps/nvidia-cap1 dev/nvidia-caps/nvidia-cap1 none bind,optional,create=file
    • lxc.mount.entry: /dev/nvidia-caps/nvidia-cap2 dev/nvidia-caps/nvidia-cap2 none bind,optional,create=file
  5. Take a snapshot!! "gpu-mounted" or something like that.
  6. Start the LXC. Install the proprietary drivers as described in the Jellyfin instructions here: https://jellyfin.org/docs/general/administration/hardware-acceleration/nvidia#linux-setups
    • https://wiki.debian.org/NvidiaGraphicsDrivers and go to the LXC distribution. I'm using debian 12
    • Add "contrib", "non-free" and "non-free-firmware" components to
      Code:
      /etc/apt/sources.list
    • Code:
      apt update
    • Code:
      apt install firmware-misc-nonfree
      NOTE: I removed nvidia-driver from the instructions. We will let the next step take care of this.
    • Back to the jellyfin instructions https://jellyfin.org/docs/general/administration/hardware-acceleration/nvidia#linux-setups
    • Code:
      apt install -y jellyfin-ffmpeg5
      (Should already be installed)
    • Code:
      apt install -y libnvcuvid1 libnvidia-encode1
      This is where you'll see drivers installed, take note of the version. Mine was 525.147.05-4.
  7. Now roll back to the "gpu-mounted" snapshot once you noted the correct driver version.
  8. On the host, uninstall the drivers from step one using the same command you used to install them, but add the
    Code:
    --uninstall
    flag.
  9. Download the correct drivers from the site in step one, and install them. Make sure the GPU registers when you run
    Code:
    nvidia-smi
  10. On the LXC, repeat step 6 again.
  11. Reboot the LXC
  12. You'll be able to run
    Code:
    nvidia-smi
    from the LXC, though you won't see any processes (these only show up on the host).

At this point you should be able to transcode something using the GPU. Set up Nvidia transcoding in the Jellyfin settings, start a movie, and change the resolution to something in the 480 range just force a transcode. Now, on the host, run
Code:
nvidia-smi
and you should see the transcoding process listed. It'll have "jellyfin-ffmpeg" in the process name. Open up a second window and start another movie simultaneously, drop the resolution, and run
Code:
nvidia-smi
on the host. You should now see two processes indicating the second transcode is working.

The final step: Unlimited encoding streams

Nvidia has a cap on their consumer cards. This can be easily removed using this repo: https://github.com/keylase/nvidia-patch

This needs to be done on both the host and the LXC

  1. Go to
    Code:
    /opt/nvidia
    which might already exist
  2. Code:
    wget https://raw.githubusercontent.com/keylase/nvidia-patch/master/patch.sh
  3. Code:
    bash ./patch.sh

Now you should be able to open many instances of jellyfin and have them all transcoding at the same time. Running nvidia-smi on the host will show you how many transcoding streams are actually on the GPU. If you can get to 6 you know you applied the patch correctly.

Good luck!

RE: Proxmox LXC with Nvidia Transcoding and Network Share - jellyshield - 2024-01-11

It didn't work for me.

You can see similar issues with others : https://forum.jellyfin.org/t-hardware-encoding-broken-after-updating-proxmox?pid=13315#pid13315


For me it used to work with just installing the latest driver from NVIDIA in both host and lxc. I didn't even had to install libnvcuvid1 libnvidia-encode1 separately. After some kernel update jellyfin-ffmpeg5/6 is broken. I did what was mentioned and downgraded the driver to match the Debian version. I was also able to install the  libnvcuvid1 libnvidia-encode1 in lxc and successfully ran nvidia-smi in both host and lxc. Still I have the same error.

Both the libraries are installed

Code:
root@jellyfin:~# apt install -y libnvcuvid1 libnvidia-encode1
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
libnvcuvid1 is already the newest version (525.147.05-4~deb12u1).
libnvidia-encode1 is already the newest version (525.147.05-4~deb12u1).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

NVIDIA-SMI

Code:
root@jellyfin:~# nvidia-smi
Thu Jan 11 00:43:19 2024
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.147.05  Driver Version: 525.147.05  CUDA Version: 12.0    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|        Memory-Usage | GPU-Util  Compute M. |
|                              |                      |              MIG M. |
|===============================+======================+======================|
|  0  Quadro M2000        On  | 00000000:21:00.0 Off |                  N/A |
| 56%  28C    P8    6W /  75W |      1MiB /  4096MiB |      0%      Default |
|                              |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU  GI  CI        PID  Type  Process name                  GPU Memory |
|        ID  ID                                                  Usage      |
|=============================================================================|
|  No running processes found                                                |
+-----------------------------------------------------------------------------+

Still get the same error in Jellyfin during transcoding

Code:
ffmpeg version 5.1.4-Jellyfin Copyright (c) 2000-2023 the FFmpeg developers
  built with gcc 12 (Debian 12.2.0-14)
  configuration: --prefix=/usr/lib/jellyfin-ffmpeg --target-os=linux --extra-libs=-lfftw3f --extra-version=Jellyfin --disable-doc --disable-ffplay --disable-ptx-compression --disable-static --disable-libxcb --disable-sdl2 --disable-xlib --enable-lto --enable-gpl --enable-version3 --enable-shared --enable-gmp --enable-gnutls --enable-chromaprint --enable-libdrm --enable-libass --enable-libfreetype --enable-libfribidi --enable-libfontconfig --enable-libbluray --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libopenmpt --enable-libdav1d --enable-libwebp --enable-libvpx --enable-libx264 --enable-libx265 --enable-libzvbi --enable-libzimg --enable-libfdk-aac --arch=amd64 --enable-libsvtav1 --enable-libshaderc --enable-libplacebo --enable-vulkan --enable-opencl --enable-vaapi --enable-amf --enable-libmfx --enable-ffnvcodec --enable-cuda --enable-cuda-llvm --enable-cuvid --enable-nvdec --enable-nvenc
  libavutil      57. 28.100 / 57. 28.100
  libavcodec    59. 37.100 / 59. 37.100
  libavformat    59. 27.100 / 59. 27.100
  libavdevice    59.  7.100 / 59.  7.100
  libavfilter    8. 44.100 /  8. 44.100
  libswscale      6.  7.100 /  6.  7.100
  libswresample  4.  7.100 /  4.  7.100
  libpostproc    56.  6.100 / 56.  6.100
[AVHWDeviceContext @ 0x562bf18b8d00] cu->cuInit(0) failed -> CUDA_ERROR_UNKNOWN: unknown error
Device creation failed: -542398533.
Failed to set value 'cuda=cu:0' for option 'init_hw_device': Generic error in an external library
Error parsing global options: Generic error in an external library