SOLVED: Transcoding issues -- SOLVED

[ThexTallxDude]

I recently updated my Debian host machine, and NVIDIA transcoding within the Docker container no longer works.

Whenever a video needs to be transcoded, the ffmpeg log displays this error:

Code:

[mpeg4_cuvid @ 0x561fcc395a40] Cannot load libnvcuvid.so.1
[mpeg4_cuvid @ 0x561fcc395a40] Failed loading nvcuvid.
Stream mapping:
  Stream #0:0 -> #0:0 (mpeg4 (mpeg4_cuvid) -> h264 (h264_nvenc))
  Stream #0:1 -> #0:1 (ac3 (native) -> aac (libfdk_aac))
Error while opening decoder for input stream #0:0 : Operation not permitted

Running NVIDIA-SMI within the container works well, and 

Code:

/usr/lib/jellyfin-ffmpeg/ffmpeg -v debug -init_hw_device cuda

returns without any noticeable problems.
I am on Jellyfin version 10.8.13
The NVIDIA stuff: Device:Quadro P2000, Driver Version: 545.23.08, CUDA Version: 12.3


This is my first post, so please let me know what I forgot to add

[tmsrxzar]

oh joy, more nvidia+linux issues
(edit: sorry, perhaps unfair statement, only reflects my personal experience with nvidia while on linux)

since it can't load libraries then i'm guessing not everything was installed as per directions

see https://docs.linuxserver.io/images/docke...in/#nvidia

Hardware acceleration users for Nvidia will need to install the container runtime provided by Nvidia on their host, instructions can be found here:

https://github.com/NVIDIA/nvidia-docker

We automatically add the necessary environment variable that will utilise all the features available on a GPU on the host. Once nvidia-docker is installed on your host you will need to re/create the docker container with the nvidia container runtime --runtime=nvidia and add an environment variable -e NVIDIA_VISIBLE_DEVICES=all (can also be set to a specific gpu's UUID, this can be discovered by running nvidia-smi --query-gpu=gpu_name,gpu_uuid --format=csv ). NVIDIA automatically mounts the GPU and drivers from your host into the jellyfin docker container.

[ThexTallxDude]

I did all that on initial setup. It was transcoding without issue for over a year. This happened only after updating the host computer, a system running OpenMediaVault6, to version 6.9.12-1. The Nvidia drivers seem to have updated during that process though, so I'm _guessing_ there may be a driver mismatch between the host and container. I deleted and grabbed a fresh jellyfin image the other day so it'd be forced to pull new everything, but the issue persists. The driver version/hw info from the nvidia-smi matches between the host and container. I may still be missing something, but hopefully the additional context helps.
Edit: here's the jellyfin part of my stack:

Code:

version: "2.1"
services:
  jellyfin:
    image: lscr.io/linuxserver/jellyfin:latest
    container_name: Jellyfin
    runtime: nvidia
    privileged: true
  # deploy:
  #  resources:
  #    reservations:
  #      devices:
  #        - driver: nvidia
  #          count: all
  #          capabilities: [gpu]
    volumes:
      - /srv/dev-disk-by-uuid-6ab4066e-7303-4419-a71d-0207cf488a15/NetworkFiles/Config/JellyFin:/config
      - /Media/Jelly/Movies:/Data/Movies
      - /Media/Jelly/Shows:/Data/Shows
      - /Media/NetworkMedia/Audio:/Data/Audio
    environment:
      - PUID=998
      - PGID=100
      - TZ=America/New_York
      - NVIDIA_VISIBLE_DEVICES=all
      - NVIDIA_DRIVER_CAPABILITIES=all
     
    ports:
      - 8001:8096
    devices:
      - /dev/dri:/dev/dri
    restart: unless-stopped

[tmsrxzar]

see if this link helps https://github.com/NVIDIA/nvidia-docker/issues/1001

[ThexTallxDude]

Apologies, I'm not suuuuuuper familiar with any of this, unfortunately.
running ldconfig on the host machine gets me the following.
I'm interpreting that as the library/module/component or whatever is not there at all?

Code:

ldconfig: File /lib/x86_64-linux-gnu/libcuda.so.470.161.03 is empty, not checked.
ldconfig: File /lib/x86_64-linux-gnu/libnvidia-ml.so.470.161.03 is empty, not checked.
ldconfig: File /lib/x86_64-linux-gnu/libnvidia-ptxjitcompiler.so.470.161.03 is empty, not checked.

[tmsrxzar]

correct, "is empty, not checked." would mean they are not there "at all"

[ThexTallxDude]

Ended up just nuking and reinstalling the cuda driver and nvidia container stuff.
I had both the new toolkit and the old nvidia-docker stuff so cleaned it up in case it was part of the issue. Looks like we're set now, appreciate it.