SOLVED: H.265 Media not playing with HWA

[KipshidzeNikoloz]

Greetings all,

I've come up on a peculiar problem. When using HWA (Nvidia NVENC), media that is h.265 does not play on any client. Disabling the hardware acceleration plays the video, but with obvious stuttering.

The media plays when using VLC, the disk is mounted (visible in docker and other files play normally). The issue popped up very strangely. I was watching one episode of a series, when I started the second episode the error "This client isn't compatible with the media and the server isn't sending a compatible media format" popped up, and from that point on I couldn't play any h.265 (HEVC) files.

I'm attaching the latest FFmpeg and server logs (In a zip file because I cannot attach files in another way for some reason), here's the pastebin link for the ffmpeg log https://pastebin.com/jw791asG. Particularly interesting is the ffmpeg log that displays the following error.

[AVHWDeviceContext @ 0x55c6c6219640] Cannot load libcuda.so.1
[AVHWDeviceContext @ 0x55c6c6219640] Could not dynamically load CUDA
Device creation failed: -1.
Failed to set value 'cuda=cu:0' for option 'init_hw_device': Operation not permitted
Error parsing global options: Operation not permitted

I'm using the windows docker version of jellyfin, with the latest NVIDIA drivers, I've updated ffmpeg in the docker container to ffmpeg6 with no avail.

To solve the issue I've tried to add the following environment variable to the docker-compose file, even though NVIDIA drivers should be accessible without them (and could access them before the sudden error appeared)

    environment:
      - NVIDIA_DRIVER_CAPABILITIES=all
      - NVIDIA_VISIBLE_DEVICES=all

I've also checked the Audio configuration in Jellyfin, changed it to 5.1 stereo, Stereo and Mono, but none of those worked (some others have had issues with 7.1 stereo playback). No amount of restarting, recreating the container, redownloading the image has worked so far.

Some extra information that might help out: The jellyfin docker image seems to have some issues for me. After being on for a long time, it seems to hang, stuck when completing scheduled tasks, not updating the library. Restarting the container doesn't work, and If i forcefully turn it off and on the following error pops up

"Bind for 0.0.0.0:8920 failed: port is already allocated"

the only solution for this that i've found is forcefully shutting down docker through

wsl --shutdown

which restarts docker and the container, after which jellyfin works fine for some time. The same issue does not affect other containers.

Any suggestions or ideas?

UPDATE: For everyone that might have struggled with this like me.
It seems somewhere along the line, the Jellyfin configuration lost access to the NVIDIA GPU's, while docker was able to see it, and by all means jellyfin should be able to access it by default, for some reason it just absolutely refused to. I came across this wonderful answer on stackoverflow https://stackoverflow.com/questions/7076...all-option
If like me you have encountered this issue, and if like me you have gone down insane rabbitholes, and if like me you use the standard docker-compose file recommended by jellyfin, you simply need to add the following code to the compose file for Jellyfin to correctly detect and access your gpu. (Change the count variable to all "count: all" to get access to all gpu's if you're running and using more than one)

    
deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

Jellyfin can now properly transcode all compatible files.