Troubleshooting HWA

[scrooge]

Hey all, recently started having some issues with Jellyfin's Hardware Acceleration. It's a bit of a doozie but I'll try to keep it succinct.

Specs

• CPU: i7 6700k
  
• RAM: 16 GB Corsair Vengeance 3200 MHz
  
• GPU: Nvidia GTX 1660 (Driver Version: 535.154.05)
  

Setup

• Host OS: Proxmox 8.1.0 (running kernel: 6.5.11-8-pve)
  
• LXC Container Running Fedora 38 with GPU passed through (drivers all work, can run nvidia-smi to verify)
  
• Jellyfin installed from rpmfusion repository version 10.8.13-1
  

Problem
Recently I tried to watch a 4K HDR10 Remux on my 1440p laptop and was getting terrible stuttering while watching even though I had Hardware Acceleration setup and was hard-wired to my home network. I checked the playback info in the browser player and saw that I was transcoding a reported 33 FPS, which should be enough I thought. Then I checked nvidia-smi and sure enough a process was running on my Nvidia GTX 1660 GPU for the transcoding. However when I checked top on the LXC container running jellyfin I saw that an ffmpeg process was taking up 380% cpu, which confused me because I would've thought that it was handed off to the GPU. I tried some tinkering but I am no expert in this stuff by any means, so I decided the next step would be to update the container's packages, the proxmox host's packages, and the nvidia driver.

After doing all that and getting things setup again I noticed that my transcoded FPS was even lower, and after checking the logs (see below) I saw that I was no longer using the nvenc encoder/decoder. I looked at nvidia-smi and there were no processes running on the GPU, but checking top I saw the same ffmpeg process using 380% CPU. My next step was going into the transcoding logs and pulling an old command that utilized hevc_nvenc and tried using that command by hand to see if anything happened and got the error Unknown encoder 'hevc_nvenc'. After some googling on that error I found a stackoverflow talking about compiling jellyfin-ffmpeg by hand, so I started down that rabbit hole, but I figured it'd be best to reach out before going headlong into that. I just have some reservations because I didn't have to do that to get it working with previous version of Jellyfin (10.8.11) before I updated versions which makes me think that that's not going to fix my issue (not to mention that in this newest version I can't update the ffmpeg path in the playback settings).

So all in all I really need some guidance about this whole setup. Clearly I've never really had a super great setup for transcoding since it couldn't handle one 4K transcode when there were no other jobs or streams to transcode besides mine. Any help would be greatly appreciated

Here's the important log messages I saw, I can provide the whole files if need be, I just don't want to clutter up the post.
A transcode log from 2-12-24 before I updated everything utilizing nvenc:

Code:

ffmpeg -analyzeduration 200M  -i file:"/jellyfin/TV/Show/S01E01.mkv" -map_metadata -1 -map_chapters -1 -threads 0 -map 0:0 -map 0:1 -map -0:s -codec:v:0 hevc_nvenc -tag:v:0 hvc1 -preset p1 -b:v 10000000 -maxrate 10000000 -bufsize 20000000 -g:v:0 72 -keyint_min:v:0 72 -vf "setparams=color_primaries=bt709:color_trc=bt709:colorspace=bt709,scale=trunc(min(max(iw\,ih*a)\,min(1920\,1080*a))/2)*2:trunc(min(max(iw/a\,ih)\,min(1920/a\,1080))/2)*2,format=yuv420p" -codec:a:0 aac -ac 2 -ab 384000 -af "volume=2" -copyts -avoid_negative_ts disabled -max_muxing_queue_size 2048 -f hls -max_delay 5000000 -hls_time 3 -hls_segment_type mpegts -start_number 0 -hls_segment_filename "/var/lib/jellyfin/transcodes/b3d7fe243463d053981ca1e1d06633b7%d.ts" -hls_playlist_type vod -hls_list_size 0 -y "/var/lib/jellyfin/transcodes/b3d7fe243463d053981ca1e1d06633b7.m3u8"

A transcode log from today 2-14-24 after updating no nvenc:

Code:

ffmpeg -analyzeduration 200M  -i file:"/jellyfin/Movies/4KMovieHDR10Remux.mkv" -map_metadata -1 -map_chapters -1 -threads 0 -map 0:0 -map 0:1 -map -0:s -codec:v:0 libx264 -preset veryfast -crf 24 -maxrate 108752778 -bufsize 217505556 -x264opts:0 subme=0:me_range=4:rc_lookahead=10:me=dia:no_chroma_me:8x8dct=0:partitions=none -force_key_frames:0 "expr:gte(t,0+n_forced*3)" -sc_threshold:v:0 0 -vf "setparams=color_primaries=bt709:color_trc=bt709:colorspace=bt709,scale=trunc(min(max(iw\,ih*a),min(3840\,2160*a))/2)*2:trunc(min(max(iw/a\,ih)\,min(3840/a\,2160))/2)*2,format=yuv420p" -codec:a:0 aac -ac 2 -ab 384000 -af "volume=2" -copyts -avoid_negative_ts disabled -max_muxing_queue_size 2048 -f hls -max_delay 5000000 -hls_time 3 -hls_segment_type mpegts -start_number 0 -hls_segment_filename "/var/lib/jellyfin/transcodes/c9bbd87bf7827591706f71fce95fa196%d.ts" -hls_playlist_type vod -hls_list_size 0 -y "/var/lib/jellyfin/transcodes/c9bbd87bf7827591706f71fce95fa196.m3u8[i]"[/i]

**UPDATE**
After much more googling and troubleshooting I have resolved the issue. But I would like to leave this up in case anyone stumbles onto similar issues.
Here is everything I did that brought me to the path of redemption:

1. Pull down the jellyfin-ffmpeg binaries from their github release here: https://github.com/jellyfin/jellyfin-ffmpeg/releases
   
2. Extract the binaries and either update jellyfin to point at it or replace the old ffmpeg/ffprobe binaries (I did this as a bandaid, don't recommend) then restart jellyfin
   
3. This fixed the whole "unknown encoder" issue. However I started getting a weird (what I thought was a) cuda error
   
   Code:

   [AVHWDeviceContext @ 0x55bbf927f780] cu->cuInit(0) failed -> CUDA_ERROR_UNKNOWN: 
unknown errorDevice creation failed: -542398533.Failed to set value 'cuda=cu:0' for option 'init_hw_device': Generic error in an external libraryError parsing global options:Generic error in an external library
4. Upon more digging I stumbled upon this post and I had totally forgotten to check the group IDs after doing a full kernel update, turns out two of them were off. Updated the ids in /etc/pve/lxc/<id>.conf and just like that everything was smooth sailing.
   

I just wanted to say that while I didn't get my solution from a response here, there was so much good help on this forum and in github issues for leading to the right answer. It took awhile to get it all straight, but I just want to say that it's very well done and kudos to all who contribute.

[TheDreadPirate]

This has nothing to do with ffmpeg/jellyfin-ffmpeg. Check that all the nvidia packages were updated. Nvidia container toolkit, etc.