Jellyfin Forum
NVENC not working - Printable Version

+- Jellyfin Forum (https://forum.jellyfin.org)
+-- Forum: Support (https://forum.jellyfin.org/f-support)
+--- Forum: Troubleshooting (https://forum.jellyfin.org/f-troubleshooting)
+--- Thread: NVENC not working (/t-nvenc-not-working)

NVENC not working - friki67 - 2023-08-08

Computer: Intel Xeon 2670v3, 32GB ECC RAM, Nvidia GTX 750 (GM107, Maxwell 2nd gen) (tried a GTX 1070 too, same result)
OS: Ubuntu server 22.04 + Xfce as GUI, Nvidia driver 535, nvidia-smi returning all ok.
Jellyfin 10.8.10 is installed using the recommended method (curl the installation script) and works fine, except for the hw encode/decode part.

I've checked user groups and all the suggestions I've found but I'm failing in make it work.

This was working on my old installation. I needed to reinstall and I've been unable to get it working again. I'm suffering this in a Fedora installation too.

Please check https://developer.nvidia.com/video-encode-and-decode-gpu-support-matrix-new
   

I've tried lot of configurations, the minimal is 
   

But I always get "Playback error: the client isn't compatible with the media and the server isn't sending a compatible media format"

So what am I missing?
PS: all files played ok in VLC, all H.265, one of them info:
Code:
Codec: MPEG-H Part 2/HEVC (H.265)(hevc)
Video Resolution: 3832x1632
Frame rate: 29.970628
Decoded format: Planar 4.2.0 YUV full scale

EDIT: ffmpeg log for this file
Quote:ffmpeg version 5.1.3-Jellyfin Copyright © 2000-2022 the FFmpeg developers
  built with gcc 11 (Ubuntu 11.4.0-1ubuntu1~22.04)
  configuration: --prefix=/usr/lib/jellyfin-ffmpeg --target-os=linux --extra-libs=-lfftw3f --extra-version=Jellyfin --disable-doc --disable-ffplay --disable-ptx-compression --disable-static --disable-libxcb --disable-sdl2 --disable-xlib --enable-lto --enable-gpl --enable-version3 --enable-shared --enable-gmp --enable-gnutls --enable-chromaprint --enable-libdrm --enable-libass --enable-libfreetype --enable-libfribidi --enable-libfontconfig --enable-libbluray --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libopenmpt --enable-libdav1d --enable-libwebp --enable-libvpx --enable-libx264 --enable-libx265 --enable-libzvbi --enable-libzimg --enable-libfdk-aac --arch=amd64 --enable-libsvtav1 --enable-libshaderc --enable-libplacebo --enable-vulkan --enable-opencl --enable-vaapi --enable-amf --enable-libmfx --enable-ffnvcodec --enable-cuda --enable-cuda-llvm --enable-cuvid --enable-nvdec --enable-nvenc
  libavutil      57. 28.100 / 57. 28.100
  libavcodec    59. 37.100 / 59. 37.100
  libavformat    59. 27.100 / 59. 27.100
  libavdevice    59.  7.100 / 59.  7.100
  libavfilter    8. 44.100 /  8. 44.100
  libswscale      6.  7.100 /  6.  7.100
  libswresample  4.  7.100 /  4.  7.100
  libpostproc    56.  6.100 / 56.  6.100
Input #0, matroska,webm, from 'fileConfused-facemnt/media/Star_Citizen_Hornet_Commercial_Launch(1).mkv':
  Metadata:
    encoder        : libDivXMediaFormat 4.0.0.0578
  Duration: 00:02:57.98, start: 0.000000, bitrate: 28726 kb/s
  Stream #0:0(eng): Video: hevc (Main), yuvj420p(pc), 3832x1632 [SAR 1:1 DAR 479:204], 29.97 fps, 29.97 tbr, 1k tbn (default)
  Stream #0:1(en): Audio: aac (LC), 44100 Hz, stereo, fltp (default)
Stream mapping:
  Stream #0:0 -> #0:0 (hevc (native) -> h264 (h264_nvenc))
  Stream #0:1 -> #0:1 (aac (native) -> aac (libfdk_aac))
Press [q] to stop, [?] for help
[hevc @ 0x558ef04ef180] Hardware is lacking required capabilities
[hevc @ 0x558ef04ef180] Failed setup for format cuda: hwaccel initialisation returned error.
Impossible to convert between the formats supported by the filter 'Parsed_setparams_0' and the filter 'auto_scale_0'
Error reinitializing filters!
Failed to inject frame into filter network: Function not implemented
Error while processing the decoded data for stream #0:0
Conversion failed

EDIT2: nvidia-smi
Tue Aug  8 11:07:50 2023     

+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.86.05              Driver Version: 535.86.05    CUDA Version: 12.2    |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf          Pwr:Usage/Cap |        Memory-Usage | GPU-Util  Compute M. |
|                                        |                      |              MIG M. |
|=========================================+======================+======================|
|  0  NVIDIA GeForce GTX 750        Off | 00000000:03:00.0  On |                  N/A |
| 24%  35C    P8              1W /  38W |    240MiB /  1024MiB |    10%      Default |
|                                        |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
                                                                                       
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU  GI  CI        PID  Type  Process name                            GPU Memory |
|        ID  ID                                                            Usage      |
|=======================================================================================|
|    0  N/A  N/A      1346      G  /usr/lib/xorg/Xorg                          218MiB |
|    0  N/A  N/A      2398      G  xfwm4                                        0MiB |
|    0  N/A  N/A    20649      G  /snap/vlc/3078/usr/bin/vlc                  13MiB |
+---------------------------------------------------------------------------------------+

This client isn't compatible with the media and the server isn't sending a compatible media format.Playback Error
This client isn't compatible with the media and the server isn't sending a compatible media format.

RE: NVENC not working - nyanmisaka - 2023-08-08

This error is reported to FFmpeg by the NVDEC driver. I have to suspect it's a regression. Please try downgrading the driver to an older version.

https://github.com/FFmpeg/FFmpeg/blob/b98ee1a355e45d617e2b2a19722f74b4fe724ed3/libavcodec/nvdec.c#L138-L141

Code:
    if (!caps.bIsSupported) {
        av_log(logctx, AV_LOG_ERROR, "Hardware is lacking required capabilities\n");
        return AVERROR(EINVAL);
    }

Running this command can expose more NVDEC capabilities for easy debugging.

Code:
/usr/lib/jellyfin-ffmpeg/ffmpeg -v verbose -init_hw_device cuda=cu:0 -hwaccel cuda -hwaccel_output_format cuda -i /path/to/the/media -an -sn -vframes 1 -f null -


RE: NVENC not working - friki67 - 2023-08-09

I'm attaching two files, one with the output of the ffmpeg command using the 535 driver and one using the 525 driver (EDIT: added 515 output too), but they look to me the same.

The nvidia-smi result for the 525 driver:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.125.06  Driver Version: 525.125.06  CUDA Version: 12.0    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|        Memory-Usage | GPU-Util  Compute M. |
|                              |                      |              MIG M. |
|===============================+======================+======================|
|  0  NVIDIA GeForce ...  Off  | 00000000:03:00.0  On |                  N/A |
| 23%  33C    P8    1W /  38W |    117MiB /  1024MiB |      0%      Default |
|                              |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                             
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU  GI  CI        PID  Type  Process name                  GPU Memory |
|        ID  ID                                                  Usage      |
|=============================================================================|
|    0  N/A  N/A      1439      G  /usr/lib/xorg/Xorg                112MiB |
|    0  N/A  N/A      2666      G  xfwm4                              0MiB |
+-----------------------------------------------------------------------------+

I know that the CUDA version 12 is causing issues in other softwares.... maybe something related to this? 

So I'm going with 515 driver (CUDA 11.7) but I'm getting same result!

+-----------------------------------------------------------------------------+

| NVIDIA-SMI 515.86.01    Driver Version: 515.86.01    CUDA Version: 11.7    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|        Memory-Usage | GPU-Util  Compute M. |
|                              |                      |              MIG M. |
|===============================+======================+======================|
|  0  NVIDIA GeForce ...  Off  | 00000000:03:00.0  On |                  N/A |
| 23%  33C    P8    1W /  38W |    100MiB /  1024MiB |      0%      Default |
|                              |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                             
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU  GI  CI        PID  Type  Process name                  GPU Memory |
|        ID  ID                                                  Usage      |
|=============================================================================|
|    0  N/A  N/A      1397      G  /usr/lib/xorg/Xorg                95MiB |
|    0  N/A  N/A      2621      G  xfwm4                              0MiB |
+-----------------------------------------------------------------------------+

Which driver should I install to test this in Ubuntu Server? This is a test computer, so I don't mind "play" with it

EDIT: installed OBS Studio (snap one), set recording output codec to Hardware-NVENC, and it seems to work ok (h.264). The thing is that H.265 decoding/encoding was working with this card (this is GTX750, and with GTX1070) in my old installation, in Jellyfin. What is happening? I'm sure I'm missing something.

RE: NVENC not working - nyanmisaka - 2023-08-09

I checked your GPU name GM107. It's 1st Gen Maxwell, so that makes sense. You may have different problems with GTX1070.

GTX750/GM107 https://www.techpowerup.com/gpu-specs/nvidia-gm107.g762

GTX750/GM206 https://www.techpowerup.com/gpu-specs/nvidia-gm206.g775

RE: NVENC not working - friki67 - 2023-08-10

(2023-08-09, 01:03 PM)nyanmisaka Wrote: I checked your GPU name GM107. It's 1st Gen Maxwell, so that makes sense. You may have different problems with GTX1070.

GTX750/GM107 https://www.techpowerup.com/gpu-specs/nvidia-gm107.g762

GTX750/GM206 https://www.techpowerup.com/gpu-specs/nvidia-gm206.g775

Hello again. First, thank you for your time and knowledge.

So I went with my other computer.
Quote:# lshw -C video
*-display
description: VGA compatible controller
product: GP104 [GeForce GTX 1070]
vendor: NVIDIA Corporation
physical id: 0
bus info: pci@0000:03:00.0
version: a1
width: 64 bits
clock: 33MHz
capabilities: vga_controller bus_master cap_list rom
configuration: driver=nvidia latency=0
resources: iomemory:381f0-381ef iomemory:381f0-381ef irq:74 memory:fa000000-faffffff memory:381fe0000000-381fefffffff memory:381ff0000000-381ff1ffffff ioport:d000(size=12Nerd-face memory:c0000-dffff
*-graphics
product: EFI VGA
physical id: 2
logical name: fb0
capabilities: fb
configuration: depth=32 resolution=800,600

Then I did the ffmpeg test and got a positive result:

Quote:~# /usr/lib/jellyfin-ffmpeg/ffmpeg -v verbose -init_hw_device cuda=cu:0 -hwaccel cuda -hwaccel_output_format cuda -i '/mnt/storage/data/downloads/media/Panasonic Lumix DMC-GH4
-4K-x265.mp4' -an -sn -vframes 1 -f null -
ffmpeg version 5.1.3-Jellyfin Copyright © 2000-2022 the FFmpeg developers
built with gcc 11 (Ubuntu 11.4.0-1ubuntu1~22.04)
configuration: --prefix=/usr/lib/jellyfin-ffmpeg --target-os=linux --extra-libs=-lfftw3f --extra-version=Jellyfin --disable-doc --disable-ffplay --disable-ptx-compression --disable-static
--disable-libxcb --disable-sdl2 --disable-xlib --enable-lto --enable-gpl --enable-version3 --enable-shared --enable-gmp --enable-gnutls --enable-chromaprint --enable-libdrm --enable-libass
--enable-libfreetype --enable-libfribidi --enable-libfontconfig --enable-libbluray --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libopenmpt --enable-l
ibdav1d --enable-libwebp --enable-libvpx --enable-libx264 --enable-libx265 --enable-libzvbi --enable-libzimg --enable-libfdk-aac --arch=amd64 --enable-libsvtav1 --enable-libshaderc --enable
-libplacebo --enable-vulkan --enable-opencl --enable-vaapi --enable-amf --enable-libmfx --enable-ffnvcodec --enable-cuda --enable-cuda-llvm --enable-cuvid --enable-nvdec --enable-nvenc
libavutil      57. 28.100 / 57. 28.100
libavcodec     59. 37.100 / 59. 37.100
libavformat    59. 27.100 / 59. 27.100
libavdevice    59.  7.100 / 59.  7.100
libavfilter     8. 44.100 /  8. 44.100
libswscale      6.  7.100 /  6.  7.100
libswresample   4.  7.100 /  4.  7.100
libpostproc    56.  6.100 / 56.  6.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '/mnt/storage/data/downloads/media/Panasonic Lumix DMC-GH4-4K-x265.mp4':
Metadata:
major_brand     : hvc1
minor_version   : 0
compatible_brands: iso4hvc1
creation_time   : 2014-02-07T23:58:49.000000Z
encoder         : Hybrid 2014.01.22.1
Duration: 00:03:07.32, start: 0.000000, bitrate: 8636 kb/s
Stream #0:0[0x1](und): Video: hevc (Main), 1 reference frame (hvc1 / 0x3163766Nerd-face, yuv420p(tv, left), 3840x2160, 8509 kb/s, 29.97 fps, 29.97 tbr, 30k tbn (default)
Metadata:
creation_time   : 2014-02-07T23:58:49.000000Z
handler_name    : 265#video:fps=29.97@GPAC0.5.1-DEV-rev4992
vendor_id       : [0][0][0][0]
Stream #0:1[0x2](und): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 125 kb/s (default)
Metadata:
creation_time   : 2014-02-07T23:58:50.000000Z
handler_name    : AAC#audio:lang=@GPAC0.5.1-DEV-rev4992
vendor_id       : [0][0][0][0]
Stream mapping:
Stream #0:0 -> #0:0 (hevc (native) -> wrapped_avframe (native))
Press [q] to stop, [?] for help
[hevc @ 0x5610ff126ec0] NVDEC capabilities:
[hevc @ 0x5610ff126ec0] format supported: yes, max_mb_count: 262144
[hevc @ 0x5610ff126ec0] min_width: 144, max_width: 8192
[hevc @ 0x5610ff126ec0] min_height: 144, max_height: 8192
[graph 0 input from stream 0:0 @ 0x561104427c40] w:3840 h:2160 pixfmt:cuda tb:1/30000 fr:30000/1001 sar:0/1
Output #0, null, to 'pipe:':
Metadata:
major_brand     : hvc1
minor_version   : 0
compatible_brands: iso4hvc1
encoder         : Lavf59.27.100
Stream #0:0(und): Video: wrapped_avframe, 1 reference frame, cuda(tv, progressive, left), 3840x2160 (0x0), q=2-31, 200 kb/s, 29.97 fps, 29.97 tbn (default)
Metadata:
creation_time   : 2014-02-07T23:58:49.000000Z
handler_name    : 265#video:fps=29.97@GPAC0.5.1-DEV-rev4992
vendor_id       : [0][0][0][0]
encoder         : Lavc59.37.100 wrapped_avframe
No more output streams to write to, finishing.:00.10 bitrate=N/A speed=0.939x
frame=    1 fps=0.0 q=-0.0 Lsize=N/A time=00:00:00.10 bitrate=N/A speed=0.883x
video:0kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
Input file #0 (/mnt/storage/data/downloads/media/Panasonic Lumix DMC-GH4-4K-x265.mp4):
Input stream #0:0 (video): 18 packets read (3937 bytes); 2 frames decoded;
Input stream #0:1 (audio): 1 packets read (364 bytes);
Total: 19 packets (4301 bytes) demuxed
Output file #0 (pipeSmiling-face:
Output stream #0:0 (video): 1 frames encoded; 1 packets muxed (472 bytes);
Total: 1 packets (472 bytes) muxed
[AVIOContext @ 0x5610ff11e1c0] Statistics: 163899 bytes read, 2 seeks

But, when trying to play using the web interface in firefox, it fails

Quote:ffmpeg version 5.1.3-Jellyfin Copyright © 2000-2022 the FFmpeg developers
  built with gcc 11 (Ubuntu 11.4.0-1ubuntu1~22.04)
  configuration: --prefix=/usr/lib/jellyfin-ffmpeg --target-os=linux --extra-libs=-lfftw3f --extra-version=Jellyfin --disable-doc --disable-ffplay --disable-ptx-compression --disable-static --disable-libxcb --disable-sdl2 --disable-xlib --enable-lto --enable-gpl --enable-version3 --enable-shared --enable-gmp --enable-gnutls --enable-chromaprint --enable-libdrm --enable-libass --enable-libfreetype --enable-libfribidi --enable-libfontconfig --enable-libbluray --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libopenmpt --enable-libdav1d --enable-libwebp --enable-libvpx --enable-libx264 --enable-libx265 --enable-libzvbi --enable-libzimg --enable-libfdk-aac --arch=amd64 --enable-libsvtav1 --enable-libshaderc --enable-libplacebo --enable-vulkan --enable-opencl --enable-vaapi --enable-amf --enable-libmfx --enable-ffnvcodec --enable-cuda --enable-cuda-llvm --enable-cuvid --enable-nvdec --enable-nvenc
  libavutil      57. 28.100 / 57. 28.100
  libavcodec    59. 37.100 / 59. 37.100
  libavformat    59. 27.100 / 59. 27.100
  libavdevice    59.  7.100 / 59.  7.100
  libavfilter    8. 44.100 /  8. 44.100
  libswscale      6.  7.100 /  6.  7.100
  libswresample  4.  7.100 /  4.  7.100
  libpostproc    56.  6.100 / 56.  6.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'fileConfused-facedata/downloads/media/Panasonic Lumix DMC-GH4-4K-x265.mp4':
  Metadata:
    major_brand    : hvc1
    minor_version  : 0
    compatible_brands: iso4hvc1
    creation_time  : 2014-02-07T23:58:49.000000Z
    encoder        : Hybrid 2014.01.22.1
  Duration: 00:03:07.32, start: 0.000000, bitrate: 8636 kb/s
  Stream #0:0[0x1](und): Video: hevc (Main) (hvc1 / 0x3163766Nerd-face, yuv420p(tv), 3840x2160, 8509 kb/s, 29.97 fps, 29.97 tbr, 30k tbn (default)
    Metadata:
      creation_time  : 2014-02-07T23:58:49.000000Z
      handler_name    : 265#video:fps=29.97@GPAC0.5.1-DEV-rev4992
      vendor_id      : [0][0][0][0]
  Stream #0:1[0x2](und): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 125 kb/s (default)
    Metadata:
      creation_time  : 2014-02-07T23:58:50.000000Z
      handler_name    : AAC#audio:lang=@GPAC0.5.1-DEV-rev4992
      vendor_id      : [0][0][0][0]
Stream mapping:
  Stream #0:0 -> #0:0 (hevc (native) -> h264 (h264_nvenc))
  Stream #0:1 -> #0:1 (aac (native) -> aac (libfdk_aac))
Press [q] to stop, [?] for help
[h264_nvenc @ 0x55b4fe43fe00] Cannot load libnvidia-encode.so.1
[h264_nvenc @ 0x55b4fe43fe00] The minimum required Nvidia driver for nvenc is 470.57.02 or newer
Error initializing output stream 0:0 -- Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height
Conversion failed!


In configuration, I only checked H264, HEVC and "Enable enhanced NVDEC decoder", "Enable hardware encoding". I've tried to check the corresponding settings for the card, but same thing happens.

As you said, it is another problem, something about a library but... What could be happening? And why?

# nvidia-smi
+-----------------------------------------------------------------------------+

| NVIDIA-SMI 525.116.04  Driver Version: 525.116.04  CUDA Version: 12.0    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|        Memory-Usage | GPU-Util  Compute M. |
|                              |                      |              MIG M. |
|===============================+======================+======================|
|  0  NVIDIA GeForce ...  Off  | 00000000:03:00.0 Off |                  N/A |
|  0%  36C    P8    6W / 151W |    154MiB /  8192MiB |      0%      Default |
|                              |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU  GI  CI        PID  Type  Process name                  GPU Memory |
|        ID  ID                                                  Usage      |
|=============================================================================|
+-----------------------------------------------------------------------------+

RE: NVENC not working - nyanmisaka - 2023-08-10

FFmpeg failed to load the "libnvidia-encode.so.1". It's a separate package and may not be included when installing drivers and CUDA.

On Debian, run "sudo apt install libnvidia-encode1". On Ubuntu, run "sudo apt install libnvidia-encode".

For more details please check https://jellyfin.org/docs/general/administration/hardware-acceleration/nvidia#debian-and-ubuntu-linux

RE: NVENC not working - friki67 - 2023-08-11

(2023-08-10, 02:44 PM)nyanmisaka Wrote: On Ubuntu, run "sudo apt install libnvidia-encode".

For more details please check https://jellyfin.org/docs/general/administration/hardware-acceleration/nvidia#debian-and-ubuntu-linux

Hello again. I checked it and it was installed, but no so.1 was present. So I reinstalled, and then I got
Quote:[AVHWDeviceContext @ 0x557337c25d80] cu->cuInit(0) failed -> CUDA_ERROR_COMPAT_NOT_SUPPORTED_ON_DEVICE: forward compatibility was attempted on non supported HW
Device creation failed: -542398533.
Failed to set value 'cuda=cu:0' for option 'init_hw_device': Generic error in an external library
Error parsing global options: Generic error in an external library

This is a library incompatibility. I think I messed up things when going driver up and down to make the testing.

I'm going to reinstall my OS, and I'm sure this is going to work.

Thank you for your patience and help. Best regards.