NVENC not working

[friki67]

Computer: Intel Xeon 2670v3, 32GB ECC RAM, Nvidia GTX 750 (GM107, Maxwell 2nd gen) (tried a GTX 1070 too, same result)
OS: Ubuntu server 22.04 + Xfce as GUI, Nvidia driver 535, nvidia-smi returning all ok.
Jellyfin 10.8.10 is installed using the recommended method (curl the installation script) and works fine, except for the hw encode/decode part.

I've checked user groups and all the suggestions I've found but I'm failing in make it work.

This was working on my old installation. I needed to reinstall and I've been unable to get it working again. I'm suffering this in a Fedora installation too.

Please check https://developer.nvidia.com/video-encod...matrix-new
   

I've tried lot of configurations, the minimal is 
   

But I always get "Playback error: the client isn't compatible with the media and the server isn't sending a compatible media format"

So what am I missing?
PS: all files played ok in VLC, all H.265, one of them info:

Code:

Codec: MPEG-H Part 2/HEVC (H.265)(hevc)
Video Resolution: 3832x1632
Frame rate: 29.970628
Decoded format: Planar 4.2.0 YUV full scale

EDIT: ffmpeg log for this file

ffmpeg version 5.1.3-Jellyfin Copyright © 2000-2022 the FFmpeg developers
  built with gcc 11 (Ubuntu 11.4.0-1ubuntu1~22.04)
  configuration: --prefix=/usr/lib/jellyfin-ffmpeg --target-os=linux --extra-libs=-lfftw3f --extra-version=Jellyfin --disable-doc --disable-ffplay --disable-ptx-compression --disable-static --disable-libxcb --disable-sdl2 --disable-xlib --enable-lto --enable-gpl --enable-version3 --enable-shared --enable-gmp --enable-gnutls --enable-chromaprint --enable-libdrm --enable-libass --enable-libfreetype --enable-libfribidi --enable-libfontconfig --enable-libbluray --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libopenmpt --enable-libdav1d --enable-libwebp --enable-libvpx --enable-libx264 --enable-libx265 --enable-libzvbi --enable-libzimg --enable-libfdk-aac --arch=amd64 --enable-libsvtav1 --enable-libshaderc --enable-libplacebo --enable-vulkan --enable-opencl --enable-vaapi --enable-amf --enable-libmfx --enable-ffnvcodec --enable-cuda --enable-cuda-llvm --enable-cuvid --enable-nvdec --enable-nvenc
  libavutil      57. 28.100 / 57. 28.100
  libavcodec    59. 37.100 / 59. 37.100
  libavformat    59. 27.100 / 59. 27.100
  libavdevice    59.  7.100 / 59.  7.100
  libavfilter    8. 44.100 /  8. 44.100
  libswscale      6.  7.100 /  6.  7.100
  libswresample  4.  7.100 /  4.  7.100
  libpostproc    56.  6.100 / 56.  6.100
Input #0, matroska,webm, from 'filemnt/media/Star_Citizen_Hornet_Commercial_Launch(1).mkv':
  Metadata:
    encoder        : libDivXMediaFormat 4.0.0.0578
  Duration: 00:02:57.98, start: 0.000000, bitrate: 28726 kb/s
  Stream #0:0(eng): Video: hevc (Main), yuvj420p(pc), 3832x1632 [SAR 1:1 DAR 479:204], 29.97 fps, 29.97 tbr, 1k tbn (default)
  Stream #0:1(en): Audio: aac (LC), 44100 Hz, stereo, fltp (default)
Stream mapping:
  Stream #0:0 -> #0:0 (hevc (native) -> h264 (h264_nvenc))
  Stream #0:1 -> #0:1 (aac (native) -> aac (libfdk_aac))
Press [q] to stop, [?] for help
[hevc @ 0x558ef04ef180] Hardware is lacking required capabilities
[hevc @ 0x558ef04ef180] Failed setup for format cuda: hwaccel initialisation returned error.
Impossible to convert between the formats supported by the filter 'Parsed_setparams_0' and the filter 'auto_scale_0'
Error reinitializing filters!
Failed to inject frame into filter network: Function not implemented
Error while processing the decoded data for stream #0:0
Conversion failed

EDIT2: nvidia-smi
Tue Aug  8 11:07:50 2023     

+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.86.05              Driver Version: 535.86.05    CUDA Version: 12.2    |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf          Pwr:Usage/Cap |        Memory-Usage | GPU-Util  Compute M. |
|                                        |                      |              MIG M. |
|=========================================+======================+======================|
|  0  NVIDIA GeForce GTX 750        Off | 00000000:03:00.0  On |                  N/A |
| 24%  35C    P8              1W /  38W |    240MiB /  1024MiB |    10%      Default |
|                                        |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
                                                                                       
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU  GI  CI        PID  Type  Process name                            GPU Memory |
|        ID  ID                                                            Usage      |
|=======================================================================================|
|    0  N/A  N/A      1346      G  /usr/lib/xorg/Xorg                          218MiB |
|    0  N/A  N/A      2398      G  xfwm4                                        0MiB |
|    0  N/A  N/A    20649      G  /snap/vlc/3078/usr/bin/vlc                  13MiB |
+---------------------------------------------------------------------------------------+

This client isn't compatible with the media and the server isn't sending a compatible media format.Playback Error
This client isn't compatible with the media and the server isn't sending a compatible media format.