Transcoding on i7 4770k & GTX 770 (docker)

[zjeffer]

I moved my jellyfin configuration from a Raspberry Pi 4 to a secondhand desktop PC I just bought. These are the specs:

• Intel i7 4770k
  
• GTX 770
  
• 16 GB RAM
  

I'm running it with the following docker-compose file:

Code:

services:
  jellyfin:
    image: linuxserver/jellyfin:latest
    container_name: jellyfin
    restart:  unless-stopped
    ports:
      - "8096:8096"
      - "8920:8920"
    networks:
      - proxy
    volumes:
      - ./data/jellyfin-config:/config
      - ./data/jellyfin-cache:/cache
      - /run/media/admin/Elements:/media
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=Europe/Brussels
      - NVIDIA_VISIBLE_DEVICES=all
      - NVIDIA_DRIVER_CAPABILITIES=compute,utility
    runtime: nvidia
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

 
  caddy:
    image: caddy:latest
    container_name: caddy
    restart: unless-stopped
    ports:
      - "80:80"
      - "443:443"
      - "443:443/udp"
    networks:
      - "proxy"
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile
      - ./data/caddy_data:/data
      - ./data/caddy_config:/config

networks:
  proxy:
    driver: bridge

If I try to play something with transcoding enabled (Nvidia NVENC), the ffmpeg logs show this, and playback stops:

Code:

[AVHWDeviceContext @ 0x605e7a2ffd00] Cannot load libcuda.so.1 
[AVHWDeviceContext @ 0x605e7a2ffd00] Could not dynamically load CUDA Device creation failed: -1. 
Failed to set value 'cuda=cu:0' for option 'init_hw_device': Operation not permitted

I took a look at the GPU matrix from Nvidia, and noticed the GTX 770 isn't even present in the list. I assume it doesn't have NVENC/NVDEC at all?

---

Here are my questions:

• Can I use NVENC with my GTX 770?
  
• If not, can I use the i7 4770k's integrated GPU with Intel QuickSync instead?
  
• I notice there's also VAAPI and Video4Linux2 support. Are any of these useful/compatible with my setup?
  

Thanks

[zjeffer]

I tested Intel QSV and now the following happens:

Code:

ffmpeg version 6.0.1-Jellyfin Copyright (c) 2000-2023 the FFmpeg developers
  built with gcc 11 (Ubuntu 11.4.0-1ubuntu1~22.04)
  configuration: --prefix=/usr/lib/jellyfin-ffmpeg --target-os=linux --extra-version=Jellyfin --disable-doc --disable-ffplay --disable-ptx-compression --disable-static --disable-libxcb --disable-sdl2 --disable-xlib --enable-lto --enable-gpl --enable-version3 --enable-shared --enable-gmp --enable-gnutls --enable-chromaprint --enable-opencl --enable-libdrm --enable-libass --enable-libfreetype --enable-libfribidi --enable-libfontconfig --enable-libbluray --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libopenmpt --enable-libdav1d --enable-libsvtav1 --enable-libwebp --enable-libvpx --enable-libx264 --enable-libx265 --enable-libzvbi --enable-libzimg --enable-libfdk-aac --arch=amd64 --enable-libshaderc --enable-libplacebo --enable-vulkan --enable-vaapi --enable-amf --enable-libvpl --enable-ffnvcodec --enable-cuda --enable-cuda-llvm --enable-cuvid --enable-nvdec --enable-nvenc
  libavutil      58.  2.100 / 58.  2.100
  libavcodec    60.  3.100 / 60.  3.100
  libavformat    60.  3.100 / 60.  3.100
  libavdevice    60.  1.100 / 60.  1.100
  libavfilter    9.  3.100 /  9.  3.100
  libswscale      7.  1.100 /  7.  1.100
  libswresample  4. 10.100 /  4. 10.100
  libpostproc    57.  1.100 / 57.  1.100
[AVHWDeviceContext @ 0x5d01da5ddec0] libva: /usr/lib/jellyfin-ffmpeg/lib/dri/iHD_drv_video.so init failed
[AVHWDeviceContext @ 0x5d01da5ddec0] Failed to initialise VAAPI connection: 1 (operation failed).
Device creation failed: -5.
Failed to set value 'vaapi=va:,kernel_driver=i915,driver=iHD' for option 'init_hw_device': Input/output error

In docker-compose, I deleted the nvidia stuff and added this:

Code:

devices:
      - /dev/dri:/dev/dri

---

Setting hardware acceleration to use VAAPI instead of Intel QSV throws this error in the ffmpeg logs:

Code:

[AVHWDeviceContext @ 0x5b9c63522580] Failed to initialise VAAPI connection: -1 (unknown libva error).
Device creation failed: -5.
No device available for decoder: device type vaapi needed for codec hevc.
Stream mapping:

[TheDreadPirate]

Nvidia dropped support for that generation of GPU. You are SOL. VAAPI MIGHT work, definitely not V4L2. But don't expect much from the 770 if VAAPI does work.

The 4770k could also work, also only with VAAPI. Of the two, I would choose your iGPU over the 770.

---

I tested Intel QSV and now the following happens:

Code:

ffmpeg version 6.0.1-Jellyfin Copyright (c) 2000-2023 the FFmpeg developers
  built with gcc 11 (Ubuntu 11.4.0-1ubuntu1~22.04)
  configuration: --prefix=/usr/lib/jellyfin-ffmpeg --target-os=linux --extra-version=Jellyfin --disable-doc --disable-ffplay --disable-ptx-compression --disable-static --disable-libxcb --disable-sdl2 --disable-xlib --enable-lto --enable-gpl --enable-version3 --enable-shared --enable-gmp --enable-gnutls --enable-chromaprint --enable-opencl --enable-libdrm --enable-libass --enable-libfreetype --enable-libfribidi --enable-libfontconfig --enable-libbluray --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libopenmpt --enable-libdav1d --enable-libsvtav1 --enable-libwebp --enable-libvpx --enable-libx264 --enable-libx265 --enable-libzvbi --enable-libzimg --enable-libfdk-aac --arch=amd64 --enable-libshaderc --enable-libplacebo --enable-vulkan --enable-vaapi --enable-amf --enable-libvpl --enable-ffnvcodec --enable-cuda --enable-cuda-llvm --enable-cuvid --enable-nvdec --enable-nvenc
  libavutil      58.  2.100 / 58.  2.100
  libavcodec    60.  3.100 / 60.  3.100
  libavformat    60.  3.100 / 60.  3.100
  libavdevice    60.  1.100 / 60.  1.100
  libavfilter    9.  3.100 /  9.  3.100
  libswscale      7.  1.100 /  7.  1.100
  libswresample  4. 10.100 /  4. 10.100
  libpostproc    57.  1.100 / 57.  1.100
[AVHWDeviceContext @ 0x5d01da5ddec0] libva: /usr/lib/jellyfin-ffmpeg/lib/dri/iHD_drv_video.so init failed
[AVHWDeviceContext @ 0x5d01da5ddec0] Failed to initialise VAAPI connection: 1 (operation failed).
Device creation failed: -5.
Failed to set value 'vaapi=va:,kernel_driver=i915,driver=iHD' for option 'init_hw_device': Input/output error

In docker-compose, I deleted the nvidia stuff and added this:

Code:

devices:
      - /dev/dri:/dev/dri

---

Setting hardware acceleration to use VAAPI instead of Intel QSV throws this error in the ffmpeg logs:

Code:

[AVHWDeviceContext @ 0x5b9c63522580] Failed to initialise VAAPI connection: -1 (unknown libva error).
Device creation failed: -5.
No device available for decoder: device type vaapi needed for codec hevc.
Stream mapping:

In your docker compose you need to pass in the render group into the container. My docker compose as an example.

Code:

services:
  jellyfin:
    image: jellyfin/jellyfin:10.9.2
    container_name: jellyfin-stable-testing
    user: 105:104
    group_add:
      - "993" # Intel render group
    network_mode: 'host'
    volumes:
      - ./jellyfin-data/config:/config
      - ./jellyfin-data/cache:/cache
      - /media/library:/media/library:ro
      - /media/storage2:/media/testLibrary:ro
    devices:
      - /dev/dri/renderD128:/dev/dri/renderD128
    restart: 'unless-stopped'
    extra_hosts:
      - "host.docker.internal:host-gateway"

On the host system you need to add the user the container is running as, looks like your user, to the render group.

Code:

sudo usermod -aG render <insert your username here>

[zjeffer]

I added group 989 (my render group) to the docker compose file, added render to my user's groups, and rebooted my whole system just to be sure.

New error:

Code:

DRM_IOCTL_I915_GEM_APERTURE failed: Invalid argument
Assuming 131072kB available aperture size.
May lead to reduced performance or incorrect rendering.
get chip id failed: -1 [22]
param: 4, val: 0
i915 does not support EXECBUFER2
[AVHWDeviceContext @ 0x5a668ecbf480] libva: /usr/lib/jellyfin-ffmpeg/lib/dri/i965_drv_video.so init failed
[AVHWDeviceContext @ 0x5a668ecbf480] Failed to initialise VAAPI connection: -1 (unknown libva error).
[AVFilterGraph @ 0x5a669029c000] Error initializing filters
Error reinitializing filters!
Failed to inject frame into filter network: Input/output error
Error while processing the decoded data for stream #0:0
[libfdk_aac @ 0x5a668e170c80] 2 frames left in the queue on closing

But if I choose a different media file (1080p H264), hardware accelerated transcoding with VAAPI does seem to work now!

The above error is with a media file that is 4K HEVC HDR. I haven't enabled hardware acceleration for HEVC, though.

[TheDreadPirate]

Yeah. HEVC is supported by your CPU/iGPU. The only codec boxes you can are H264, MPEG2, VC1, and VP8.

[zjeffer]

Yeah.  HEVC is supported by your CPU/iGPU.  The only codec boxes you can are H264, MPEG2, VC1, and VP8.

Do you mean *not* supported by my CPU/iGPU? I would assume if it's supported I could check them without errors.
According to this table the 4770k's architecture (Haswell) only supports MPEG-2 and AVC. VC1 and VP8 are supported partially (for transcoding, I guess I need both encoding and decoding support?).

The only codec boxes you can are H264, MPEG2, VC1, and VP8.

I now checked only those codecs, and I'm still getting this error when trying to play any HEVC content:

Code:

DRM_IOCTL_I915_GEM_APERTURE failed: Invalid argument
Assuming 131072kB available aperture size.
May lead to reduced performance or incorrect rendering.
get chip id failed: -1 [22]
param: 4, val: 0
i915 does not support EXECBUFER2
[AVHWDeviceContext @ 0x5d5b20540800] libva: /usr/lib/jellyfin-ffmpeg/lib/dri/i965_drv_video.so init failed
[AVHWDeviceContext @ 0x5d5b20540800] Failed to initialise VAAPI connection: -1 (unknown libva error).
[AVFilterGraph @ 0x5d5b20543dc0] Error initializing filters
Error reinitializing filters!
Failed to inject frame into filter network: Input/output error
Error while processing the decoded data for stream #0:0
[libfdk_aac @ 0x5d5b1f25e680] 2 frames left in the queue on closing

If I uncheck HEVC, doesn't that mean it should do software transcoding instead of trying to do hardware accelerated transcoding?

And thanks for the help so far

[TheDreadPirate]

I did mean unsupported.

The codec check boxes are for hardware DECODING. As long as you have VAAPI selected it will still do hardware ENCODING and the CPU will decode the original video.

Can you share the full ffmpeg log via pastebin? I want to see the full input and output.

[zjeffer]

I did mean unsupported.

The codec check boxes are for hardware DECODING.  As long as you have VAAPI selected it will still do hardware ENCODING and the CPU will decode the original video.

Can you share the full ffmpeg log via pastebin?  I want to see the full input and output.

https://pastebin.com/Wf0iwSSq

I just disabled hardware encoding, now the HEVC video plays fine (with software transcoding).
But doesn't this mean that if I play a video with a codec that my system *does* support (for example H264), it will now also do software encoding?

[TheDreadPirate]

Unchecking HEVC only means that the CPU DECODES the codec. If you play a HEVC video the CPU will DECODE the video, but the GPU still ENCODES the H264 transcode.

[zjeffer]

So if I understand correctly, if I enable hardware decoding for all supported codecs, and I check the hardware encoding box:

* HEVC will not be playable at all (due to the above errors)
* Supported codecs will be hardware decoded & encoded

But if I uncheck hardware encoding:

* HEVC will be playable because it relies completely on the CPU
* Supported codecs will be hardware decoded & software encoded

From my testing, this seems to check out.

Can't I have both playable (software encoded) HEVC content and fully hardware transcoded supported codecs?