SOLVED: Poor transcoding performance in proxmox

[croxis]

Hello! I have jellyfin running in an arch linux lxc container in proxmox. I've been successfully running transcoding on an old nvidia with nvenc. For reasons* I had to switch it out with an amd 560 with vaapi. 10bit 4k transcoding to 1080 is now about 14 fps, resulting in jumpy playback. Is the card that poor performing? Or are there other things I can try to diagnose where the problem is?

*FFmpeg on arch now needs nvenc 12, but the legacy drivers for my 660 only suppot nvec 11

[skribe]

The first step here would probably be to share an ffmpeg log for one of the poorly performing streams so that we can look to make sure that the hardware acceleration is even working correctly. But also, yes, AMD cards are significantly worse, as a category, at HWA transcoding. So it MIGHT just be the card. But go ahead and share your ffmpeg log and we can look at it to get a better picture.

[croxis]

Code:

/videos/ce3c6745-6914-5f9d-8e5a-e305d562c4f8/hls1/main/0.ts

{"Protocol":0,"Id":"ce3c674569145f9d8e5ae305d562c4f8","Path":"/opt/media/TV/Our Flag Means Death/Season 1/our.flag.means.death.s01e01.dv.hdr.2160p.web.h265-ggwp.mkv","EncoderPath":null,"EncoderProtocol":null,"Type":0,"Container":"mkv,webm","Size":5290805760,"Name":"our.flag.means.death.s01e01.dv.hdr.2160p.web.h265-ggwp","IsRemote":false,"ETag":"3cce09a70dd55d3fa8430ae1f029e3a6","RunTimeTicks":19694080000,"ReadAtNativeFramerate":false,"IgnoreDts":false,"IgnoreIndex":false,"GenPtsInput":false,"SupportsTranscoding":true,"SupportsDirectStream":true,"SupportsDirectPlay":true,"IsInfiniteStream":false,"RequiresOpening":false,"OpenToken":null,"RequiresClosing":false,"LiveStreamId":null,"BufferMs":null,"RequiresLooping":false,"SupportsProbing":true,"VideoType":0,"IsoType":null,"Video3DFormat":null,"MediaStreams":[{"Codec":"hevc","CodecTag":null,"Language":null,"ColorRange":null,"ColorSpace":"bt2020nc","ColorTransfer":"smpte2084","ColorPrimaries":"bt2020","DvVersionMajor":1,"DvVersionMinor":0,"DvProfile":8,"DvLevel":6,"RpuPresentFlag":1,"ElPresentFlag":0,"BlPresentFlag":1,"DvBlSignalCompatibilityId":1,"Comment":null,"TimeBase":"1/1000","CodecTimeBase":null,"Title":null,"VideoRange":"HDR","VideoRangeType":"HDR10","VideoDoViTitle":"DV Profile 8.1 (HDR10)","LocalizedUndefined":null,"LocalizedDefault":null,"LocalizedForced":null,"LocalizedExternal":null,"DisplayTitle":"4K HEVC HDR","NalLengthSize":null,"IsInterlaced":false,"IsAVC":null,"ChannelLayout":null,"BitRate":21491963,"BitDepth":10,"RefFrames":1,"PacketLength":null,"Channels":null,"SampleRate":null,"IsDefault":true,"IsForced":false,"Height":1920,"Width":3840,"AverageFrameRate":23.976025,"RealFrameRate":23.976025,"Profile":"Main 10","Type":1,"AspectRatio":"2:1","Index":0,"Score":null,"IsExternal":false,"DeliveryMethod":null,"DeliveryUrl":null,"IsExternalUrl":null,"IsTextSubtitleStream":false,"SupportsExternalStream":false,"Path":null,"PixelFormat":"yuv420p10le","Level":153,"IsAnamorphic":null},{"Codec":"eac3","CodecTag":null,"Language":"eng","ColorRange":null,"ColorSpace":null,"ColorTransfer":null,"ColorPrimaries":null,"DvVersionMajor":null,"DvVersionMinor":null,"DvProfile":null,"DvLevel":null,"RpuPresentFlag":null,"ElPresentFlag":null,"BlPresentFlag":null,"DvBlSignalCompatibilityId":null,"Comment":null,"TimeBase":"1/1000","CodecTimeBase":null,"Title":null,"VideoRange":null,"VideoRangeType":null,"VideoDoViTitle":null,"LocalizedUndefined":null,"LocalizedDefault":null,"LocalizedForced":null,"LocalizedExternal":null,"DisplayTitle":"English - Dolby Digital\u002B - 5.1 - Default","NalLengthSize":null,"IsInterlaced":false,"IsAVC":null,"ChannelLayout":"5.1","BitRate":640000,"BitDepth":null,"RefFrames":null,"PacketLength":null,"Channels":6,"SampleRate":48000,"IsDefault":true,"IsForced":false,"Height":null,"Width":null,"AverageFrameRate":null,"RealFrameRate":null,"Profile":null,"Type":0,"AspectRatio":null,"Index":1,"Score":null,"IsExternal":false,"DeliveryMethod":null,"DeliveryUrl":null,"IsExternalUrl":null,"IsTextSubtitleStream":false,"SupportsExternalStream":false,"Path":null,"PixelFormat":null,"Level":0,"IsAnamorphic":null},{"Codec":"subrip","CodecTag":null,"Language":"eng","ColorRange":null,"ColorSpace":null,"ColorTransfer":null,"ColorPrimaries":null,"DvVersionMajor":null,"DvVersionMinor":null,"DvProfile":null,"DvLevel":null,"RpuPresentFlag":null,"ElPresentFlag":null,"BlPresentFlag":null,"DvBlSignalCompatibilityId":null,"Comment":null,"TimeBase":"1/1000","CodecTimeBase":null,"Title":"English [SDH]","VideoRange":null,"VideoRangeType":null,"VideoDoViTitle":null,"LocalizedUndefined":"Undefined","LocalizedDefault":"Default","LocalizedForced":"Forced","LocalizedExternal":"External","DisplayTitle":"English [SDH] - SUBRIP","NalLengthSize":null,"IsInterlaced":false,"IsAVC":null,"ChannelLayout":null,"BitRate":null,"BitDepth":null,"RefFrames":null,"PacketLength":null,"Channels":null,"SampleRate":null,"IsDefault":false,"IsForced":false,"Height":null,"Width":null,"AverageFrameRate":null,"RealFrameRate":null,"Profile":null,"Type":2,"AspectRatio":null,"Index":2,"Score":null,"IsExternal":false,"DeliveryMethod":null,"DeliveryUrl":null,"IsExternalUrl":null,"IsTextSubtitleStream":true,"SupportsExternalStream":true,"Path":null,"PixelFormat":null,"Level":0,"IsAnamorphic":null},{"Codec":"subrip","CodecTag":null,"Language":"spa","ColorRange":null,"ColorSpace":null,"ColorTransfer":null,"ColorPrimaries":null,"DvVersionMajor":null,"DvVersionMinor":null,"DvProfile":null,"DvLevel":null,"RpuPresentFlag":null,"ElPresentFlag":null,"BlPresentFlag":null,"DvBlSignalCompatibilityId":null,"Comment":null,"TimeBase":"1/1000","CodecTimeBase":null,"Title":"Latin America Spanish","VideoRange":null,"VideoRangeType":null,"VideoDoViTitle":null,"LocalizedUndefined":"Undefined","LocalizedDefault":"Default","LocalizedForced":"Forced","LocalizedExternal":"External","DisplayTitle":"Latin America Spanish - SUBRIP","NalLengthSize":null,"IsInterlaced":false,"IsAVC":null,"ChannelLayout":null,"BitRate":null,"BitDepth":null,"RefFrames":null,"PacketLength":null,"Channels":null,"SampleRate":null,"IsDefault":false,"IsForced":false,"Height":null,"Width":null,"AverageFrameRate":null,"RealFrameRate":null,"Profile":null,"Type":2,"AspectRatio":null,"Index":3,"Score":null,"IsExternal":false,"DeliveryMethod":null,"DeliveryUrl":null,"IsExternalUrl":null,"IsTextSubtitleStream":true,"SupportsExternalStream":true,"Path":null,"PixelFormat":null,"Level":0,"IsAnamorphic":null},{"Codec":"subrip","CodecTag":null,"Language":"por","ColorRange":null,"ColorSpace":null,"ColorTransfer":null,"ColorPrimaries":null,"DvVersionMajor":null,"DvVersionMinor":null,"DvProfile":null,"DvLevel":null,"RpuPresentFlag":null,"ElPresentFlag":null,"BlPresentFlag":null,"DvBlSignalCompatibilityId":null,"Comment":null,"TimeBase":"1/1000","CodecTimeBase":null,"Title":"Portuguese Brazilian","VideoRange":null,"VideoRangeType":null,"VideoDoViTitle":null,"LocalizedUndefined":"Undefined","LocalizedDefault":"Default","LocalizedForced":"Forced","LocalizedExternal":"External","DisplayTitle":"Portuguese Brazilian - SUBRIP","NalLengthSize":null,"IsInterlaced":false,"IsAVC":null,"ChannelLayout":null,"BitRate":null,"BitDepth":null,"RefFrames":null,"PacketLength":null,"Channels":null,"SampleRate":null,"IsDefault":false,"IsForced":false,"Height":null,"Width":null,"AverageFrameRate":null,"RealFrameRate":null,"Profile":null,"Type":2,"AspectRatio":null,"Index":4,"Score":null,"IsExternal":false,"DeliveryMethod":null,"DeliveryUrl":null,"IsExternalUrl":null,"IsTextSubtitleStream":true,"SupportsExternalStream":true,"Path":null,"PixelFormat":null,"Level":0,"IsAnamorphic":null}],"MediaAttachments":[],"Formats":[],"Bitrate":22131963,"Timestamp":null,"RequiredHttpHeaders":{},"TranscodingUrl":null,"TranscodingSubProtocol":null,"TranscodingContainer":null,"AnalyzeDurationMs":null,"DefaultAudioStreamIndex":null,"DefaultSubtitleStreamIndex":null}

ffmpeg -analyzeduration 200M -init_hw_device vaapi=va:/dev/dri/renderD128 -filter_hw_device va -hwaccel vaapi -autorotate 0 -i file:"/opt/media/TV/Our Flag Means Death/Season 1/our.flag.means.death.s01e01.dv.hdr.2160p.web.h265-ggwp.mkv" -autoscale 0 -map_metadata -1 -map_chapters -1 -threads 0 -map 0:0 -map 0:1 -map -0:s -codec:v:0 h264_vaapi -rc_mode VBR -b:v 21491963 -maxrate 21491963 -bufsize 42983926 -force_key_frames:0 "expr:gte(t,0+n_forced*3)" -vf "setparams=color_primaries=bt709:color_trc=bt709:colorspace=bt709,scale=trunc(min(max(iw\,ih*a)\,min(3840\,1920*a))/2)*2:trunc(min(max(iw/a\,ih)\,min(3840/a\,1920))/2)*2,format=nv12,hwupload=derive_device=vaapi" -codec:a:0 aac -ac 2 -ab 384000 -af "volume=2" -copyts -avoid_negative_ts disabled -max_muxing_queue_size 2048 -f hls -max_delay 5000000 -hls_time 3 -hls_segment_type mpegts -start_number 0 -hls_segment_filename "/var/lib/jellyfin/transcodes/2386f6f890c5acef8e7c6b8eef867d27%d.ts" -hls_playlist_type vod -hls_list_size 0 -y "/var/lib/jellyfin/transcodes/2386f6f890c5acef8e7c6b8eef867d27.m3u8"


ffmpeg version n6.0 Copyright (c) 2000-2023 the FFmpeg developers
  built with gcc 13.1.1 (GCC) 20230429
  configuration: --prefix=/usr --disable-debug --disable-static --disable-stripping --enable-amf --enable-avisynth --enable-cuda-llvm --enable-lto --enable-fontconfig --enable-gmp --enable-gnutls --enable-gpl --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libdav1d --enable-libdrm --enable-libfreetype --enable-libfribidi --enable-libgsm --enable-libiec61883 --enable-libjack --enable-libjxl --enable-libmfx --enable-libmodplug --enable-libmp3lame --enable-libopencore_amrnb --enable-libopencore_amrwb --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librav1e --enable-librsvg --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libsvtav1 --enable-libtheora --enable-libv4l2 --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxcb --enable-libxml2 --enable-libxvid --enable-libzimg --enable-nvdec --enable-nvenc --enable-opencl --enable-opengl --enable-shared --enable-version3 --enable-vulkan
  libavutil      58.  2.100 / 58.  2.100
  libavcodec    60.  3.100 / 60.  3.100
  libavformat    60.  3.100 / 60.  3.100
  libavdevice    60.  1.100 / 60.  1.100
  libavfilter    9.  3.100 /  9.  3.100
  libswscale      7.  1.100 /  7.  1.100
  libswresample  4. 10.100 /  4. 10.100
  libpostproc    57.  1.100 / 57.  1.100
Input #0, matroska,webm, from 'file:/opt/media/TV/Our Flag Means Death/Season 1/our.flag.means.death.s01e01.dv.hdr.2160p.web.h265-ggwp.mkv':
  Metadata:
    encoder        : libebml v1.4.2 + libmatroska v1.6.4
  Duration: 00:32:49.41, start: 0.000000, bitrate: 21491 kb/s
  Stream #0:0: Video: hevc (Main 10), yuv420p10le(tv, bt2020nc/bt2020/smpte2084), 3840x1920 [SAR 1:1 DAR 2:1], 23.98 fps, 23.98 tbr, 1k tbn (default)
    Side data:
      DOVI configuration record: version: 1.0, profile: 8, level: 6, rpu flag: 1, el flag: 0, bl flag: 1, compatibility id: 1
  Stream #0:1(eng): Audio: eac3, 48000 Hz, 5.1(side), fltp, 640 kb/s (default)
  Stream #0:2(eng): Subtitle: subrip
    Metadata:
      title          : English [SDH]
  Stream #0:3(spa): Subtitle: subrip
    Metadata:
      title          : Latin America Spanish
  Stream #0:4(por): Subtitle: subrip
    Metadata:
      title          : Portuguese Brazilian
Stream mapping:
  Stream #0:0 -> #0:0 (hevc (native) -> h264 (h264_vaapi))
  Stream #0:1 -> #0:1 (eac3 (native) -> aac (native))
Press [q] to stop, [?] for help
[h264_vaapi @ 0x562bddd7a9c0] Driver does not support some wanted packed headers (wanted 0xd, found 0).
[h264_vaapi @ 0x562bddd7a9c0] Driver does not support packed sequence headers, but a global header is requested.
[h264_vaapi @ 0x562bddd7a9c0] No global header will be written: this may result in a stream which is not usable for some purposes (e.g. not muxable to some containers).
Output #0, hls, to '/var/lib/jellyfin/transcodes/2386f6f890c5acef8e7c6b8eef867d27.m3u8':
  Metadata:
    encoder        : Lavf60.3.100
  Stream #0:0: Video: h264 (High), vaapi(tv, bt709, progressive), 3840x1920 [SAR 1:1 DAR 2:1], q=2-31, 21491 kb/s, 23.98 fps, 90k tbn (default)
    Metadata:
      encoder        : Lavc60.3.100 h264_vaapi
    Side data:
      DOVI configuration record: version: 1.0, profile: 8, level: 6, rpu flag: 1, el flag: 0, bl flag: 1, compatibility id: 1
  Stream #0:1(eng): Audio: aac (LC), 48000 Hz, stereo, fltp, 384 kb/s (default)
    Metadata:
      encoder        : Lavc60.3.100 aac
frame=    0 fps=0.0 q=-0.0 size=N/A time=00:00:00.00 bitrate=N/A speed=  0x   
frame=    7 fps=0.0 q=-0.0 size=N/A time=00:00:00.51 bitrate=N/A speed=0.885x   
frame=  15 fps= 14 q=-0.0 size=N/A time=00:00:00.76 bitrate=N/A speed=0.699x   
frame=  23 fps= 14 q=-0.0 size=N/A time=00:00:01.02 bitrate=N/A speed=0.635x   
frame=  31 fps= 15 q=-0.0 size=N/A time=00:00:01.30 bitrate=N/A speed=0.611x   
frame=  39 fps= 15 q=-0.0 size=N/A time=00:00:01.79 bitrate=N/A speed=0.672x   
frame=  47 fps= 15 q=-0.0 size=N/A time=00:00:02.04 bitrate=N/A speed=0.644x   
frame=  55 fps= 15 q=-0.0 size=N/A time=00:00:02.30 bitrate=N/A speed=0.624x   
frame=  63 fps= 15 q=-0.0 size=N/A time=00:00:02.81 bitrate=N/A speed=0.665x   
frame=  71 fps= 15 q=-0.0 size=N/A time=00:00:03.32 bitrate=N/A speed=0.703x   
[hls @ 0x562bddc2a300] Opening '/var/lib/jellyfin/transcodes/2386f6f890c5acef8e7c6b8eef867d270.ts' for writing
frame=  80 fps= 15 q=-0.0 size=N/A time=00:00:03.58 bitrate=N/A speed=0.68x   
frame=  88 fps= 15 q=-0.0 size=N/A time=00:00:03.66 bitrate=N/A speed=0.636x   
frame=  96 fps= 15 q=-0.0 size=N/A time=00:00:04.03 bitrate=N/A speed=0.64x   
frame=  104 fps= 15 q=-0.0 size=N/A time=00:00:04.30 bitrate=N/A speed=0.631x   
frame=  113 fps= 15 q=-0.0 size=N/A time=00:00:04.80 bitrate=N/A speed=0.652x   
frame=  122 fps= 15 q=-0.0 size=N/A time=00:00:05.05 bitrate=N/A speed=0.639x   
frame=  129 fps= 15 q=-0.0 size=N/A time=00:00:05.37 bitrate=N/A speed=0.639x   
frame=  138 fps= 15 q=-0.0 size=N/A time=00:00:05.82 bitrate=N/A speed=0.65x   
[hls @ 0x562bddc2a300] Opening '/var/lib/jellyfin/transcodes/2386f6f890c5acef8e7c6b8eef867d271.ts' for writing
frame=  147 fps= 15 q=-0.0 size=N/A time=00:00:06.33 bitrate=N/A speed=0.667x   

[q] command received. Exiting.

[hls @ 0x562bddc2a300] Opening '/var/lib/jellyfin/transcodes/2386f6f890c5acef8e7c6b8eef867d272.ts' for writing
frame=  151 fps= 15 q=-0.0 Lsize=N/A time=00:00:06.37 bitrate=N/A speed=0.648x   
video:517kB audio:225kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
[aac @ 0x562bddd01540] Qavg: 64305.176

[nyanmisaka]

You have to setup hardware acceleration by checking our docs. Jellyfin FFmpeg is required for the best performance.

https://jellyfin.org/docs/general/admini...arch-linux

[croxis]

Solved

The instruction I failed to follow was switching jellyfin's playback's ffmpeg's path to /usr/lib/jellyfin-ffmpeg/ffmpeg For some reason it was set to the vanilla ffmpeg.