SOLVED: need help with hardware accelerated transcoding using AMD card

[physicsme]

I'm using Jellyfin on a Truenas Scale system via virtualization.
I'm having trouble transcoding stuff using an AMD card. Nvidia is working just fine on the same installation.

Here's the transcode log:

Code:

/usr/lib/jellyfin-ffmpeg/ffmpeg -analyzeduration 200M -probesize 1G -init_hw_device drm=dr:/dev/dri/renderD128 -init_hw_device vaapi=va@dr -init_hw_device vulkan=vk@dr -filter_hw_device vk -hwaccel vaapi -hwaccel_output_format vaapi -noautorotate -i file:"(filename redacted).mkv" -noautoscale -map_metadata -1 -map_chapters -1 -threads 0 -map 0:0 -map 0:1 -map -0:s -codec:v:0 hevc_vaapi -tag:v:0 hvc1 -rc_mode VBR -b:v 36433184 -maxrate 36433184 -bufsize 72866368 -profile:v:0 main -sei -a53_cc -force_key_frames:0 "expr:gte(t,n_forced*3)" -flags:v -global_header -vf "setparams=color_primaries=bt2020:color_trc=smpte2084:colorspace=bt2020nc,hwmap=derive_device=vulkan,format=vulkan,libplacebo=upscaler=none:downscaler=none:format=bgra:tonemapping=bt.2390:peak_detect=0:color_primaries=bt709:color_trc=bt709:colorspace=bt709,hwmap=derive_device=vaapi,format=vaapi,scale_vaapi=format=nv12" -codec:a:0 libfdk_aac -ac 6 -ab 640000 -copyts -avoid_negative_ts disabled -max_muxing_queue_size 2048 -f hls -max_delay 5000000 -hls_time 3 -hls_segment_type mpegts -start_number 0 -hls_segment_filename "/cache/transcodes/946755f83897f1da43a30c43a8d82787%d.ts" -hls_playlist_type vod -hls_list_size 0 -y "/cache/transcodes/946755f83897f1da43a30c43a8d82787.m3u8"

.........
.........


Stream mapping:
  Stream #0:0 -> #0:0 (hevc (native) -> hevc (hevc_vaapi))
  Stream #0:1 -> #0:1 (dts (dca) -> aac (libfdk_aac))
Press [q] to stop, [?] for help
[hevc_vaapi @ 0x55c275cc5900] No usable encoding entrypoint found for profile VAProfileHEVCMain (17).
[vost#0:0/hevc_vaapi @ 0x55c275017e00] Error initializing output stream: Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height
[libfdk_aac @ 0x55c274faf940] 2 frames left in the queue on closing
Conversion failed!

Here is my vainfo output :

Code:

Trying display: drm
libva info: VA-API version 1.22.0
libva info: Trying to open /usr/lib/jellyfin-ffmpeg/lib/dri/radeonsi_drv_video.so
libva info: Found init function __vaDriverInit_1_22
amdgpu: os_same_file_description couldn't determine if two DRM fds reference the same file description.
If they do, bad things may happen!
libva info: va_openDriver() returns 0
vainfo: VA-API version: 1.22 (libva 2.22.0)
vainfo: Driver version: Mesa Gallium driver 24.0.9 for AMD Radeon RX 6400 (radeonsi, navi24, LLVM 16.0.6, DRM 3.54, 6.6.32-production+truenas)
vainfo: Supported profile and entrypoints
      VAProfileH264ConstrainedBaseline: VAEntrypointVLD
      VAProfileH264Main              : VAEntrypointVLD
      VAProfileH264High              : VAEntrypointVLD
      VAProfileHEVCMain              : VAEntrypointVLD
      VAProfileHEVCMain10            : VAEntrypointVLD
      VAProfileVP9Profile0            : VAEntrypointVLD
      VAProfileVP9Profile2            : VAEntrypointVLD
      VAProfileNone                  : VAEntrypointVideoProc

I'm aware that there's no encoding profile. What could lead to this problem?

[Efficient_Good_5784]

I'm using Jellyfin on a Truenas Scale system via virtualization.
I'm having trouble transcoding stuff using an AMD card. Nvidia is working just fine on the same installation.

Do you have multiple GPU devices on the same system?

[physicsme]

I'm using Jellyfin on a Truenas Scale system via virtualization.
I'm having trouble transcoding stuff using an AMD card. Nvidia is working just fine on the same installation.

Do you have multiple GPU devices on the same system?

My motherboard does have an Aspeed integrated display as part of the BMC. I'm using an AMD CPU with no iGPU (Ryzen 5 4500) but strangely I still see an iGPU listed in lspci.

Code:

14:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Navi 24 [Radeon RX 6400/6500 XT/6500M] (rev c7)
22:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 41)
30:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Renoir

If you were asking about the Nvidia GPU. I wasn't using it along side the AMD one. I pulled it out and rebooted the system before testing the AMD card.

[TheDreadPirate]

If I'm reading AMD's git correctly, the RX6400 doesn't have ANY encoding capability?!?!

https://github.com/GPUOpen-LibrariesAndS...e-features

Which makes sense. The vainfo output does not indicate any encoding support. If it did you would see "VAEntrypointEncSliceLP". But your vainfo output only indicates decoding with "VAEntrypointVLD".

[physicsme]

I'm using Jellyfin on a Truenas Scale system via virtualization.
I'm having trouble transcoding stuff using an AMD card. Nvidia is working just fine on the same installation.

Do you have multiple GPU devices on the same system?

If I'm reading AMD's git correctly, the RX6400 doesn't have ANY encoding capability?!?!

https://github.com/GPUOpen-LibrariesAndS...e-features

Which makes sense.  The vainfo output does not indicate any encoding support.  If it did you would see "VAEntrypointEncSliceLP".  But your vainfo output only indicates decoding with "VAEntrypointVLD".

looks like this is the reason. 
I wasn't reading the specs very carefully when I bought the card for some casual gaming. 
Way to go AMD! So that's how Nvidia's getting away charging crazy prices for entry level cards like rtx 3050, rtx a400 etc....


https://www.reddit.com/r/buildapc/commen...orkaround/
https://news.ycombinator.com/item?id=39135243