SOLVED: Transcoding HEVC -> h264 error

[UbijamSceno]

Hey everyone; I've ran into this transcoding issue with my Jellyfin setup, when trying to play HEVC content on devices that don't natively support HEVC streaming. (cannot direct stream)
So when a device is trying to play HEVC file, it tries to transcode it to h264 but that results in error.

This is what I've been able to dig out from the ffpmeg logs, and it is always the same error:
Stream mapping:
  Stream #0:0 -> #0:0 (hevc (native) -> h264 (h264_amf))
  Stream #0:1 -> #0:1 (dts (dca) -> aac (libfdk_aac))
Press [q] to stop, [?] for help
[AVHWDeviceContext @ 000001e48c6fddc0] D3D11 to OpenCL mapping not usable.
[Parsed_hwmap_1 @ 000001e48f7e86c0] Failed to map frame: -40.
Error while filtering: Function not implemented
Failed to inject frame into filter network: Function not implemented
Error while processing the decoded data for stream #0:0
[libfdk_aac @ 000001e4ec4500c0] 2 frames left in the queue on closing
Conversion failed!

I'm running Jellyfin 10.9.4 on Windows Server 2022 virtual machine, that resides on a Intel NUC NUC8i7HVK running ESXi 8.0u2.
The system has a embedded discrete graphics card that is passed through to this virtual machine: AMD RX Vega M GH
I have AMD AMF enabled with decoding enabled for all the available formats.
I am using AMD Adrenalin drivers version: 23.20.792.2048 (Q1 / 2016) which are quite old but are the only drivers that actually work for my setup and recognize the GPU.
I haven't been able to find any way to update these to any newer version.

Does anyone have any ideas how I would be able to make this work or a workaround for playing HEVC files?

[nyanmisaka]

It looks like the required OpenCL extension is not available due to outdated AMD GPU driver.

Please download and install the latest Vega M driver (Version 23.10.01.46 1.08 GB 2023/12/27) and try again.

https://www.asus.com/supportonly/nuc8i7h..._download/

[UbijamSceno]

Yes that is what I assumed was the issue.
And I've found those drivers you linked before, but the issue with my setup is that this driver is for Windows 10/11 operating systems and not for Windows Server 2022 and cannot be installed.
The only Windows Server OS drivers ever released for this NUC are the ones I am using, hence why they're so old.
I had hoped is there was some kind of workaround or a way seperatley install/get these OpenCL extensions.
Or I have to find a way to "hack" these drivers to force them to install on my OS.

[UbijamSceno]

After a bunch of testing I was actually able to update the driver to Adrenalin version 22.8.1 (driver version 31.0.12019.5007) which also resolved the issue I had with transcoding on Jellyfin

If anyone ever has a similar problem to me, I had to completely wipe the installed drivers I had and then manually apply the driver file in device manger. And even when I did this, while installing it froze my whole NUC and had to do a hard reset, and after reseting the NUC and ESXi booting up and getting the VM back up, the driver was installed succesfully and the GPU was detected succesfully! The downside here is I cannot have any of the Adrenalin driver program suite installed, but not really a problem since I didn't have a use for any of the functionality it adds.

So far everything works normally and transcoding also works without any issues. So I'm not touching the drivers again hopefully.