ASS subtitle burn in fails

[bitmap]

previously I used binhex version but then I couldn't even play any videos after enabling HWA.

This sounds like QSV is not working properly.

The fact that ffmpeg is quitting immediately, milliseconds into transcoding suggests that QuickSync isn't configured properly. For low-power encoding, you need a number of things configured, none of which were listed on the page you linked. I don't see a hardware acceleration config section for Unraid and I know there isn't a lot of low-level access on those systems (on purpose), but that makes it hard, particularly with hardware like the N5105 which requires the HWE kernel, HuC/GuC access, specific drivers, render/video group access, things that you can't just plug into a Portainer-like GUI.

Are you able to get bash access to the Jellyfin container anywhere? If so, you could test whether QSV is functioning (the linked HWA documentation has several ways to test/verify). That might be a good place to start.