I can't believe I didn't notice this earlier. You are trying to encode to AV1. I'm not aware of any Synology devices that support encoding to AV1. Uncheck AV1 ENCODING and try again.
Jellyfin 10.10.7 (Docker)
Ubuntu 24.04.2 LTS w/HWE
Intel i3 12100
Intel Arc A380
OS drive - SK Hynix P41 1TB
Storage
4x WD Red Pro 6TB CMR in RAIDZ1