I fixed the failed to set value 'cuda=cu:0' error some how. I think it was reinstalling the drivers or it was restarting the nas, but i am encountering another problem. When transcoding the transcode just doesn't get processed it think. I see the process in nvidia-smi, but in nvidia-smi dmon no resources are being used. In the ffmpeg log i just see it getting stuck on a frame and no resources are being used.
For hardware decoding I have h264, hevc, hevc 10bit, vp9 10bit and I don't have advanced nvdec enabled i did turn it on and off a couple of times to test don't know if you need te restart so not sure it's on or off