New user looking for hardware and setup advice please

[bitmap]

Will the hardware above be capable of what I want to do?

Doesn't sound like there should be much transcoding and without 4K content, tonemapping shouldn't be a concern. It's a low-powered device, but could handle a bit more than you want to throw at it presently.

What codec/quality etc. should I be using?

Probably avoid 4K HDR unless you have devices that will play it back. The best advice is to match your library to the devices you plan to play it on. Generic recommendations are to use H264, main profile, 8-bit with AAC audio no more than 5.1 channels. That's not necessarily "the best" but it's likely the most compatible.

What installation method would you recommend?

I only have experience with Docker. It's my platform of choice. LXCs are interesting, but I try to keep things simple. Docker has two different image options and HA works great in a container as well. You miss out slightly on HACS, but you can get that working too. I would go Docker or bare metal, but I also hate extra overhead, which VMs suck up pretty quickly.