Unable to select "too low" of quality of video

[Karmalakas]

The only thing that triggers a transcode is the bit rate value.  If you have a 10Mbps video and you select "1080P - 15Mbps", nothing happens because the bit rate in that quality setting is higher than the video's native bit rate.  If you select "720P - 8Mbps" it will trigger a transcode.

That makes sense

---

you must not have enabled tone mapping.  What GPU do you have?  Tone mapping is only supported on GTX 1000 series and newer.

Enabling tone mapping helped, but I'm a bit confused. If GPU doesn't support AV1, it falls back to SW encoding, doesn't it? If stream is SW encoded, how does the tone mapping work if it requires GPU?  
I have a GTX 1070Ti

[TheDreadPirate]

There are two AV1 check boxes.  The box at the top, in the list of codecs, is for DECODING.  There is a check box farther down called, in English "Allow encoding in AV1 format".

   

Both AV1 boxes should be unchecked.

If the TOP box is unchecked, the CPU will software decode AV1.  If the bottom box is unchecked it will ENCODE in other codecs, HEVC or H264 depending on the client.

[Karmalakas]

it will ENCODE in other codecs, HEVC or H264 depending on the client.

Ah.. Understood  I have both unchecked