is it enough to use caddy and your own domain name for more security ?

[TheDreadPirate]

Since SSH does not listen on port 80 or 443, a hypothetical attacker would not be able to gain ssh access to your PC.

If you expose port 22 for remote shell access, that is a different story. But you can take extra steps to protect yourself.

The primary way I protect ssh is by using PKI ONLY, no password logins via ssh. That significantly increases the difficulty of gaining access via brute force.

Even with password logins, as long as you use a strong password and practive good password hygiene you are fine. Specifically limiting, or straight up not, reusing of passwords and definitely avoiding using passwords that have been leaked via data breaches on other sites.

Another user asked a similar question and my message boils down to "Take security seriously, but also don't worry too much." As long as you update regularly, make efforts to do basic hardening of exposed applications, and practice good password hygiene, the likelihood of being hacked are very very very remote.