It’s been so long since X broke that most people won’t know what you mean by xorg.conf

Simple doesn’t mean well done. Badly written code can be simple but still bad

If you do want to open 22, and there are plenty of good reasons to want to, just implement something called port knocking and you can do it safely.

Note with this you still need good authentication. That means no passwords, key based auth only.