It depends what you want to do with it. If it’s just for storing files/backups then encrypt them before uploading and make sure the key never goes anywhere near the VPS. If it’s for serving up something like a simple website, you probably care more about data integrity than exfiltration, so make sure you have the security, including selinux or equivalent, locked down, and regularly run integrity checks. If it’s for running something interactive, or where data will be generated or downloaded to the machine, you’re out of luck, there’s no even theoretical way of securing that against an adversary with that much access.
I’m using Hacker’s Keyboard, it’s got all the keys where I expect them. None of the others feel right, but the fact it hasn’t been updated in years does worry me. If anyone knows of a keyboard with a similar layout (separate number row, ctrl, esc, alt and cursor keys in place and the usual symbols as long press on the numbers) I’d love to try it out.