"Too many authentication failures" through VNC after a day of having a droplet run?

I’ve set up a server running Debian 9 and installed VNC using this guide:

https://www.digitalocean.com/community/tutorials/how-to-set-up-vnc-server-on-debian-8

Everything seems to work fine but it doesn’t last for long. After a day or two of running without issues my VNC viewer gives me the error “Too many authentication failures.” I can still SSH into the server without any issues, but I really need to access the desktop without resetting VNC.

It’s not even a password issue or limited to a single client, as I tried to enter through my Raspberry Pi and I got the same error even though it gave me no password input and I’ve never entered through the Pi before.

Long ago I set up a server with Ubuntu and I had the same issue, but remaking the droplet with Debian seemed to fix the issue, but now I’m having it again in a new droplet even though the distro is still Debian. I’ve already tried resetting vncpasswd but it doesn’t change anything.

What can I do to enter the server again, and hopefully fix the issue for good so it doesn’t happen again?