I have not tested this solution but from what I understand of webmin/virtualmin authentication it should solve the issue.
First you’ll want to set a root password by logging in via ssh and running
Then, make sure that in your /etc/ssh/sshd_config file the following is set:
This will allow you to log in as root with a password from the console in the control panel (which your droplet sees as a local display and keyboard) but will not allow remote ssh access with a password.
Now that a password is set for the root user virtualmin should allow you to log in and you will have access to this without making your ssh service susceptible to brute force attacks.