Based on the description you gave, it sounds like VestaCP is using a self-signed certificate. Chrome and Firefox cannot verify the certificate’s authenticity as it is not signed by a known Certificate Authority, so they show you a warning.

I have not tried it myself, but this tutorial looks good—give it a shot:

• https://blog.flaunt7.com/how-to-use-lets-encrypt-on-vesta-login-panel/

I’d recommend taking a Snapshot of your Droplet as a backup image that you can restore if anything goes wrong.