Point Floating Ip To Droplet with SSL

I am trying to configure a setup where my Floating IP sits in front of two Droplets so I am able to switch traffic between the two of them in order to have a High Availability configuration where I can deploy the latest version of my software to the Droplet that is not ‘active’ at that moment, similar to what is described here https://www.digitalocean.com/community/tutorials/how-to-set-up-highly-available-haproxy-servers-with-keepalived-and-floating-ips-on-ubuntu-14-04

So, that used to worked fine until I implemented ssl with Let’s Encrypt in my Droplets, since I only want my users to access my website (powered by nginx) through https. Now, when accessing my website using the public IP of my floating IP, I keep getting a certificate error from the server because that IP does not match the name of the certificate. It works as expected when I directly hit my server with https (or http, redirection to https works fine) with or without the www. This is my nginx configuration:

server {

        root /var/www/vue/package;
        index index.html index.htm index.nginx-debian.html;

        server_name droplet_domain.com www.droplet_domain.com;

        location / {
      		proxy_set_header   X-Forwarded-For $remote_addr;
      		proxy_set_header   Host $http_host;
      		proxy_pass         "http://droplet_public_ip:8081";
        }

	listen [::]:443 ssl ipv6only=on; # managed by Certbot
    	listen 443 ssl; # managed by Certbot
    	ssl_certificate /etc/letsencrypt/live/droplet_domain.com/fullchain.pem; # managed by Certbot
    	ssl_certificate_key /etc/letsencrypt/live/droplet_domain.com/privkey.pem; # managed by Certbot
    	include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    	ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot


}

server {
        listen 80;
        listen [::]:80;

        server_name droplet_anchor_ip droplet_domain.com www.droplet_domain.com;
 	return 301 https://$host$request_uri;
}

I expected the Floating IP to hit the server_name droplet_anchor_ip and get redirected, but I get the ERR_CERT_COMMON_NAME_INVALID on the browser. I wonder how I can get my floating IP working fine with https.

Thanks.