Ghost droplet: unable to redirect https://ip to my main domain

I have added multiple redirects to my ghost droplet, to make sure that

• HTTP goes to HTTPS
• www.domain.com goes to domain.com
• http://ip goes to domain.com

The only thing that I am unable to figure out to make httpS://ip go to domain.com

I followed the instructions in this topic (link) to make an extra SSL certificate for the www.domain.com portion. I reused it for my ip adress and that works fine for the http portion which now redirect to domain.com. Given Let’s encrypt does not give certificates for ip adresses it did not auto create a config for https://ip on port 443

I tried the following:

• Add the listen 443 to my ip.conf file (which does port 80; this causes nothing to work
• some variation of the below files, some with SSL certs, some with pointing to default_server etc. But nothing seems to be picked up.

server {
    listen 443 ssl http2;
    listen [::]:443 ssl http2;

    server_name 123.123.123.123.443;
    root /var/www/ghost/system/nginx-root;

    return 301 https://domain.com$request_uri;


    }

    location ~ /.well-known {
        allow all;
}

    client_max_body_size 50m;
}