Make sure server_name is set to mysite.com and that the listen line doesn't contain default_server in /etc/nginx/sites-enabled/wordpress.

Then, add a new virtualhost called default with the following config:

server {
    listen 80 default_server;
    listen [::]:80 default_server;
    server_name _;

    return 403;
}

Finally, restart nginx:

sudo service nginx restart

This should make sure your site is served only when you're accessing it using the domain name. The IP address will just return a 403 Forbidden error.