Setting up NGINX Docker with allowed public IPs

I currently have my nginx.conf file for my DigitalOcean app as follows

server {
    allow <my public ip>
    deny all;

    ...website config
}

However I still get a 403 forbidden when accessing my website, and looking at the logs in “Runtime logs” it looks to be picking up a completely different IP address, in the same range as the app’s local IP when running ifconfig. Is this going through some kind of proxy, and if so how do I allow access for public IP addresses?