How to open port for specific IP (Droplet with VPN)

I have a Wireguard VPN on my droplet and want to open ports for my email server which is connected via VPN. As i have seen the firewall only supports opening ports for specific IP’s from the outside? So is it possible or should i route it? I already tried it to route with nginx with this but it did not work

# nginx wireguard config
server {
       listen 51820;

       location / {
               proxy_pass http://wireguard:51820;
               proxy_set_header Host $host;
               proxy_set_header X-Real-IP $remote_addr;
       }
}