Whitelisted IP not working with Nginx-Ingress Load Balancer

I deployed an App that is running fine in my cluster. During the cluster creation I installed the Nginx-Ingress, created a k8s Ingress and configured the Forwarding Rules of the load balancer as HTTPS with a valid certificate. All this is working fine. My Ingress is as follows:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-ingress
  annotations:
    kubernetes.io/tls-acme: "true"
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/externalTrafficPolicy: Local
spec:
  rules:
  - host: dev.mysite.com
    http:
      paths:
      - path: /api
        pathType: Prefix
        backend:
          service:
            name: bkendapi
            port:
              number: 8005
      - path: /rec
        pathType: Prefix
        backend:
          service:
            name: bkendrec
            port:
              number: 5005

All this is working fine.

Now I am trying to add a Whitelist, this is the first time I am doing it, to do so I added the following line under annotations.

nginx.ingress.kubernetes.io/whitelist-source-range: xxx.xxx.xxx.xxx

Where the xxx.xxx.xxx.xxx is an IP of a specific DO droplet that will connect to the cluster. But now everything get blocked, I can not access the cluster from the whitelisted IP or from anywhere else. I tried different IPs but failed.

Please, what am I doing wrong/missing?

Thanks!