Thanks to @jasonjpeters and @pjha I understood that it is indeed possible and started experimenting then I accidentally figured it out.

If you would like to deny all traffic coming to your public IP and only allow connections over your Floating IP, you need to find your droplet’s Anchor IP and set your firewall to allow connections from it.

Find out your Anchor IP:
curl -s 169.254.169.254/metadata/v1/interfaces/public/0/anchor_ipv4/address

After that, you need to create your rules with ufw like this:

ufw allow from any to [your-anchor-ip] port [some-port]

with this configuration your droplet will accept connections to some port if they are coming over with your Floating IP. All direct connections to your droplet will be dropped/refused depending your configuration. If you would like to, for example, allow HTTP/S connections over your Floating IP but restrict SSH to only your real public ip, then it is something like that:

ufw allow from any to [your-anchor-ip] port 80
ufw allow from any to [your-anchor-ip] port 443
ufw allow from any to [your-real-ip] port 22

Hey there!

You can create an inbound rule for the specific port allowing that Floating IP as the inbound source to restrict all the traffic except that Floating IP.

Let me know if you have any further questions.

Regards,
Prakash Jha
Developer Support