UFW not blocking connections from other IPs

I have setup UFW with:

sudo ufw default allow outgoing
sudo ufw default deny incoming
sudo ufw allow ssh
sudo ufw allow from 123.456.78.9

sudo ufw status:

To                         Action      From
--                         ------      ----
22                         ALLOW       Anywhere                  
Anywhere                   ALLOW       123.456.78.9        
22 (v6)                    ALLOW       Anywhere (v6) 

How ever I am still able to connect to the node from other IPs on ports other than 22. What am I doing wrong? Tried reloading and restarting.

This is a very basic installation, at this point I have really only created a new user.