Try to use tcpdump command to capture the traffic especially the mail server ports like 110 - 25 .... (depends on your service you provide) then analyze the captured traffic and block the inappropriate sources with iptables.

Examples of commands just to get started:

tcpdump -nty any port 25

Block the traffic from specific source

iptables -t filter -s 1.2.3.4 -j DROP