I was able to figure out the issue. It was OSSEC. I had an ossec-hids agent also installed in the server. The active-response feature in the ossec was blocking IP for ssh failures. It was fixed by adding the IP to be ignored inside <white_list>1.2.3.4</white_list> in a global block in ossec-manager ossec.conf. Thanks for all the helpful tips guys.

PAM (Pluggable authentication modules) allows you to define flexible mechanism for authenticating users. My previous post demonstrated how to deny or allow users using sshd configuration option. However, if you want to block or deny a large number of users, use PAM configuration.

A note for new sys admins
Backup all data and PAM configuration files before any modification 🙂
Please be careful to perform the configuration option. Wrong configuration can lock down all login access including root access.
Read this Linux-PAM configuration file syntax guide
Now continue reading below for pamlistfile.so configration…
Use of pamlistfile.so module
This PAM module authenticates users based on the contents of a specified file. For example, if username exists in a file /etc/sshd/ssh.allow, sshd will grant login access.

Hi,
There is at least one PAM module which could cause the behavior you described. It is pam_faildelay, which allows you to set the delay on failure per-application. In that case delay value is specified in config file in /etc/pam.d/ directory. Delay is in microseconds. Of course, there may be another PAM modules with similar functionality, which I know nothing about. But it is quite likely that their delay values are being defined in configuration files too. However, their delay values may be in different units (seconds, milliseconds, etc.). So, to find them, try to look through contents of the config files using grep command searching for two or more digit strings.

sudo grep -rnwE "/etc/pam.d/" -e "[0-9]{2}"

You will get a list of files with numbered lines containing two or more digit strings.

Be aware, that the behavior you described may be caused by other utilities than PAM modules (e.g. SSHGuard, Fail2ban, DenyHosts), or even advanced firewall rules.