Is iptables rejecting TLS negotiation packets?

November 2, 2015 1.9k views
VPN Networking Firewall


I have an interesting problem that I can't seem to find the solution.

Basically, I have my email server hosted at home, as well as a couple of other services, like a HTTP server (with SSL). To properly expose them to the public internet, I have a VPS at Digital Ocean with OpenVPN plus iptables, which I use mostly to get a public IP to my internal VMs, as well as to get a valid hostname for my email server with a PTR record.

The setup look like this:

  • Public internet => Droplet => iptables forward to 10.0.0.x (a VM at home) => 10.0.0.x processes and replies back => iptables forwards reply to requester

This works pretty much most of the time, and has worked since a couple of years now, but, TLS traffic stopped working a few days ago when the VPS is involved. In other words: I have no problems if I connect to my VM from my internal home network, no problems if I connect from the public internet to port 80, but the connection "hangs" if connecting to 443 (at the beginning of the TLS negotiation) or SMTP after "starttls" is issued. I'm trying now what happens if I ditch Digital Ocean by another provider (to see if it's something on their network).

I have tried to create a new VM (Fedora 22 and CentOS), to rule out problems specific to the VM, but I got the same problem. I also tried to boot a new VPS, to rule out problems specific to the old one, but I get the same problem there as well.

All in all, the only thing that seems to matter is whether the connection is passing thru Digital Ocean/iptables or not.

I got a few TCP dumps for the connections, but my knowledge doesn't go that far into understanding what's wrong. My feeling is that "something" in the TLS negotiation is being filtered out by iptables, but I have no idea what, why and how. Any ideas would be certainly welcome!

What I have for debugging is this (those logs are for the new VPS and new VM, but the same thing happen with the ones that are visible as right now):

TCP dump from the OpenVPN server (Droplet):

TCP dump from the internal VM:

Firewall rules on the Droplet:

Successful connection from the public internet to the internal VM, thru the Droplet, on the port 80:

Sample of connection that hangs when connecting from the public internet to the internal VM, thru the Droplet, on port 443:

Successful connection from the internal network on port 443:


  • After some more debugging, it seems the problem might be either on my router or ISP: I tried connecting the host of the VMs into my phone's data connection and everything worked.

  • It definitely seems like your ISP might be blocking traffic on that port. Hosting email is sometimes, fraught, especially from a private network, and ISPs often will block email sending traffic.

  • True, and that's the main reason I have a droplet with Digital Ocean: so that the ISP is just a "dumb pipe".

Be the first one to answer this question.