port 80 is accessible through localhost but the connection is refused externally

February 23, 2016 6.7k views
Networking Ubuntu

The domain name pointing to my droplet can't be accessed at port 80, but 443 works:
$ wget http://{my ip}
--2016-02-22 23:33:51-- http://{my ip}/
Connecting to {my ip}... failed: Connection refused.
$ wget https://{my ip}
--2016-02-22 23:34:09-- https://{my ip}/
Connecting to {my ip}:443... connected.

In the droplet itself, I can access port 80 (where the web server redirects to 443)
$ wget http://localhost
--2016-02-23 02:35:29-- http://localhost/
Resolving localhost (localhost)...
Connecting to localhost (localhost)||:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://localhost/ [following]
--2016-02-23 02:35:29-- https://localhost/
Connecting to localhost (localhost)||:443... connected.

The server itself is a Play Framework server which listens on both port 80 and 443 using authbind, running as an unprivileged user. (both with IPv6)
$ authbind --deep myserver -Dhttp.port=80 -Dhttps.port=443
[info] - play.api.Play - Application started (Prod)
[info] - play.core.server.NettyServer - Listening for HTTP on /0:0:0:0:0:0:0:0:80
[info] - play.core.server.NettyServer - Listening for HTTPS on port /0:0:0:0:0:0:0:0:443

It is basically a vanilla Ubuntu 15.10 x32 droplet. Is there some obvious place to look for configuration problems? I would just like to keep the networking simple and serve directly from port 80, and it's hard to determine what is stopping me from doing so.

  • One thing is what the program itself says it's listening on. Another thing is what it's actually listening on. Double check that you are actually bound to the correct interfaces by looking at netstat -nlp.

    I'm going to guess that it's just listening on port 80 for localhost, and that is why you're not able to connect from the outside. Could be that the port was bound by another process or just an old authbind process that for some reason failed to shut down completely.

  • Thanks for the help! Everything looks normal in netstat:
    Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
    tcp6 0 0 :::443 :::* LISTEN 11337/java

    tcp6 0 0 :::80 :::* LISTEN 11337/java

    And lsof matches as well:
    $ lsof -i :80
    java 11337 mgruen 136u IPv6 268554 0t0 TCP *:http (LISTEN)

  • @gracenotes - Do you have the firewall up?

    ufw status verbose
  • @gndo: does not look like it at the moment,
    $ sudo ufw status verbose
    Status: inactive

  • @gracenotes - when you ran wget with "--verbose", did it hang (connection timeout) or did it say "connect" at some point and then a http response code that is not 200?

    wget --verbose http://yourIPorDomain:80
  • @gracenotes - Ignore my last question, I am starting to see the problem now. Your http site is redirecting to your https site, and for some reason the security handshake is failing.

  • @gracenotes - OK, I think I understand the problem. Somewhere in your configuration, you are redirecting traffic to "localhost:443" instead of "yourIPorDomain:443". You need to find where you define that.

  • @gndo: sorry for delay. The website is all set up (and running on HTTPS successfully for a week or so now), it seems like the port is just not going through/listening externally.
    $ wget --verbose <domain>
    --2016-03-06 21:13:53-- http://<domain>/
    Resolving <domain>... <ip>
    Connecting to <domain>|<ip>|:80... failed: Connection refused.

    This happens pretty much immediately.

  • @gracenotes - no problem with the delay. The output "...: Connection refused." confirms that there is an entity listening at external port 80. One possible reason for the rejection is that it is redirecting port 80 explicitly to "localhost:443" instead of "yourDomain:443", which would explain why it works from your droplet but not externally.

    Another possible reason for the rejection is that some security info is not being carried over to requests over port 80. A quick googling shows me a configuration value of "session.secure=false" would allow this security info to be stored in a session cookie for both ports. I didn't recommend this since you were able to access both ports locally on your droplet.

  • @gndo - ah, I see what you mean! There is HTTP -> HTTPS redirect, but I wrote it myself, and all it does it change the URL scheme and nothing else. It has to go through a routing file in my server (i.e. start a normal request/response cycle) before it aborts the request and adds the Location header in the response.

    I did see that doing this inside the droplet did work:
    $ wget --verbose -S <domain>:80
    Connecting to <domain> (<domain>)|<ip>|:80... connected.
    HTTP request sent, awaiting response...
    HTTP/1.1 301 Moved Permanently
    Location: https://<domain>/
    <subsequent successful fetch of page over HTTPS>

    ifconfig shows that eth0 is bound to the resolved IP address, so the HTTP request doesn't have to go very far to get that redirect response. Meanwhile, no requests initiated from my home network will produce any response at port 80, or actually any port except 443 or 22! (Except if I start the server at a high port, like port 9000, I can connect to it over plain HTTP.)

1 Answer

@gracenotes - On your server, try modifying and trying the following before starting your authbind command:

touch /etc/authbind/byaddr/${MYINET},80
chown ${PLAYUSERNAME} /etc/authbind/byaddr/${MYINET},80
chmod 755 /etc/authbind/byaddr/${MYINET},80
  • Unfortunately this did not work (the process could bind to port 80 as before, but not externally accessible). But I discovered something extraordinarily strange and now it "works".

    In order to see if any binding to port 80 could work at all, I started the server up at port 9000 and ran sudo python -m SimpleHTTPServer 80 from an empty directory. To my surprise, I could now access my server at port 80 from both my web browser and wget, where it correctly redirected to HTTPS, and it was not possible to access the Python server running at port 80, even given this netstat and lsof (for ports 80 and 9000) output:

    tcp 0 0* LISTEN
    tcp6 0 0 :::22 :::* LISTEN
    tcp6 0 0 :::443 :::* LISTEN
    tcp6 0 0 :::9000 :::* LISTEN
    python 2338 root 3u IPv4 2772995 0t0 TCP *:http (LISTEN)
    java 3744 mgruen 138u IPv6 2775649 0t0 TCP *:9000 (LISTEN)

    Starting up the server on ports 9001 or 9002 caused port 80 to be no longer accessible externally. This was independent of whether the Python server was running, and independent of which was started up first between the two of them. At this point I should mention that I have a default iptables config (ACCEPTs everything), and as far as I know no other processes which listen to ports besides postgres and the ones above.

    What could possibly be causing this? Surely nothing in Play Framework itself?

    • @gracenotes - The fix I was going for was with authbind misbehaving, since it is acting as an intermediary between the privileged ports (ports numbers < 1024) and the processes/special users that are allowed to bind to them. At this point I'm out of ideas as to what could possibly be going on with your system. Sorry couldn't find the correct solution.

Have another answer? Share your knowledge.