rdkaiser
By:
rdkaiser

What the heck is this extra IP address that sometimes masquerades as a client IP?

July 9, 2017 719 views
Apache High Availability Security Networking Ubuntu 16.04

When I request various pages from the local Apache virtual host (locally, from the same droplet) whose hostname resolves to a floating IP, the apache log shows two different IP addresses making the requests.

One of those IPs is my droplet's anchor IP, as expected. The other one is from an address in the same subnet, but is not configured on my droplet.

Usually, and for most urls, it shows the anchor IP.

However, requests for certain urls ("/server-status", for example) always appear to come from the other IP.

This is a problem for me because I want to limit access to certain pages from local and NATed tunnel address. However, I can't do that until I know whether it is safe to allow access from this other pesky IP address: 10.10.0.2

Configured interface:

  • The droplet has two interfaces: eth0 and lo.
  • eth0 has only two addresses: my_server_ip and floating_ip_anchor
  • Internal networking is not enabled.
  • ip addr show dev eth0
Output
. . .
    inet 10.10.0.66/16 scope global eth0
. . .

Verified anchor address info:

$ for attr in address netmask gateway; do
    echo -n "${attr}="
    curl -s http://169.254.169.254/metadata/v1/interfaces/public/0/anchor_ipv4/$attr
    echo
done
Output
address=10.10.0.66
netmask=255.255.0.0
gateway=10.10.0.1

Examples:

  • curl -IL https://www.mydomain.tld/server-status
  • curl -IL https://www.mydomain.tld/
Output
HTTP/1.1 401 Unauthorized
. . .
HTTP/1.1 200 OK
. . .
/var/log/apache2/mydomain-access.log
10.10.0.2 - - [08/Jul/2017:23:43:53 -0400] "HEAD /server-status HTTP/1.1" 401 4449 "-" "curl/7.47.0"
10.10.0.2 - - [09/Jul/2017:00:06:59 -0400] "HEAD / HTTP/1.1" 200 4583 "-" "curl/7.47.0"
10.10.0.66 - - [09/Jul/2017:00:06:59 -0400] "GET /s/img/logo.png HTTP/1.1" 200 27233  "-" "curl/7.47.0"
. . .

Probably irrelevant, but FYI:

  • The HTTP 401 is expected, as I have yet to allow access from 10.10.0.2 .
  • The Apache server is actually proxying requests to a uwsgi instance.
    • uwsgi is listening on a domain socket and serving a django project.

Its not just Apache and HTTP:

  • After shutting down Apache and related HTTP services, I can still ping 10.10.0.2.
  • After inspecting some other log files, I noticed that ssh logins to the floating-ip-mapped hostname also appear to come from 10.10.0.2.

So, my first thought was that 10.10.0.2 is doing SNAT on traffic destined for my anchor IP

If that is true, it raises the question, "Is it even possible to safely control access based on IP addresses to hosts which resolve to a floating IP?"

The only problem with that hypothesis is that 10.10.0.2 only ever appears to make the request in two situations:

  1. HTTP/HTTPS requests initiated from the droplet itself
  2. SSH logins to the hostname in question from any remote address

All other connections, so far as I can tell, appear to come from the actual IP address of the requesting client.

If someone knows what actually is going on here, please let me know.

6 comments
  • @dwilkin Hi Darian, this is a very interesting question - can you help with an answer?

  • @hansen Happy to! Looking into this now, have to do some digging to figure out what may be causing this. Thanks for your patience in the meantime!

    Regards,
    Darian
    Platform Support Advocate
    Don't forget to check out our fantastic community articles!
    https://www.digitalocean.com/community

  • @hansen @rdkaiser Thanks for your patience on this, I was able to talk with our networking team and get an explanation!

    In short: When a Droplet tries to use its public IP to talk to its own Floating IP (FLIP), we NAT the destination IP (the FLIP) to the anchor IP, and source it from the anchor IP's gateway +1 before sending it back to the original Droplet. This is a "meta" address that represents the public IP of the Droplet, and in each case you see this you can pretend that address is actually your Droplet's public IP.

    More details: When your Droplet uses its public IP to talk to its Floating IP, there's a bit of NAT routing that goes on to make sure everything goes where it's supposed to. Here's a rundown of the steps:

    1) A Droplet sends a packet via its public IP to its Floating IP.

    2) The packet egresses through eth0, sources from the Droplet's public IP to the Droplet's Floating IP.

    While this would work, it would mean the packet would have to go through normal routing to get to the FLIP, when we can trivially verify that it's going back to the same place. (This is my understanding of the reasoning, and may not be exactly correct)

    3) Instead, when the frame reaches OVS, we have NAT rules in place to shortcut that routing. We NAT the destination IP to the anchor IP and source it from the "meta" address (the anchor's gateway +1). We then send the packet right back to the original Droplet.

    4) The original Droplet sees the ingress as destined for the anchor IP, and replies with a source of the anchor IP and a dest of the meta address.

    5) When the reply reaches OVS, the source IP is NATed to the FLIP and the destination is NATed to the public IP. The packet is once again sent back to the original Droplet.

    6) Profit! Now the Droplet can talk to itself over its FLIP without going through actual routing in the datacenter, which reduces overall congestion and slightly improves speeds for those specific connections.

    The upshot: If your Droplet is talking to itself between its public and Floating IP, you'll see traffic coming in over the meta address (in your case 10.10.0.2), but it will differ for others). This doesn't mean your traffic is actually being routed somewhere else or that someone else is accessing it, it's just a bit of routing magic to reduce what's effectively loopback traffic in the datacenter.

    If you configure your applications to use the loopback address when talking to the FLIP, or to not egress through eth0 when talking to the FLIP, then you won't see any of these connections. Otherwise, you may want to allow traffic from this address when configuring your firewall to ensure your Droplet can communicate with itself!

    I hope that answers your questions, but I'd love to hear if you still have concerns!

    Regards,
    Darian
    Platform Support Advocate

  • That's what I needed to know! Thanks @dwilkin! And @hansen too!

  • @dwilkin Awesome writeup Darian - thank you!

  • You're very welcome, and thanks for pinging me on this @hansen!

2 Answers
rdkaiser July 30, 2017
Accepted Answer

This answer is simply so I can mark an answer as accepted. All the credit goes to hansen and dwilkin's comment above.

see dwilkin's comment

Your question is really difficult. Maybe you need to use some tools to check information about your IP address or domain of this site. I often use the service IP whois
it can helps me to find out more information about needed me IP adress or efficiency of proxy or VPN software.

  • No. I think you're misunderstanding the issue here. It's fundamentally a question about DigitalOcean's internal implementation of floating IPs.

    • It's a good question, and I can tell you for sure that we're not routing traffic to your FLIP through anything other than your Anchor IP, but I'm not sure what this 10.10.0.2 IP is at this time. My inclination is that it's just some sort of internal scanner we're using to gather metrics, but I'm reaching out to the internal teams responsible for confirmation. Thanks for your patience in the meantime!

      Regards,
      Darian
      Platform Support Advocate
      Don't forget to check out our fantastic community articles!
      https://www.digitalocean.com/community

Have another answer? Share your knowledge.