Why is my droplet still receiving requests after removing it from Load Balancer?

October 18, 2019 1.6k views
Load Balancing
kevin45cad5f354d24466d9d7e
By:
kevin45cad5f354d24466d9d7e

Hi, we are running three droplets on our Load Balancer. After removing droplet 03-instance from the Load Balancer and tailing the server logs, I can see that there are still requests coming through.

Here are more details:

  • Health Check for 03-instance: The health check was still 100% passing while 03-instance was on the Load Balancer (although requests that were hitting it were getting server errors)
  • Load: 03-instance was hitting peak load (100%) and we were getting 100% Utilization alerts and some downtime alert pings. The 03-instance was receiving requests until it eventually ran out of memory (after it had been removed from the Load Balancer)
  • After manually turning off 03-instance and turning it back on from the Digital Ocean console UI, tailing the server again showed no more requests coming through (it is still removed from the Load Balancer)
  • Load Balancer configuration:
    • Sticky Sessions: Sticky sessions are Off (only recently were they turned off from last week). Is it possible that there are old cookies that are still able to have sticky sessions for the 03-instance?
    • Algorithm: Algorithm is set to Least Connections (all three instances are configured to have same specs)
    • Proxy Protocol:
  • Code:
    • All three instances are running dokku on Node.js. Checking the containers running on 03-instance showed there was one properly running, as expected.
    • We are running a library pm2, which caused us issues recently because we didn’t have our Load Balancer settings configured to Sticky Sessions as Off.

As I wasn’t able to find much from searching these forums / Google, any help would be greatly appreciated!

Sign In to Comment
1 Answer
rrosa MOD October 21, 2019

Hello,

Thanks for the detailed explanation of the problem. One thing that wasn’t super clear is how you configured the LB to track the backend targets, did you provide specific Droplets to target or were you using tags?

If you assigned specific Droplets as targets, the LB will stop sending new connections to that droplet, but won’t kill the existing ones, to avoid disrupting the ongoing traffic. That would explain why it stopped getting traffic after you manually restarted the node, the connections were broken and it didn’t get new requests.

The scenario above is similar if you used tags, but there might be a delay between removing tags from a Droplet and removing it from the LB, which could explain why it would get traffic, but it’s more likely that the situation was due to existing connections.

I hope this clarified the situation, let us know if you have more questions.

Cheers

  • dereke93617c50b95b83ac2f85 October 22, 2019

    With or without sticky sessions enabled, the server is still receiving requests after being taken out of the load balancer rotation. This will continue anywhere from 5 to 10 minutes. From my understanding once a response from our server has been sent back to the client the connection is complete and the connection no longer exists within the connection pool.

    For some reason the client (browser) is still able to hit the server. Any idea why that is happening?

    • rrosa MOD October 23, 2019

      Hello

      The situation I describe is not related to sticky sessions, which does steer the traffic to specific but doesn’t affect the lifetime of a connection.

      Most browser connections are not closed when the response come back, that would require a handshake for every single request, and that would be wasteful in most situations, so the connections will remain alive.

      There’s a good explanation of the process here, most case fall under the HTTP 1.1 scenarios: https://en.wikipedia.org/wiki/HTTP_persistent_connection

      If you want to dig more on this issue I would suggest you open a ticket with our support so we can see the specific case, but my first reply is still the baseline explanation.

      Cheers

Have another answer? Share your knowledge.

Related