Question

Why is my droplet still receiving requests after removing it from Load Balancer?

Hi, we are running three droplets on our Load Balancer. After removing droplet 03-instance from the Load Balancer and tailing the server logs, I can see that there are still requests coming through.

Here are more details:

  • Health Check for 03-instance: The health check was still 100% passing while 03-instance was on the Load Balancer (although requests that were hitting it were getting server errors)
  • Load: 03-instance was hitting peak load (100%) and we were getting 100% Utilization alerts and some downtime alert pings. The 03-instance was receiving requests until it eventually ran out of memory (after it had been removed from the Load Balancer)
  • After manually turning off 03-instance and turning it back on from the Digital Ocean console UI, tailing the server again showed no more requests coming through (it is still removed from the Load Balancer)
  • Load Balancer configuration:
    • Sticky Sessions: Sticky sessions are Off (only recently were they turned off from last week). Is it possible that there are old cookies that are still able to have sticky sessions for the 03-instance?
    • Algorithm: Algorithm is set to Least Connections (all three instances are configured to have same specs)
    • Proxy Protocol:
  • Code:
    • All three instances are running dokku on Node.js. Checking the containers running on 03-instance showed there was one properly running, as expected.
    • We are running a library pm2, which caused us issues recently because we didn’t have our Load Balancer settings configured to Sticky Sessions as Off.

As I wasn’t able to find much from searching these forums / Google, any help would be greatly appreciated!


Submit an answer

This textbox defaults to using Markdown to format your answer.

You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!

Sign In or Sign Up to Answer

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

Want to learn more? Join the DigitalOcean Community!

Join our DigitalOcean community of over a million developers for free! Get help and share knowledge in Q&A, subscribe to topics of interest, and get courses and tools that will help you grow as a developer and scale your project or business.

Hello,

Thanks for the detailed explanation of the problem. One thing that wasn’t super clear is how you configured the LB to track the backend targets, did you provide specific Droplets to target or were you using tags?

If you assigned specific Droplets as targets, the LB will stop sending new connections to that droplet, but won’t kill the existing ones, to avoid disrupting the ongoing traffic. That would explain why it stopped getting traffic after you manually restarted the node, the connections were broken and it didn’t get new requests.

The scenario above is similar if you used tags, but there might be a delay between removing tags from a Droplet and removing it from the LB, which could explain why it would get traffic, but it’s more likely that the situation was due to existing connections.

I hope this clarified the situation, let us know if you have more questions.

Cheers