Question
Why is my droplet still receiving requests after removing it from Load Balancer?
- Posted on October 18, 2019
- Load Balancing
Asked by kevin45cad5f354d24466d9d7e
Hi, we are running three droplets on our Load Balancer. After removing droplet 03-instance from the Load Balancer and tailing the server logs, I can see that there are still requests coming through.
Here are more details:
- Health Check for
03-instance: The health check was still 100% passing while03-instancewas on the Load Balancer (although requests that were hitting it were getting server errors) - Load:
03-instancewas hitting peak load (100%) and we were getting 100% Utilization alerts and some downtime alert pings. The03-instancewas receiving requests until it eventually ran out of memory (after it had been removed from the Load Balancer) - After manually turning off
03-instanceand turning it back on from the Digital Ocean console UI, tailing the server again showed no more requests coming through (it is still removed from the Load Balancer) - Load Balancer configuration:
- Sticky Sessions: Sticky sessions are Off (only recently were they turned off from last week). Is it possible that there are old cookies that are still able to have sticky sessions for the
03-instance? - Algorithm: Algorithm is set to Least Connections (all three instances are configured to have same specs)
- Proxy Protocol:
- Sticky Sessions: Sticky sessions are Off (only recently were they turned off from last week). Is it possible that there are old cookies that are still able to have sticky sessions for the
- Code:
- All three instances are running
dokkuon Node.js. Checking the containers running on03-instanceshowed there was one properly running, as expected. - We are running a library
pm2, which caused us issues recently because we didn’t have our Load Balancer settings configured to Sticky Sessions as Off.
- All three instances are running
As I wasn’t able to find much from searching these forums / Google, any help would be greatly appreciated!
This textbox defaults to using Markdown to format your answer.
You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!
These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.
Want to learn more? Join the DigitalOcean Community!
Join our DigitalOcean community of over a million developers for free! Get help and share knowledge in Q&A, subscribe to topics of interest, and get courses and tools that will help you grow as a developer and scale your project or business.
Hello,
Thanks for the detailed explanation of the problem. One thing that wasn’t super clear is how you configured the LB to track the backend targets, did you provide specific Droplets to target or were you using tags?
If you assigned specific Droplets as targets, the LB will stop sending new connections to that droplet, but won’t kill the existing ones, to avoid disrupting the ongoing traffic. That would explain why it stopped getting traffic after you manually restarted the node, the connections were broken and it didn’t get new requests.
The scenario above is similar if you used tags, but there might be a delay between removing tags from a Droplet and removing it from the LB, which could explain why it would get traffic, but it’s more likely that the situation was due to existing connections.
I hope this clarified the situation, let us know if you have more questions.
Cheers