Socket/Node app memory leak or garbage collection? Or HAproxy/redis issue?
Hi everyone. Looking for some insight or guidance..
[HAProxy/Redis LB server] -> [broker] -> [rabbitmq] all over wildcard SSL
I have a HAproxy/redis loadbalancer pointing at a socket.io/express broker.
This app continues to use more and more MB's of memory, so much that it starts to hiccup, then eventually clog the whole system (even if I cluster the broker layer, if 1 broker clogs/stops it halts all.. which is why I thought it was an HAproxy problem. No major errors in the logs there, but I do see a
ssl missing handshake, but I couldn't tell if thats the problem or a product of the problem? the SSL works fine, up until one of the socket brokers starts to choke. I explored this avenue (trying to fix my SSL cert order, etc) but with no luck.
Pic of my socker servers Memory usage
So I started thinking it was my socket server, because the memory seems to be just going up and up, until the hiccups/stall happens. The drops you see are me restarting the server with
The broker app does not actually fail/crash, it just seems to get clogged.. and it stores all subsequent messages in the memory, even when down.. So I woke up this morning with a queue of messages in memory (i THINK this is what is going on) that all ended up passing through when I restarted the node/broker app.
I would love to be able to clarify my question better than this, but I simply can't right now. Can anyone point me in the right direction? RIght now I am looking into things like
--expose-gc to see if the js garbage collector is the cause. I really have no idea whats going on.
Any help from the community would be greatly appreciated!
- Is it possible for me to inspect my servers memory closer, to find out what exactly is causing it to steadily rise?
- I've tried starting it with
–-nouse-idle-notification.. dosen't seem to affect anything
- Also tried
--expose-gc.. again, no change in server response. Mem steadily climbs.