Every 2-4 weeks I get (111: Connection refused) while connecting to upstream. Why?

November 7, 2016 2.4k views
Ruby on Rails Ubuntu
nwalter
By:
nwalter

I have a rails app that uses Nginx and Puma. Every 2-4 weeks my app starts giving 502 errors. When digging into log files, this is the error that comes up

[error] 8353#8353: *38966 connect() to unix:///home/deploy/apps/later_backend/shared/tmp/sockets/later_backend-puma.sock failed (111: Connection refused) while connecting to upstream, client: 69.162.124.234, server: later.bz, request: "GET / HTTP/1.1", upstream: "http://unix:///home/deploy/apps/later_backend/shared/tmp/sockets/later_backend-puma.sock:/", host: "later.bz", referrer: "http://later.bz"

A restart will fix the issue but I’m getting really sick of my app crashing every 2-4 weeks and having to do a manual restart. Amy thoughts on what this could be?

Log In to Comment
1 Answer
xMudrii November 7, 2016
Accepted Answer

What is your Droplet size? It could be that app gets killed due to it ran out of memory.
Syslog could give you more information on that. Open it with your favorite text editor:

  • sudo nano /var/log/syslog

Look at it, and if you see any Out Of Memory entry for your app, it was killed because it ran out of RAM.

In that case Droplet upgrade is recommend. You can add Swap space as alternative to upgrade.
Because it is only after 2-4 weeks, you could have memory leaks somewhere. That means that app is not releasing used memory. This is something you have to find in source code if it’s available.

How To Add Swap Space on Ubuntu 16.04
by Justin Ellingwood
One of the easiest way of increasing the responsiveness of your server and guarding against out of memory errors in your applications is to add some swap space. In this guide, we will cover how to add a swap file to an Ubuntu 16.04 server. <$>[warning] [label...
Reply
  • nwalter November 7, 2016

    Wow, you nailed it. It was a memory issue. 

    Out of memory: Kill process 3486 (ruby) score 428 or sacrifice child

    My app doesn’t get much traffic, so I’d bet that it has a memory leak. Any good ways to debug/find the code causing a memory leak?

    • xMudrii November 7, 2016

      I’m sorry but this is something I can’t help you with. I’m not a Ruby dev so it’s unknown to me.
      As far as I know there is no tutorials on DO for it, but there is many articles that can explain you how to do it. :)

      For example I found this on site samsaffron.com which looks promising to me. :) there are many many others.
      Good luck with it :)

Have another answer? Share your knowledge.

Related Questions