Question

Ghost droplet 502s on large images

Posted February 22, 2017 2.5k views
Ubuntu Ghost

My ghost blog works great on most articles but on one high traffic article with large images, it 502’s when it has more than 16 ish large screenshots.

There is no error log. It’s all very strange. I thought it might be just me, but it goes down everywhere.

Add a comment
rawsh
By:
rawsh
Add a comment

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

3 answers
hansen February 22, 2017

Are you using Nginx as proxy, so it’s serving the files directly or is Nodejs handling everything? It sounds like your Nodejs is crashing or stop accepting more connections.
Are you sure there nothing logged anywhere?

Reply Report
  • rawsh February 22, 2017

    I looked at the ghost logs and nothing showed except for the 502 message. Nginx is used, and I serve my files statically with

    server {
    listen 80 default_server;
    listen [::]:80 default_server ipv6only=on;

    server_name www.devpy.me;

    root /usr/share/nginx/html;
    index index.html index.htm;

    client_max_body_size 10G;

    location / {
        proxy_pass http://localhost:2368;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $http_host;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_buffering off;
    }

    location /content/images {
        alias /var/www/ghost/content/images;
    }

    location /assets {
        alias /var/www/ghost/content/themes/devpy/assets;
    }

    location /public {
        alias /var/www/ghost/core/built/public;
    }

    location /ghost/scripts {
        alias /var/www/ghost/core/built/scripts;
    }
}
    Reply Report
    • hansen February 22, 2017

      First things first. This shouldn’t be needed at all - client_max_body_size 10G; - at least not 10 GigaBytes. I highly doubt your droplet has 10GB of memory, which would be required to serve that size. You should be able to lower that to the default amount by removing it - otherwise use M instead, which means MegaBytes.

      Are you using forever or PM2 or SuperVisor to control your Nodejs apps? There you can find the log file.
      The reason why you’re getting 502 error is because Nginx has it’s connection with Nodejs. So it sounds like Nodejs/Ghost is crashing somehow.

      Reply Report
      • rawsh February 22, 2017

        I am using the default ghost image on DO, so service ghost.

        I removed the body_size but I still get crashes.

        Thanks for the reply though.

        Reply Report
        • hansen February 22, 2017

          But when Ghost or Nodejs goes away, do you need to manually restart it? Or does it automatically come back after X seconds?

          Try to replace your location / section with this - and restart nginx:

          location / {
  proxy_set_header X-Real-IP $remote_addr;
  proxy_set_header HOST $http_host;
  proxy_set_header X-NginX-Proxy true;
  proxy_pass http://127.0.0.1:2368;
  proxy_redirect off;
}
          Reply Report
          • rawsh February 22, 2017

            So I tried this and cleaning the cache and neither worked. I did figure out the log, however. By default there is not log, I had to start ghost with npm start --production >> /var/log/ghost.log. Here is what I got:

            Ghost is running in production...                                                                                                                                                                                                         
Your blog is now available on http://www.devpy.me                                                                                                                                                                                         
Ctrl+C to shut down                                                                                                                                                                                                                       
2620:0101:4035:0324:0150:0070:0173:0010 - - [22/Feb/2017:23:47:49 +0000] "GET /shared/ghost-url.min.js?v=7a490aa7ac HTTP/1.0" 200 759 "-" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0)"                               
2a03:2880:1020:8fea:face:b00c:0000:8000 - - [22/Feb/2017:23:47:55 +0000] "GET /rss HTTP/1.0" 301 - "-" "facebookexternalhit/1.1"                                                                                                          
2a03:2880:1020:8fe6:face:b00c:0000:8000 - - [22/Feb/2017:23:47:56 +0000] "GET /rss/ HTTP/1.0" 200 - "-" "facebookexternalhit/1.1"                                                                                                         
177.148.157.160 - - [22/Feb/2017:23:48:09 +0000] "GET /your-guide-to-a-practical-linux-desktop-with-i3wm/ HTTP/1.0" 200 - "http://www.devpy.me/your-guide-to-a-comfortable-linux-desktop-with-openbox/" "Mozilla/5.0 (X11; Linux i586; rv:
31.0) Gecko/20100101 Firefox/31.0"                                                                                                                                                                                                        
177.148.157.160 - - [22/Feb/2017:23:48:10 +0000] "GET /shared/ghost-url.min.js?v=0219c4a54e HTTP/1.0" 200 759 "http://www.devpy.me/your-guide-to-a-practical-linux-desktop-with-i3wm/" "Mozilla/5.0 (X11; Linux i586; rv:31.0) Gecko/20100
101 Firefox/31.0"                                                                                                                                                                                                                         
105.100.33.199 - - [22/Feb/2017:23:48:14 +0000] "GET /5-of-the-best-linux-desktop-themes-compared/ HTTP/1.0" 200 - "http://www.devpy.me/10-of-the-best-linux-themes-compared/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:47.0) Gecko/201
00101 Firefox/47.0"                                                                                                                                                                                                                       
105.100.33.199 - - [22/Feb/2017:23:48:15 +0000] "GET /shared/ghost-url.min.js?v=0219c4a54e HTTP/1.0" 200 759 "http://www.devpy.me/5-of-the-best-linux-desktop-themes-compared/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:47.0) Gecko/20
100101 Firefox/47.0"                                                                                                                                                                                                                      
108.226.178.213 - - [22/Feb/2017:23:48:15 +0000] "GET /10-of-the-best-linux-themes-compared/ HTTP/1.0" 200 - "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36"               
2620:0101:4035:0324:0150:0070:0173:0008 - - [22/Feb/2017:23:48:19 +0000] "GET /p/cd55f66b-754b-4509-9268-1281bf38174e/ HTTP/1.0" 301 72 "-" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0)"                             
2620:0101:4035:0324:0150:0070:0173:0008 - - [22/Feb/2017:23:48:20 +0000] "GET /10-of-the-best-linux-themes-compared/ HTTP/1.0" 200 - "-" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0)"                                
108.226.178.213 - - [22/Feb/2017:23:48:23 +0000] "GET / HTTP/1.0" 200 - "http://www.devpy.me/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36"                                 
108.226.178.213 - - [22/Feb/2017:23:48:23 +0000] "GET /shared/ghost-url.min.js?v=0219c4a54e HTTP/1.0" 200 759 "http://www.devpy.me/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/53
7.36"                                                                                                                                                                                                                                     
108.226.178.213 - - [22/Feb/2017:23:48:25 +0000] "GET /building-a-python-music-indicating-start-page-2/ HTTP/1.0" 200 - "http://www.devpy.me/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87
 Safari/537.36"                                                                                                                                                                                                                           
108.226.178.213 - - [22/Feb/2017:23:48:26 +0000] "GET / HTTP/1.0" 200 - "http://www.devpy.me/building-a-python-music-indicating-start-page-2/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87
 Safari/537.36"                                                                                                                                                                                                                           
108.226.178.213 - - [22/Feb/2017:23:48:27 +0000] "GET /10-of-the-best-linux-themes-compared/ HTTP/1.0" 200 - "http://www.devpy.me/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537
.36"                                                                                                                                                                                                                                      

Ghost has shut down                                                                                                                                                                                                                       

Your blog is now offline

            The problem url is http://www.devpy.me/10-of-the-best-linux-themes-compared/. There seems to be no error whatsoever…

            I looked at the nginx log as well, no mention of a 502. This is starting to get strange. What I will try now is upgrading the droplet to the 1gb ram version, it’s supposed to run fine on the cheaper droplet but whatever. I will also try restarting.

            Reply Report
        • hansen February 22, 2017

          Try cleaning the Nodejs cache with npm cache clean.
          I’m not sure where Nodejs or Ghost puts it’s error logs, but you need to find them. Perhaps try to ls -ltr /var/log/ to list all log files after last edit time.

          Reply Report
hansen February 23, 2017

Sorry for starting a new answer, but this forum makes it difficult to answer the sub-thread you posted.
Well, it’s interesting, because I have been visiting exactly that page (could see your domain in the config) many, many times a couple of hours ago - and did none-cache browser refresh. I could not get it to crash.
It looks like it’s just an access log - not the error log, which we could have used for debugging.
But don’t you use forever, pm2 or supervisor to control Ghost? Otherwise, how does Nodejs and Ghost automatically start, when you restart your server?

Reply Report
  • rawsh February 23, 2017

    I believe it’s upstart. And yes, I stopped the page from being down by removing images; if I add them back in it 502’s.

    Reply Report
    • hansen February 23, 2017

      Okay, didn’t know upstart could be used for Node+apps.
      Which version of Ubuntu, Node and Ghost are you using? If it’s up-to-date, then I’m pretty much at a loss and would suggest you try https://slack.ghost.org/ where they know a lot more about Ghost and probably where to find the error logs too.
      But please just come back and write the solution, when you get it, so others can learn from this thread.

      Reply Report
jtittle1 February 24, 2017

@rawsh

You should be able to run tail on the NGINX error log to see what’s being logged:

tail -20 /var/log/nginx/error.log

If nothing is being logged to that file, my first guess would be that the Ghost instance is crashing and since NGINX is unable to proxy the request to the Ghost instance (due to it crashing), it throws 502.

If the app is indeed crashing, it’d most likely due to OOM (Out of Memory). This can happen on 512M droplets where resources are far more constrained than they would be on say, a 1GB or 2GB.

If you can post the output of:

free -mh

That’ll give us a better idea of what’s being used, cached, etc when it comes to RAM.

You can also use top to monitor usage. To sort top, you can use SHIFT+M, c, then e which will sort processes by usage, expand the process commands, and show it in human-readable format. Those are keystrokes you’d use after running top.

Also, for the sake of looking at NGINX’s configuration, can you post the contents of:

/etc/nginx/nginx.conf
Reply Report
Submit an Answer

Related

Looking for something else?

Ask a new question Search for more help