Installed ELk Log Stack gives 502 Bad Gateway

July 26, 2015 12.7k views
Nginx Logging Ubuntu

Hi,

I just installed Elk Log Stack with one click install on Ubuntu 14.04,

When i logged into the ip with the user password as described in:
https://www.digitalocean.com/community/tutorials/how-to-use-the-digitalocean-elk-stack-one-click-application

Then I get a Nginx 502 Bad Gateway, i have following in the /var/log/nginx/error.log

2015/07/26 08:57:40 [error] 934#0: *3 connect() failed (111: Connection refused) while connecting to upstream, client: 37.24.151.202, server: , request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:5601/", host: "46.101.233.103"
2015/07/26 08:57:40 [error] 934#0: *3 connect() failed (111: Connection refused) while connecting to upstream, client: 37.24.151.202, server: , request: "GET /favicon.ico HTTP/1.1", upstream: "http://127.0.0.1:5601/favicon.ico", host: "46.101.233.103", referrer: "http://46.101.233.103/"

I'm new to the log stack and sort of new to nginx, and i expected the one click install to work out of the box.. What can i do to fix this ?

2 comments
  • Good morning,

    If you are still having this issue, which I have been having with more frequency lately likely due to an Elasticsearch issue after increasing load on our server close to four-fold, please give this a try:

    Restart the ELK stack, from Kibana down to Logstash, in order.

    For example:

    1. sudo service kibana4 stop
    2. sudo service nginx stop
    3. sudo service elasticsearch stop
    4. sudo service logstash restart
    5. sudo service elasticsearsh start
    6. sudo service nginx start
    7. sudo service kibana4 start

    Let your services get back to a stable status, especially in the case of Elasticsearch (e.g. all of the nodes are coming back online properly, which can be checked using the HEAD plugin (https://www.elastic.co/guide/en/elasticsearch/reference/1.4/modules-plugins.html#site-plugins)).

    I hope that this helps.

    -- Michael

  • there is a typo in "sudo service elasticsearsh start", should be "sudo service elasticsearch start"

4 Answers

Hi tomasnorre,

I installed also an Elk Log Stack with one click install on Ubuntu 14.04.
But for me it worked fine.
Your error was that the service kibana was not running.
Maybe the droplet deployment of ELK Stack was not ready.
How I found out the error see analysis:

Problem:
2015/07/26 08:57:40 [error] 934#0: *3 connect() failed (111: Connection refused) while connecting to upstream, client: 37.24.151.202, server: , request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:5601/", host: "46.101.233.103"

Analysis:

Nginx trying to connect to localhost:5601

root # grep -r "5601" /etc/nginx/
/etc/nginx/sites-available/kibana:        proxy_pass http://localhost:5601;

I looked into /etc/nginx/sites-available/kibana:

root # cat /etc/nginx/sites-available/kibana 
server {
    listen 80;

    # server_name kibana.myhost.org;

    auth_basic "Restricted Access";
    auth_basic_user_file /etc/nginx/conf.d/kibana.htpasswd;

    location / {
        proxy_pass http://localhost:5601;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade;
    }
}

Nginx using a Reverse Proxy

To reproduce your error i stopped the kibana service

root # service kibana stop

Result:
502 Bad Gateway

Solution:

#check if kibana is running
root # service kibana status
root # netstat -pant | grep 5601
tcp        0      0 127.0.0.1:5601          0.0.0.0:*               LISTEN      2227/node
root # ps -ef | grep kibana | grep -v "grep "
kibana    2227     1  0 10:04 ?        00:00:01 /opt/kibana/bin/../node/bin/node /opt/kibana/bin/../src/bin/kibana.js

#if the service is not running 
root # service kibana start 

I hope this helped you.

Best regards,
MSahl

  • Thanks a lot for your detailed answer, unfortunately it didn't solve my problem.

    The kibana is running:

    root # service kibana status
    kibana start/running, process 672

    But my netstat don't return anything:

    root # netstat -pant | grep 5601

    root # ps -ef | grep kibana | grep -v "grep "
    kibana 672 1 0 11:30 ? 00:00:02 /opt/kibana/bin/../node/bin/node /opt/kibana/bin/../src/bin/kibana.js

    So everything but netstat looks good to me with respect to your advices.

That's strange,

I tried for the fun of it, to boot a new droplet in Frankfurt (1) still exact same result. Then i tried to boot on in New York (3) that worked.

Perhaps there are some problems at Frankfurt hosting right now.

  • Very strange,

    I also tried a new droplet in Frankfurt 1 again and that worked.
    Perhaps there are some temporarily problems at Frankfurt hosting.

I've been seeing this issue pop up a couple of times with the pre-installed ELK stack.
The CPU suddenly spikes and Kibana is bricked. Restarting Kibana should work but i've found that it doesn't always fix it immediately.

Check your kibana.yml file.

the server.host should be the same as what in the proxy_pass

nginx config
proxy_pass http://localhost:5601;

kibana.yml
server.host: "localhost"

after restart it should fix the bad gateway issue.

Have another answer? Share your knowledge.