Another (but different ) "502 Bad Gateway" error. (Using Nginx + Django + Gunicorn)

July 19, 2018 316 views
DigitalOcean Django Nginx

First of all I have to say I am still a newbie and I apologize in advance in case the problem is a trivial one, but I have already spent 10 days and I can not figure out what is the problem.

Basically, on the website I am trying to build the user uploads a photo and the website returns some output.
However, sometimes it gives the error: "502 Bad Gateway nginx/1.10.3 (Ubuntu)".

More specifically, the error is:

[error] 1715#1715: *6 upstream prematurely closed connection while reading response header from upstream, client: 80.181.65.234, server: _, request: "POST / HTTP/1.1", upstream: "http://unix:/home/django/gunicorn.socket:/", host: "dermaai.com", referrer: "http://dermaai.com/"

What is 'interesting' is that the error does not always comes up: with the same image sometimes it works (giving out the intended result) and some others it does not!

As the computations made on the photo are not light (Neural Networks stuff), I first though the problem was timeout after 30s. However I think I have solved that cause. In fact, now the error might pop up earlier than after 30 seconds.

I don't even think the problem is that the "ALLOWED_HOSTS", as I have tried all the different combinations and nothing much has changed.

I had the exact same problem before registering the domain I am now using ( which for example in my case was 165.227.170.231), but not when I was testing the website only locally (I mean 127.0.0.1:8000).

Here are the settings:

/etc/nginx/sites-available/django :

upstream app_server {
    server unix:/home/django/gunicorn.socket fail_timeout=0;
}

server {
    listen 80 default_server;
    listen [::]:80 default_server ipv6only=on;

    root /usr/share/nginx/html;
    index index.html index.htm;

    client_max_body_size 4G;
    server_name _;

    keepalive_timeout 5;

    # Your Django project's media files - amend as required
    location /media  {
        alias /home/django/django_project/django_project/media;
    }

    # your Django project's static files - amend as required
    location /static {
        alias /home/django/django_project/django_project/static;
    }

    # Proxy the static assests for the Django Admin panel
    location /static/admin {
       alias /usr/lib/python2.7/dist-packages/django/contrib/admin/static/admin/;
    }

    location / {
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header Host $host;
            proxy_redirect off;
            proxy_buffering off;

            proxy_connect_timeout       600;
            proxy_send_timeout          600;
            proxy_read_timeout          600;
            send_timeout                600;

            proxy_pass http://app_server;
    }
    proxy_connect_timeout       600;
    proxy_send_timeout          600;
    proxy_read_timeout          600;
    send_timeout                600;

}

The /etc/nginx/nginx.conf :

user www-data;
worker_processes auto;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;

events {
        worker_connections 768;
        # multi_accept on;
}

http {

        ##
        # Basic Settings
        ##

        sendfile on;
        tcp_nopush on;
        tcp_nodelay on;
        keepalive_timeout 65;
        types_hash_max_size 2048;
        # server_tokens off;

        # server_names_hash_bucket_size 64;
        # server_name_in_redirect off;

        include /etc/nginx/mime.types;
        default_type application/octet-stream;

        ##
        # SSL Settings
        ##

        ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE
        ssl_prefer_server_ciphers on;

        ##
        # Logging Settings
        ##

        access_log /var/log/nginx/access.log;
        error_log /var/log/nginx/error.log;

The /ect/gunicorn.d/gunicorn.py:

"""gunicorn WSGI server configuration."""
from multiprocessing import cpu_count
from os import environ


def max_workers():
    return cpu_count() * 2 + 1

max_requests = 1000
worker_class = 'gevent'
workers = max_workers()

Finally, the settings of django.
home/django/djangoproject/djangoproject/settings.py:

"""
Django settings for django_project project.

Generated by 'django-admin startproject' using Django 1.8.7.

For more information on this file, see
https://docs.djangoproject.com/en/1.8/topics/settings/

For the full list of settings and their values, see
https://docs.djangoproject.com/en/1.8/ref/settings/
"""

# Build paths inside the project like this: os.path.join(BASE_DIR, ...)
import os

BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))


# Quick-start development settings - unsuitable for production
# See https://docs.djangoproject.com/en/1.8/howto/deployment/checklist/

# SECURITY WARNING: keep the secret key used in production secret!
SECRET_KEY = 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'

# SECURITY WARNING: don't run with debug turned on in production!
DEBUG = False


ALLOWED_HOSTS = ['*']

# Application definition

INSTALLED_APPS = (
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'personal',
)

MIDDLEWARE_CLASSES = (
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.middleware.common.CommonMiddleware',
    'django.middleware.csrf.CsrfViewMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django.contrib.auth.middleware.SessionAuthenticationMiddleware',
    'django.contrib.messages.middleware.MessageMiddleware',
    'django.middleware.clickjacking.XFrameOptionsMiddleware',
    'django.middleware.security.SecurityMiddleware',
)

ROOT_URLCONF = 'django_project.urls'

TEMPLATES = [
    {
        'BACKEND': 'django.template.backends.django.DjangoTemplates',
        'DIRS': [],
        'APP_DIRS': True,
        'OPTIONS': {
            'context_processors': [
                'django.template.context_processors.debug',
                'django.template.context_processors.request',
                'django.contrib.auth.context_processors.auth',
                'django.contrib.messages.context_processors.messages',
            ],
        },
    },
]

WSGI_APPLICATION = 'django_project.wsgi.application'


# Database
# https://docs.djangoproject.com/en/1.8/ref/settings/#databases

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.sqlite3',
        'NAME': os.path.join(BASE_DIR, 'db.sqlite3'),
    }
}


# Internationalization
# https://docs.djangoproject.com/en/1.8/topics/i18n/

LANGUAGE_CODE = 'en-us'

TIME_ZONE = 'UTC'

USE_I18N = True

USE_L10N = True

USE_TZ = True


# Static files (CSS, JavaScript, Images)
# https://docs.djangoproject.com/en/1.8/howto/static-files/
STATIC_ROOT = '/home/django/django_project/django_project/static'
STATIC_URL = '/static/'
# Build paths inside the project like this: os.path.join(BASE_DIR, ...)
# Allow Django from all hosts. This snippet is installed from
# /var/lib/digitalocean/allow_hosts.py

import os
import netifaces

# Find out what the IP addresses are at run time
# This is necessary because otherwise Gunicorn will reject the connections
def ip_addresses():
    ip_list = []
    for interface in netifaces.interfaces():
        addrs = netifaces.ifaddresses(interface)
        for x in (netifaces.AF_INET, netifaces.AF_INET6):
            if x in addrs:
                ip_list.append(addrs[x][0]['addr'])
    return ip_list


# Discover our IP address
#ALLOWED_HOSTS+= ip_addresses() 


And I restart gunicorn in this way:

service gunicorn start --timeout 300 

(As can be seen above, I tried to do everything I found to avoid the timeout problem, and I think that is not the problem anymore.)

Big thanks to whoever gives a try to help a poor newbie!
Thanks a lot

Andrea

1 Answer

Very mysterious problems, I can't tell you the answer but maybe I can help you debug and give you some tips:

View gunicorn logs: By default I think gunicorn is just logging to stderr and stdout logs (viewable in /var/logs/syslog) but it's much easier to watch if you log to a separate file. Open up /etc/gunicorn.d/gunicorn.py and add a couple lines at the end:

accesslog = "/var/log/gunicorn.log"
errorlog = "/var/log/gunicorn.log"

then create the empty log file and assign it to the django user by running the commands:

touch /var/log/gunicorn.log
chown django:www-data /var/log/gunicorn.log

Restart gunicorn:

service gunicorn restart

then watch the logs in real-time with:

tail -f /var/log/gunicorn.log

Try loading the url that causes the 502, are there any clues in the gunicorn logs when you get the error?

If that doesn't give you any clues, I think it must be an error in your django app, do you have logging for that?

  • First of all thank you so much for the availability and help, really appreciate as I am still struggling with this problem!

    When everything works properly (I get the expected result) I get this message:

     - - [24/Jul/2018:07:04:14 +0000] "POST / HTTP/1.0" 200 4582 "http://dermaai.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36"
    

    While when id does not work I only get:

    [2018-07-24 07:05:31 +0000] [1924] [INFO] Booting worker with pid: 1924
    

    (The number that changes every time)
    Is this anything useful to understand what is not working?

    I will also try to insert some logging calls in the code to check that everything works, but usually when somethings does not work in the code I get a different error (even more when in debugging mode)

    Thanks again,
    Andrea

Have another answer? Share your knowledge.