Question

Another (but different ) "502 Bad Gateway" error. (Using Nginx + Django + Gunicorn)

First of all I have to say I am still a newbie and I apologize in advance in case the problem is a trivial one, but I have already spent 10 days and I can not figure out what is the problem.

Basically, on the website I am trying to build the user uploads a photo and the website returns some output. However, sometimes it gives the error: “502 Bad Gateway nginx/1.10.3 (Ubuntu)”.

More specifically, the error is:

[error] 1715#1715: *6 upstream prematurely closed connection while reading response header from upstream, client: 80.181.65.234, server: _, request: "POST / HTTP/1.1", upstream: "http://unix:/home/django/gunicorn.socket:/", host: "dermaai.com", referrer: "http://dermaai.com/"

What is ‘interesting’ is that the error does not always comes up: with the same image sometimes it works (giving out the intended result) and some others it does not!

As the computations made on the photo are not light (Neural Networks stuff), I first though the problem was timeout after 30s. However I think I have solved that cause. In fact, now the error might pop up earlier than after 30 seconds.

I don’t even think the problem is that the “ALLOWED_HOSTS”, as I have tried all the different combinations and nothing much has changed.

I had the exact same problem before registering the domain I am now using ( which for example in my case was 165.227.170.231), but not when I was testing the website only locally (I mean 127.0.0.1:8000).

Here are the settings:

/etc/nginx/sites-available/django :

upstream app_server {
    server unix:/home/django/gunicorn.socket fail_timeout=0;
}

server {
    listen 80 default_server;
    listen [::]:80 default_server ipv6only=on;

    root /usr/share/nginx/html;
    index index.html index.htm;

    client_max_body_size 4G;
    server_name _;

    keepalive_timeout 5;

    # Your Django project's media files - amend as required
    location /media  {
        alias /home/django/django_project/django_project/media;
    }

    # your Django project's static files - amend as required
    location /static {
        alias /home/django/django_project/django_project/static;
    }

    # Proxy the static assests for the Django Admin panel
    location /static/admin {
       alias /usr/lib/python2.7/dist-packages/django/contrib/admin/static/admin/;
    }

    location / {
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header Host $host;
            proxy_redirect off;
            proxy_buffering off;

            proxy_connect_timeout       600;
            proxy_send_timeout          600;
            proxy_read_timeout          600;
            send_timeout                600;

            proxy_pass http://app_server;
    }
    proxy_connect_timeout       600;
    proxy_send_timeout          600;
    proxy_read_timeout          600;
    send_timeout                600;

}

The /etc/nginx/nginx.conf :

user www-data;
worker_processes auto;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;

events {
        worker_connections 768;
        # multi_accept on;
}

http {

        ##
        # Basic Settings
        ##

        sendfile on;
        tcp_nopush on;
        tcp_nodelay on;
        keepalive_timeout 65;
        types_hash_max_size 2048;
        # server_tokens off;

        # server_names_hash_bucket_size 64;
        # server_name_in_redirect off;

        include /etc/nginx/mime.types;
        default_type application/octet-stream;

        ##
        # SSL Settings
        ##

        ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE
        ssl_prefer_server_ciphers on;

        ##
        # Logging Settings
        ##

        access_log /var/log/nginx/access.log;
        error_log /var/log/nginx/error.log;

The /ect/gunicorn.d/gunicorn.py:

"""gunicorn WSGI server configuration."""
from multiprocessing import cpu_count
from os import environ


def max_workers():
    return cpu_count() * 2 + 1

max_requests = 1000
worker_class = 'gevent'
workers = max_workers()

Finally, the settings of django. home/django/django_project/django_project/settings.py:

"""
Django settings for django_project project.

Generated by 'django-admin startproject' using Django 1.8.7.

For more information on this file, see
https://docs.djangoproject.com/en/1.8/topics/settings/

For the full list of settings and their values, see
https://docs.djangoproject.com/en/1.8/ref/settings/
"""

# Build paths inside the project like this: os.path.join(BASE_DIR, ...)
import os

BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))


# Quick-start development settings - unsuitable for production
# See https://docs.djangoproject.com/en/1.8/howto/deployment/checklist/

# SECURITY WARNING: keep the secret key used in production secret!
SECRET_KEY = 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'

# SECURITY WARNING: don't run with debug turned on in production!
DEBUG = False


ALLOWED_HOSTS = ['*']

# Application definition

INSTALLED_APPS = (
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'personal',
)

MIDDLEWARE_CLASSES = (
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.middleware.common.CommonMiddleware',
    'django.middleware.csrf.CsrfViewMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django.contrib.auth.middleware.SessionAuthenticationMiddleware',
    'django.contrib.messages.middleware.MessageMiddleware',
    'django.middleware.clickjacking.XFrameOptionsMiddleware',
    'django.middleware.security.SecurityMiddleware',
)

ROOT_URLCONF = 'django_project.urls'

TEMPLATES = [
    {
        'BACKEND': 'django.template.backends.django.DjangoTemplates',
        'DIRS': [],
        'APP_DIRS': True,
        'OPTIONS': {
            'context_processors': [
                'django.template.context_processors.debug',
                'django.template.context_processors.request',
                'django.contrib.auth.context_processors.auth',
                'django.contrib.messages.context_processors.messages',
            ],
        },
    },
]

WSGI_APPLICATION = 'django_project.wsgi.application'


# Database
# https://docs.djangoproject.com/en/1.8/ref/settings/#databases

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.sqlite3',
        'NAME': os.path.join(BASE_DIR, 'db.sqlite3'),
    }
}


# Internationalization
# https://docs.djangoproject.com/en/1.8/topics/i18n/

LANGUAGE_CODE = 'en-us'

TIME_ZONE = 'UTC'

USE_I18N = True

USE_L10N = True

USE_TZ = True


# Static files (CSS, JavaScript, Images)
# https://docs.djangoproject.com/en/1.8/howto/static-files/
STATIC_ROOT = '/home/django/django_project/django_project/static'
STATIC_URL = '/static/'
# Build paths inside the project like this: os.path.join(BASE_DIR, ...)
# Allow Django from all hosts. This snippet is installed from
# /var/lib/digitalocean/allow_hosts.py

import os
import netifaces

# Find out what the IP addresses are at run time
# This is necessary because otherwise Gunicorn will reject the connections
def ip_addresses():
    ip_list = []
    for interface in netifaces.interfaces():
        addrs = netifaces.ifaddresses(interface)
        for x in (netifaces.AF_INET, netifaces.AF_INET6):
            if x in addrs:
                ip_list.append(addrs[x][0]['addr'])
    return ip_list


# Discover our IP address
#ALLOWED_HOSTS+= ip_addresses() 


And I restart gunicorn in this way:

service gunicorn start --timeout 300 

(As can be seen above, I tried to do everything I found to avoid the timeout problem, and I think that is not the problem anymore.)

Big thanks to whoever gives a try to help a poor newbie! Thanks a lot

Andrea


Submit an answer

This textbox defaults to using Markdown to format your answer.

You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!

Sign In or Sign Up to Answer

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

Want to learn more? Join the DigitalOcean Community!

Join our DigitalOcean community of over a million developers for free! Get help and share knowledge in Q&A, subscribe to topics of interest, and get courses and tools that will help you grow as a developer and scale your project or business.

Very mysterious problems, I can’t tell you the answer but maybe I can help you debug and give you some tips:

View gunicorn logs: By default I think gunicorn is just logging to stderr and stdout logs (viewable in /var/logs/syslog) but it’s much easier to watch if you log to a separate file. Open up /etc/gunicorn.d/gunicorn.py and add a couple lines at the end:

accesslog = "/var/log/gunicorn.log"
errorlog = "/var/log/gunicorn.log"

then create the empty log file and assign it to the django user by running the commands:

touch /var/log/gunicorn.log
chown django:www-data /var/log/gunicorn.log

Restart gunicorn:

service gunicorn restart

then watch the logs in real-time with:

tail -f /var/log/gunicorn.log

Try loading the url that causes the 502, are there any clues in the gunicorn logs when you get the error?

If that doesn’t give you any clues, I think it must be an error in your django app, do you have logging for that?

I am having the same issue. I’ve tried many of the same fixes as you too.

I’d like to add that I have no problems in development, but in production I will (seemingly) randomly get a 502 error after requesting tiles from Google Earth Engine.

There does not seem to be many commonalities between the instances where this operation fails.

Hi andrearamazzina, have to find out the solution? I think I face the exactly same issue. Very appreciate if you could share your solution. Thanks