kamstan6
By:
kamstan6

recv() failed (104: Connection reset by peer) error, gunicorn and nginx and Django

April 6, 2017 8.4k views
Django Nginx Ubuntu

Hi, I am getting the recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 105.178.36.13, server: 188.166.61.158, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:8000/", host: "188.166.61.158 while connecting to my site via the IP address. The nginx error log is printing the above error msg. I am deploying a Django site using nginx and gunicorn. I followed this tutorial here: [https://www.digitalocean.com/community/tutorials/how-to-set-up-django-with-postgres-nginx-and-gunicorn-on-ubuntu-14-04]

I have tried all possible suggestion on stackoverflow, including increasing the keep alive directive in nginx.conf file, client and header buffer sizes but all seems like non is working.

The interesting thing is that when I run gunicorn in the command line it starts and I connect to the site successfully but when I try to run it from the upstart script, I get the above error. I am out of ideas how to fix this.

any directions are highly welcome.

Log In to Comment
2 Answers
hansen April 6, 2017
Accepted Answer

@kamstan6
I think you need to set the environment variable SECRET_KEY into the project, since it won't have access to OS variables, when being executed with upstart.
http://stackoverflow.com/questions/30769075/secret-key-error-with-django-using-uwsgi

Reply
  • kamstan6 April 6, 2017

    Works now, thank you for the lead....
    so is there any other way I can store the sensitive information but no in the project?? the secret keys and other credentials??

    Thank you again

    • jtittle1 April 6, 2017

      @kamstan6

      You can use export from the CLI to store the SECRET_KEY, though it'll only last for that session and it'd need to be used each time the application stops and starts (i.e. if it crashes or you restart the application, you'd need to export again) or a new session is started. The export command doesn't persist.

      There's also .env files which can be used and you would then load them from your app, though that's really no different than relying on a single file that contains all configuration.

      Normally what I do is store configuration to a non-public directory and give the file read-only access. Configuration ranging from secrets to database connection information doesn't belong anywhere in a public directory.

hansen April 6, 2017

Hi @kamstan6

Have you checked if upstart script does exactly the same as your command line?
You can find all upstart stuff in /etc/init/ and /etc/init.d/
Sounds to me that it might have something to do with port numbers or maybe the PID file.

Since you're having problems directly with Gunicorn, then focus on that. It's not a problem in Nginx until you're successfully running Gunicorn.

How do you run Gunicorn command vs upstart?

Reply
  • kamstan6 April 6, 2017

    @hansen Here is how I run the two
    gunicorn command: gunicorn --workers 3 --bind 127.0.0.1:8000 lacolombier_website.wsgi:application --error-logfile /var/log/upstart/gunicorn.log --timeout 120 --log-level debug --keep-alive 5

    upstart sudo service gunicorn start

    here is also my gunicorn.conf file

    description "Gunicorn application server handling myproject"

start on runlevel [2345]
stop on runlevel [!2345]

respawn
setuid lacolombiere_admin
setgid www-data
chdir /home/lacolombiere_admin/www/lacolombiere_live_test.rw/lacolombier_website

exec /home/lacolombiere_admin/www/lacolombiere_live_test.rw/lacolombiere_test/bin/gunicorn --workers 3 --bind 127.0.0.1:8000 lacolombier_website.wsgi:application --error-logfile /var/log/upstart/gunicorn.log --timeout 120 --log-level debug --keep-alive 5

    I appreciate any leads. Thank you

    • hansen April 6, 2017

      @kamstan6
      Let's have a look in the error log:

      tail -50 /var/log/upstart/gunicorn.log
      • kamstan6 April 6, 2017

        here are the log file output

          on_exit: <function on_exit at 0xb6e159cc>
  config: None
  logconfig: None
  check_config: False
  statsd_prefix: 
  secure_scheme_headers: {'X-FORWARDED-PROTOCOL': 'ssl', 'X-FORWARDED-PROTO': 'https', 'X-FORWARDED-SSL': 'on'}
  reload_engine: auto
  proxy_allow_ips: ['127.0.0.1']
  pre_request: <function pre_request at 0xb6e156bc>
  post_request: <function post_request at 0xb6e1572c>
  forwarded_allow_ips: ['127.0.0.1']
  worker_int: <function worker_int at 0xb6e154c4>
  raw_paste_global_conf: []
  threads: 1
  max_requests: 0
  chdir: /home/lacolombiere_admin/www/lacolombiere_live_test.rw/lacolombier_website
  daemon: False
  user: 1000
  limit_request_line: 4094
  access_log_format: %(h)s %(l)s %(u)s %(t)s "%(r)s" %(s)s %(b)s "%(f)s" "%(a)s"
  certfile: None
  on_starting: <function on_starting at 0xb6e150d4>
  post_worker_init: <function post_worker_init at 0xb6e1541c>
  child_exit: <function child_exit at 0xb6e157d4>
  worker_exit: <function worker_exit at 0xb6e1587c>
  paste: None
  default_proc_name: lacolombier_website.wsgi:application
  syslog_addr: udp://localhost:514
  syslog_prefix: None
  ciphers: TLSv1
  worker_abort: <function worker_abort at 0xb6e1556c>
  loglevel: debug
  bind: ['127.0.0.1:8000']
  raw_env: []
  initgroups: False
  capture_output: False
  reload: False
  limit_request_field_size: 8190
  nworkers_changed: <function nworkers_changed at 0xb6e15924>
  timeout: 120
  keyfile: None
  ca_certs: None
  tmp_upload_dir: None
  backlog: 2048
  logger_class: gunicorn.glogging.Logger
[2017-04-06 11:33:42 +0000] [29102] [INFO] Starting gunicorn 19.7.1
[2017-04-06 11:33:42 +0000] [29102] [DEBUG] Arbiter booted
[2017-04-06 11:33:42 +0000] [29102] [INFO] Listening at: http://127.0.0.1:8000 (29102)
[2017-04-06 11:33:42 +0000] [29102] [INFO] Using worker: sync
[2017-04-06 11:33:42 +0000] [29108] [INFO] Booting worker with pid: 29108

        Do you see anything strange, I don't

        • hansen April 6, 2017

          @kamstan6

          No, it looks like the command line version was the last execution of Gunicorn.

          Try running the upstart version and then have a look with cat /var/log/upstart/gunicorn.log - if you find anything you're unsure about, just post it here and let's look at it together.

        • kamstan6 April 6, 2017

          @hansen here is also another output when I run the logs

           [2017-04-06 11:37:41 +0000] [1583] [INFO] Worker exiting (pid: 1583)
[2017-04-06 11:37:41 +0000] [1586] [ERROR] Exception in worker process
Traceback (most recent call last):
  File "/home/lacolombiere_admin/www/lacolombiere_live_test.rw/lacolombiere_test/local/lib/python2.7/site-packages/gunicorn/arbiter.py", line 578, in spawn_worker
    worker.init_process()
  File "/home/lacolombiere_admin/www/lacolombiere_live_test.rw/lacolombiere_test/local/lib/python2.7/site-packages/gunicorn/workers/base.py", line 126, in init_process
    self.load_wsgi()
  File "/home/lacolombiere_admin/www/lacolombiere_live_test.rw/lacolombiere_test/local/lib/python2.7/site-packages/gunicorn/workers/base.py", line 135, in load_wsgi
    self.wsgi = self.app.wsgi()
  File "/home/lacolombiere_admin/www/lacolombiere_live_test.rw/lacolombiere_test/local/lib/python2.7/site-packages/gunicorn/app/base.py", line 67, in wsgi
    self.callable = self.load()
  File "/home/lacolombiere_admin/www/lacolombiere_live_test.rw/lacolombiere_test/local/lib/python2.7/site-packages/gunicorn/app/wsgiapp.py", line 65, in load
    return self.load_wsgiapp()
  File "/home/lacolombiere_admin/www/lacolombiere_live_test.rw/lacolombiere_test/local/lib/python2.7/site-packages/gunicorn/app/wsgiapp.py", line 52, in load_wsgiapp
    return util.import_app(self.app_uri)
  File "/home/lacolombiere_admin/www/lacolombiere_live_test.rw/lacolombiere_test/local/lib/python2.7/site-packages/gunicorn/util.py", line 352, in import_app
    __import__(module)
  File "/home/lacolombiere_admin/www/lacolombiere_live_test.rw/lacolombier_website/lacolombier_website/wsgi.py", line 16, in <module>
    application = get_wsgi_application()
  File "/home/lacolombiere_admin/www/lacolombiere_live_test.rw/lacolombiere_test/local/lib/python2.7/site-packages/django/core/wsgi.py", line 13, in get_wsgi_application
    django.setup(set_prefix=False)
  File "/home/lacolombiere_admin/www/lacolombiere_live_test.rw/lacolombiere_test/local/lib/python2.7/site-packages/django/__init__.py", line 22, in setup
    configure_logging(settings.LOGGING_CONFIG, settings.LOGGING)
  File "/home/lacolombiere_admin/www/lacolombiere_live_test.rw/lacolombiere_test/local/lib/python2.7/site-packages/django/conf/__init__.py", line 53, in __getattr__
    self._setup(name)
  File "/home/lacolombiere_admin/www/lacolombiere_live_test.rw/lacolombiere_test/local/lib/python2.7/site-packages/django/conf/__init__.py", line 41, in _setup
    self._wrapped = Settings(settings_module)
  File "/home/lacolombiere_admin/www/lacolombiere_live_test.rw/lacolombiere_test/local/lib/python2.7/site-packages/django/conf/__init__.py", line 97, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
  File "/home/lacolombiere_admin/www/lacolombiere_live_test.rw/lacolombier_website/lacolombier_website/settings.py", line 23, in <module>
    SECRET_KEY = os.environ["SECRET_KEY"]
  File "/home/lacolombiere_admin/www/lacolombiere_live_test.rw/lacolombiere_test/lib/python2.7/UserDict.py", line 23, in __getitem__
    raise KeyError(key)
KeyError: 'SECRET_KEY'

          I think the error is when I run through upstart it does not get the secretkey (stored in the environmental variables) which normally is set when I activate the virtual environment.
          so how can I set the secret          key and sensitive info's in the environmental variable so that when I run the upstart script, gunicorn gets them.

          when I run gunicorn through the command line the virtual environment is already active.

Have another answer? Share your knowledge.

Related Questions