kamstan6
By:
kamstan6

recv() failed (104: Connection reset by peer) error, gunicorn and nginx and Django

April 6, 2017 668 views
Django Nginx Ubuntu

Hi, I am getting the recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 105.178.36.13, server: 188.166.61.158, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:8000/", host: "188.166.61.158 while connecting to my site via the IP address. The nginx error log is printing the above error msg. I am deploying a Django site using nginx and gunicorn. I followed this tutorial here: [https://www.digitalocean.com/community/tutorials/how-to-set-up-django-with-postgres-nginx-and-gunicorn-on-ubuntu-14-04]

I have tried all possible suggestion on stackoverflow, including increasing the keep alive directive in nginx.conf file, client and header buffer sizes but all seems like non is working.

The interesting thing is that when I run gunicorn in the command line it starts and I connect to the site successfully but when I try to run it from the upstart script, I get the above error. I am out of ideas how to fix this.

any directions are highly welcome.

2 Answers
hansen April 6, 2017
Accepted Answer

@kamstan6
I think you need to set the environment variable SECRET_KEY into the project, since it won't have access to OS variables, when being executed with upstart.
http://stackoverflow.com/questions/30769075/secret-key-error-with-django-using-uwsgi

  • Works now, thank you for the lead....
    so is there any other way I can store the sensitive information but no in the project?? the secret keys and other credentials??

    Thank you again

    • @kamstan6

      You can use export from the CLI to store the SECRET_KEY, though it'll only last for that session and it'd need to be used each time the application stops and starts (i.e. if it crashes or you restart the application, you'd need to export again) or a new session is started. The export command doesn't persist.

      There's also .env files which can be used and you would then load them from your app, though that's really no different than relying on a single file that contains all configuration.

      Normally what I do is store configuration to a non-public directory and give the file read-only access. Configuration ranging from secrets to database connection information doesn't belong anywhere in a public directory.

Hi @kamstan6

Have you checked if upstart script does exactly the same as your command line?
You can find all upstart stuff in /etc/init/ and /etc/init.d/
Sounds to me that it might have something to do with port numbers or maybe the PID file.

Since you're having problems directly with Gunicorn, then focus on that. It's not a problem in Nginx until you're successfully running Gunicorn.

How do you run Gunicorn command vs upstart?

  • @hansen Here is how I run the two
    gunicorn command: gunicorn --workers 3 --bind 127.0.0.1:8000 lacolombier_website.wsgi:application --error-logfile /var/log/upstart/gunicorn.log --timeout 120 --log-level debug --keep-alive 5

    upstart sudo service gunicorn start

    here is also my gunicorn.conf file

    description "Gunicorn application server handling myproject"
    
    start on runlevel [2345]
    stop on runlevel [!2345]
    
    respawn
    setuid lacolombiere_admin
    setgid www-data
    chdir /home/lacolombiere_admin/www/lacolombiere_live_test.rw/lacolombier_website
    
    exec /home/lacolombiere_admin/www/lacolombiere_live_test.rw/lacolombiere_test/bin/gunicorn --workers 3 --bind 127.0.0.1:8000 lacolombier_website.wsgi:application --error-logfile /var/log/upstart/gunicorn.log --timeout 120 --log-level debug --keep-alive 5
    

    I appreciate any leads. Thank you

    • @kamstan6
      Let's have a look in the error log:

      tail -50 /var/log/upstart/gunicorn.log
      
      • here are the log file output

          on_exit: <function on_exit at 0xb6e159cc>
          config: None
          logconfig: None
          check_config: False
          statsd_prefix: 
          secure_scheme_headers: {'X-FORWARDED-PROTOCOL': 'ssl', 'X-FORWARDED-PROTO': 'https', 'X-FORWARDED-SSL': 'on'}
          reload_engine: auto
          proxy_allow_ips: ['127.0.0.1']
          pre_request: <function pre_request at 0xb6e156bc>
          post_request: <function post_request at 0xb6e1572c>
          forwarded_allow_ips: ['127.0.0.1']
          worker_int: <function worker_int at 0xb6e154c4>
          raw_paste_global_conf: []
          threads: 1
          max_requests: 0
          chdir: /home/lacolombiere_admin/www/lacolombiere_live_test.rw/lacolombier_website
          daemon: False
          user: 1000
          limit_request_line: 4094
          access_log_format: %(h)s %(l)s %(u)s %(t)s "%(r)s" %(s)s %(b)s "%(f)s" "%(a)s"
          certfile: None
          on_starting: <function on_starting at 0xb6e150d4>
          post_worker_init: <function post_worker_init at 0xb6e1541c>
          child_exit: <function child_exit at 0xb6e157d4>
          worker_exit: <function worker_exit at 0xb6e1587c>
          paste: None
          default_proc_name: lacolombier_website.wsgi:application
          syslog_addr: udp://localhost:514
          syslog_prefix: None
          ciphers: TLSv1
          worker_abort: <function worker_abort at 0xb6e1556c>
          loglevel: debug
          bind: ['127.0.0.1:8000']
          raw_env: []
          initgroups: False
          capture_output: False
          reload: False
          limit_request_field_size: 8190
          nworkers_changed: <function nworkers_changed at 0xb6e15924>
          timeout: 120
          keyfile: None
          ca_certs: None
          tmp_upload_dir: None
          backlog: 2048
          logger_class: gunicorn.glogging.Logger
        [2017-04-06 11:33:42 +0000] [29102] [INFO] Starting gunicorn 19.7.1
        [2017-04-06 11:33:42 +0000] [29102] [DEBUG] Arbiter booted
        [2017-04-06 11:33:42 +0000] [29102] [INFO] Listening at: http://127.0.0.1:8000 (29102)
        [2017-04-06 11:33:42 +0000] [29102] [INFO] Using worker: sync
        [2017-04-06 11:33:42 +0000] [29108] [INFO] Booting worker with pid: 29108
        
        

        Do you see anything strange, I don't

        • @kamstan6

          No, it looks like the command line version was the last execution of Gunicorn.

          Try running the upstart version and then have a look with cat /var/log/upstart/gunicorn.log - if you find anything you're unsure about, just post it here and let's look at it together.

        • @hansen here is also another output when I run the logs

           [2017-04-06 11:37:41 +0000] [1583] [INFO] Worker exiting (pid: 1583)
          [2017-04-06 11:37:41 +0000] [1586] [ERROR] Exception in worker process
          Traceback (most recent call last):
            File "/home/lacolombiere_admin/www/lacolombiere_live_test.rw/lacolombiere_test/local/lib/python2.7/site-packages/gunicorn/arbiter.py", line 578, in spawn_worker
              worker.init_process()
            File "/home/lacolombiere_admin/www/lacolombiere_live_test.rw/lacolombiere_test/local/lib/python2.7/site-packages/gunicorn/workers/base.py", line 126, in init_process
              self.load_wsgi()
            File "/home/lacolombiere_admin/www/lacolombiere_live_test.rw/lacolombiere_test/local/lib/python2.7/site-packages/gunicorn/workers/base.py", line 135, in load_wsgi
              self.wsgi = self.app.wsgi()
            File "/home/lacolombiere_admin/www/lacolombiere_live_test.rw/lacolombiere_test/local/lib/python2.7/site-packages/gunicorn/app/base.py", line 67, in wsgi
              self.callable = self.load()
            File "/home/lacolombiere_admin/www/lacolombiere_live_test.rw/lacolombiere_test/local/lib/python2.7/site-packages/gunicorn/app/wsgiapp.py", line 65, in load
              return self.load_wsgiapp()
            File "/home/lacolombiere_admin/www/lacolombiere_live_test.rw/lacolombiere_test/local/lib/python2.7/site-packages/gunicorn/app/wsgiapp.py", line 52, in load_wsgiapp
              return util.import_app(self.app_uri)
            File "/home/lacolombiere_admin/www/lacolombiere_live_test.rw/lacolombiere_test/local/lib/python2.7/site-packages/gunicorn/util.py", line 352, in import_app
              __import__(module)
            File "/home/lacolombiere_admin/www/lacolombiere_live_test.rw/lacolombier_website/lacolombier_website/wsgi.py", line 16, in <module>
              application = get_wsgi_application()
            File "/home/lacolombiere_admin/www/lacolombiere_live_test.rw/lacolombiere_test/local/lib/python2.7/site-packages/django/core/wsgi.py", line 13, in get_wsgi_application
              django.setup(set_prefix=False)
            File "/home/lacolombiere_admin/www/lacolombiere_live_test.rw/lacolombiere_test/local/lib/python2.7/site-packages/django/__init__.py", line 22, in setup
              configure_logging(settings.LOGGING_CONFIG, settings.LOGGING)
            File "/home/lacolombiere_admin/www/lacolombiere_live_test.rw/lacolombiere_test/local/lib/python2.7/site-packages/django/conf/__init__.py", line 53, in __getattr__
              self._setup(name)
            File "/home/lacolombiere_admin/www/lacolombiere_live_test.rw/lacolombiere_test/local/lib/python2.7/site-packages/django/conf/__init__.py", line 41, in _setup
              self._wrapped = Settings(settings_module)
            File "/home/lacolombiere_admin/www/lacolombiere_live_test.rw/lacolombiere_test/local/lib/python2.7/site-packages/django/conf/__init__.py", line 97, in __init__
              mod = importlib.import_module(self.SETTINGS_MODULE)
            File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
              __import__(name)
            File "/home/lacolombiere_admin/www/lacolombiere_live_test.rw/lacolombier_website/lacolombier_website/settings.py", line 23, in <module>
              SECRET_KEY = os.environ["SECRET_KEY"]
            File "/home/lacolombiere_admin/www/lacolombiere_live_test.rw/lacolombiere_test/lib/python2.7/UserDict.py", line 23, in __getitem__
              raise KeyError(key)
          KeyError: 'SECRET_KEY'
          

          I think the error is when I run through upstart it does not get the secretkey (stored in the environmental variables) which normally is set when I activate the virtual environment.
          so how can I set the secret
          key and sensitive info's in the environmental variable so that when I run the upstart script, gunicorn gets them.

          when I run gunicorn through the command line the virtual environment is already active.

Have another answer? Share your knowledge.