mzagaja
By:
mzagaja

504 Gateway Timeout with One-Click Rails Install

September 6, 2014 6.8k views

I am attempting to get my rails application to work on the Ubuntu 14.04 one click rails install image. I am following tutorial at https://www.digitalocean.com/community/tutorials/how-to-launch-your-ruby-on-rails-app-with-the-digitalocean-one-click-image. The sample application displays without issue. I developed a sample microposts application following Michael Hartl's Rails Tutorial. I modified the Gemfile to use mysql2 and then changed my database.yml to point to the mysql2 server instead of using the Heroku deploy method. I then uploaded my entire kit and kaboodle to /home/rails as the user root. After following the instructions to do the bundle install, RAILS_ENV=production rake db:migrate, and bundle assets precompile I go to load the rails application and get a 504 gateway timeout. Static pages load without issue (i.e. http://104.131.5.195/404.html) but the front page does not.

If I tail -40 my unicorn log I get:

root@nathan-hale:/home/rails# tail -40 /home/unicorn/log/unicorn.log 
/usr/local/rvm/gems/ruby-2.0.0-p353/gems/bundler-1.3.5/lib/bundler/setup.rb:17:in `<top (required)>'
/usr/local/rvm/rubies/ruby-2.0.0-p353/lib/ruby/site_ruby/2.0.0/rubygems/core_ext/kernel_require.rb:135:in `require'
/usr/local/rvm/rubies/ruby-2.0.0-p353/lib/ruby/site_ruby/2.0.0/rubygems/core_ext/kernel_require.rb:135:in `rescue in require'
/usr/local/rvm/rubies/ruby-2.0.0-p353/lib/ruby/site_ruby/2.0.0/rubygems/core_ext/kernel_require.rb:144:in `require'
/home/rails/config/boot.rb:4:in `<top (required)>'
/usr/local/rvm/rubies/ruby-2.0.0-p353/lib/ruby/site_ruby/2.0.0/rubygems/core_ext/kernel_require.rb:55:in `require'
/usr/local/rvm/rubies/ruby-2.0.0-p353/lib/ruby/site_ruby/2.0.0/rubygems/core_ext/kernel_require.rb:55:in `require'
/home/rails/config/application.rb:1:in `<top (required)>'
/usr/local/rvm/rubies/ruby-2.0.0-p353/lib/ruby/site_ruby/2.0.0/rubygems/core_ext/kernel_require.rb:55:in `require'
/usr/local/rvm/rubies/ruby-2.0.0-p353/lib/ruby/site_ruby/2.0.0/rubygems/core_ext/kernel_require.rb:55:in `require'
/home/rails/config/environment.rb:2:in `<top (required)>'
/usr/local/rvm/rubies/ruby-2.0.0-p353/lib/ruby/site_ruby/2.0.0/rubygems/core_ext/kernel_require.rb:55:in `require'
/usr/local/rvm/rubies/ruby-2.0.0-p353/lib/ruby/site_ruby/2.0.0/rubygems/core_ext/kernel_require.rb:55:in `require'
config.ru:4:in `block in <main>'
/usr/local/rvm/gems/ruby-2.0.0-p353/gems/rack-1.5.2/lib/rack/builder.rb:55:in `instance_eval'
/usr/local/rvm/gems/ruby-2.0.0-p353/gems/rack-1.5.2/lib/rack/builder.rb:55:in `initialize'
config.ru:1:in `new'
config.ru:1:in `<main>'
/usr/local/rvm/gems/ruby-2.0.0-p353/gems/unicorn-4.7.0/lib/unicorn.rb:48:in `eval'
/usr/local/rvm/gems/ruby-2.0.0-p353/gems/unicorn-4.7.0/lib/unicorn.rb:48:in `block in builder'
/usr/local/rvm/gems/ruby-2.0.0-p353/gems/unicorn-4.7.0/lib/unicorn/http_server.rb:750:in `call'
/usr/local/rvm/gems/ruby-2.0.0-p353/gems/unicorn-4.7.0/lib/unicorn/http_server.rb:750:in `build_app!'
/usr/local/rvm/gems/ruby-2.0.0-p353/gems/unicorn-4.7.0/lib/unicorn/http_server.rb:623:in `init_worker_process'
/usr/local/rvm/gems/ruby-2.0.0-p353/gems/unicorn-4.7.0/lib/unicorn/http_server.rb:643:in `worker_loop'
/usr/local/rvm/gems/ruby-2.0.0-p353/gems/unicorn-4.7.0/lib/unicorn/http_server.rb:527:in `spawn_missing_workers'
/usr/local/rvm/gems/ruby-2.0.0-p353/gems/unicorn-4.7.0/lib/unicorn/http_server.rb:538:in `maintain_worker_count'
/usr/local/rvm/gems/ruby-2.0.0-p353/gems/unicorn-4.7.0/lib/unicorn/http_server.rb:303:in `join'
/usr/local/rvm/gems/ruby-2.0.0-p353/gems/unicorn-4.7.0/bin/unicorn:126:in `<top (required)>'
/usr/local/rvm/gems/ruby-2.0.0-p353/bin/unicorn:23:in `load'
/usr/local/rvm/gems/ruby-2.0.0-p353/bin/unicorn:23:in `<main>'
/usr/local/rvm/gems/ruby-2.0.0-p353/bin/ruby_executable_hooks:15:in `eval'
/usr/local/rvm/gems/ruby-2.0.0-p353/bin/ruby_executable_hooks:15:in `<main>'
E, [2014-09-06T23:34:37.382827 #29321] ERROR -- : reaped #<Process::Status: pid 3365 exit 1> worker=1
I, [2014-09-06T23:34:37.383062 #29321]  INFO -- : worker=1 spawning...
E, [2014-09-06T23:34:37.385768 #29321] ERROR -- : reaped #<Process::Status: pid 3367 exit 1> worker=0
I, [2014-09-06T23:34:37.385948 #29321]  INFO -- : worker=0 spawning...
I, [2014-09-06T23:34:37.386960 #3371]  INFO -- : worker=1 spawned pid=3371
I, [2014-09-06T23:34:37.387144 #3371]  INFO -- : Refreshing Gem list
I, [2014-09-06T23:34:37.391896 #3373]  INFO -- : worker=0 spawned pid=3373
I, [2014-09-06T23:34:37.392136 #3373]  INFO -- : Refreshing Gem list

if I trail -f my nginx log I get:

2014/09/06 22:23:54 [error] 690#0: *65 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 70.188.169.36, server: _, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:8080/", host: "104.131.5.195"
2014/09/06 22:31:44 [error] 690#0: *80 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 70.188.169.36, server: _, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:8080/", host: "104.131.5.195"
2014/09/06 22:49:10 [error] 690#0: *84 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 70.188.169.36, server: _, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:8080/", host: "104.131.5.195"
2014/09/06 22:53:03 [error] 690#0: *86 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 70.188.169.36, server: _, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:8080/", host: "104.131.5.195"
2014/09/06 23:11:25 [crit] 13103#0: *1 connect() to unix:/home/rails/tmp/sockets/unicorn.sock failed (2: No such file or directory) while connecting to upstream, client: 70.188.169.36, server: _, request: "GET / HTTP/1.1", upstream: "http://unix:/home/rails/tmp/sockets/unicorn.sock:/", host: "104.131.5.195"
2014/09/06 23:11:27 [crit] 13103#0: *1 connect() to unix:/home/rails/tmp/sockets/unicorn.sock failed (2: No such file or directory) while connecting to upstream, client: 70.188.169.36, server: _, request: "GET / HTTP/1.1", upstream: "http://unix:/home/rails/tmp/sockets/unicorn.sock:/", host: "104.131.5.195"
2014/09/06 23:17:38 [error] 18044#0: *1 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 70.188.169.36, server: _, request: "GET / HTTP/1.1", upstream: "http://unix:/home/rails/tmp/sockets/unicorn.sock/", host: "104.131.5.195"
2014/09/06 23:18:40 [error] 18044#0: *1 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 70.188.169.36, server: _, request: "GET / HTTP/1.1", upstream: "http://unix:/home/rails/tmp/sockets/unicorn.sock/", host: "104.131.5.195"
2014/09/06 23:25:52 [error] 18044#0: *4 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 70.188.169.36, server: _, request: "GET / HTTP/1.1", upstream: "http://unix:/home/rails/tmp/sockets/unicorn.sock/", host: "104.131.5.195"
2014/09/06 23:28:55 [error] 18044#0: *6 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 70.188.169.36, server: _, request: "GET / HTTP/1.1", upstream: "http://unix:/home/rails/tmp/sockets/unicorn.sock/", host: "104.131.5.195"

I tried the following suggestions from other threads: commenting out user: rails in /home/unicorn/unicorn.conf, changing from listening on 127.0.0.1:8080 to using a socket (http://recipes.sinatrarb.com/p/deployment/nginx_proxied_to_unicorn) both of which made no difference.

I am using the default nginx.conf other than the change to the socket now:

user www-data;
worker_processes 4;
pid /var/run/nginx.pid;

events { worker_connections 1024; }

http {
        sendfile on;
        tcp_nopush on;
        tcp_nodelay on;
        keepalive_timeout 65;
        types_hash_max_size 2048;
        server_tokens off;

        # server_names_hash_bucket_size 64;
        # server_name_in_redirect off;

        include /etc/nginx/mime.types;
        default_type application/octet-stream;

        access_log /var/log/nginx/access.log;
        error_log /var/log/nginx/error.log;

        gzip on;
        gzip_disable "msie6";
        gzip_types text/plain text/xml text/css text/comma-separated-values;
        upstream app_server { server unix:/home/rails/tmp/sockets/unicorn.sock  fail_timeout=0; }

        include /etc/nginx/conf.d/*.conf;
        include /etc/nginx/sites-enabled/*;
}

My unicorn.conf is:

user www-data;
worker_processes 4;
pid /var/run/nginx.pid;

events { worker_connections 1024; }

http {
        sendfile on;
        tcp_nopush on;
        tcp_nodelay on;
        keepalive_timeout 65;
        types_hash_max_size 2048;
        server_tokens off;

        # server_names_hash_bucket_size 64;
        # server_name_in_redirect off;

        include /etc/nginx/mime.types;
        default_type application/octet-stream;

        access_log /var/log/nginx/access.log;
        error_log /var/log/nginx/error.log;

        gzip on;
        gzip_disable "msie6";
        gzip_types text/plain text/xml text/css text/comma-separated-values;
        upstream app_server { server unix:/home/rails/tmp/sockets/unicorn.sock  fail_timeout=0; }

        include /etc/nginx/conf.d/*.conf;
        include /etc/nginx/sites-enabled/*;
}

Some other interesting tidbits: New Relic sees my application but seems to believe I am in the development environment. So this leads me to believe I need to fix this somehow. It is unclear what more to do since I have done RAILS_ENV=production and if I echo $RAILS_ENV I get production.

I do not believe this is an nginx problem based on its log, it seems to be unicorn.

7 comments
  • To add additional information this error also occurs when sending up a default rails application that I have not done anything with.

  • You mention that you upload the files as the root user. Are you fixing the file permissions afterwards? sudo chown -R rails:www-data /home/rails

  • Yep:

    root@nathan-hale:/home/rails# ls -al
    total 76
    drwxr-xr-x 13 rails www-data 4096 Sep  7 01:34 .
    drwxr-xr-x  4 root  root     4096 Dec 20  2013 ..
    drwxrwxr-x  8 rails www-data 4096 Sep  7 01:33 app
    drwxrwxr-x  2 rails www-data 4096 Sep  7 01:29 bin
    drwx------  2 rails www-data 4096 Sep  6 22:08 .cache
    drwxrwxr-x  5 rails www-data 4096 Sep  7 01:33 config
    -rw-r--r--  1 rails www-data  154 Sep  7 01:29 config.ru
    drwxrwxr-x  2 rails www-data 4096 Sep  7 01:34 db
    -rw-r--r--  1 rails www-data 1176 Sep  7 01:29 Gemfile
    -rw-r--r--  1 rails www-data 2724 Sep  7 01:29 Gemfile.lock
    -rw-r--r--  1 rails www-data  455 Sep  7 01:29 .gitignore
    drwxrwxr-x  4 rails www-data 4096 Sep  7 01:29 lib
    drwxrwxr-x  2 rails www-data 4096 Sep  7 01:34 log
    drwxrwxr-x  3 rails www-data 4096 Sep  7 01:34 public
    -rw-r--r--  1 rails www-data  249 Sep  7 01:30 Rakefile
    -rw-r--r--  1 rails www-data  478 Sep  7 01:30 README.rdoc
    drwxrwxr-x  8 rails www-data 4096 Sep  7 01:30 test
    drwxrwxr-x  3 rails www-data 4096 Sep  7 01:33 tmp
    drwxrwxr-x  3 rails www-data 4096 Sep  7 01:30 vendor
    
  • Why did you change the app_server to a socket connection instead of leaving it as it was?

    The nginx error that you are receiving is that it's unable to pass off the proxy connection. Can you revert the change, restart unicorn and restart nginx and see if that works for you?

  • I tried changing it mostly to see if it would work. So nginx.conf is now:

    root@nathan-hale:~# cat /etc/nginx/nginx.conf
    user www-data;
    worker_processes 4;
    pid /var/run/nginx.pid;
    
    events { worker_connections 1024; }
    
    http {
        sendfile on;
        tcp_nopush on;
        tcp_nodelay on;
        keepalive_timeout 65;
        types_hash_max_size 2048;
        server_tokens off;
    
        # server_names_hash_bucket_size 64;
        # server_name_in_redirect off;
    
        include /etc/nginx/mime.types;
        default_type application/octet-stream;
    
        access_log /var/log/nginx/access.log;
        error_log /var/log/nginx/error.log;
    
        gzip on;
        gzip_disable "msie6";
        gzip_types text/plain text/xml text/css text/comma-separated-values;
            upstream app_server { server 127.0.0.1:8080  fail_timeout=0; }
    
        include /etc/nginx/conf.d/*.conf;
        include /etc/nginx/sites-enabled/*;
    }
    
    

    and then /home/unicorn/unicorn.conf is:

    listen "127.0.0.1:8080"
    worker_processes 2
    user "rails"
    working_directory "/home/rails"
    pid "/home/unicorn/pids/unicorn.pid"
    stderr_path "/home/unicorn/log/unicorn.log"
    stdout_path "/home/unicorn/log/unicorn.log"
    

    I restarted both things:

    root@nathan-hale:~# service nginx restart
     * Restarting nginx nginx                                                                                                                                                                                 [ OK ] 
    root@nathan-hale:~# service unicorn restart
     * Restarting Unicorn web server unicorn                                                                                                                                                                  [ OK ] 
    

    and I get

    root@nathan-hale:~# tail -f /var/log/nginx/error.log 
    2014/09/08 18:43:25 [error] 5346#0: *29 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 108.29.37.206, server: _, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:8080/", host: "104.131.5.195"
    2014/09/08 20:54:55 [error] 5346#0: *34 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 50.250.56.113, server: _, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:8080/", host: "104.131.5.195"
    2014/09/08 20:55:58 [error] 5346#0: *34 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 50.250.56.113, server: _, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:8080/", host: "104.131.5.195"
    2014/09/09 16:19:33 [error] 5346#0: *42 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 77.207.216.47, server: _, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:8080/", host: "104.131.5.195"
    2014/09/10 00:08:43 [error] 13990#0: *3 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 70.188.169.36, server: _, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:8080/", host: "104.131.5.195"
    

    After further investigation it turns out I didn't tail back far enough in my unicorn log. It started spouting out errors about missing things like activerecord 4.0.0 and railties 4.0.0. I did gem install on each of the specific versions of those gems it complained about missing and now I believe I have gotten it to work. This leads me to believe that the version of unicorn in the one click installer might be a version behind or so of the current version of rails in regards to those dependencies and that's what was causing the issue. I don't have much time to poke at this more tonight so will return to this later in the week and let you know the final outcome of trying to redeploy the sample app.

  • So here is the weird thing about this situation. I re-uploaded my sample application and in spite of having run bundle install I get complaints about missing gems. I can do:

    root@nathan-hale:/home/rails# tail -f /home/unicorn/log/unicorn.log | grep ^E
    E, [2014-09-13T21:56:21.236998 #22263] ERROR -- : reaped #<Process::Status: pid 9125 exit 1> worker=0
    E, [2014-09-13T21:56:21.725576 #9133] ERROR -- : Could not find ffi-1.9.0 in any of the sources (Bundler::GemNotFound)
    E, [2014-09-13T21:56:21.726309 #9131] ERROR -- : Could not find ffi-1.9.0 in any of the sources (Bundler::GemNotFound)
    
    root@nathan-hale:/home/rails# gem list | grep ffi
    ffi (1.9.0)
    

    So it's there but not seeing it maybe. "gem install ffi -v 1.9.0" fixes this issue and it moves on to the next gem. I'm working my way through these and hopefully will have a functional app soon.

  • Ok so I resolved the unicorn issue by manually installing all the gems, which is good. I think the issue was that I setup the rails application with its own gemset with rvm that the systemwide unicorn was accessing only the systemwide gems. So I learned a lot there.

    Now experiencing some ruby/rails growing pains as I'm learning things like the fact that force_ssl was on in production.rb and I didn't have an SSL certificate for the digital ocean server. I am learning that heroku did a lot for me, at least in relation to this Michael Hartl tutorial that I otherwise didn't consider.

1 Answer

This question was answered by @mzagaja:

Ok so I resolved the unicorn issue by manually installing all the gems, which is good. I think the issue was that I setup the rails application with its own gemset with rvm that the systemwide unicorn was accessing only the systemwide gems. So I learned a lot there.

Now experiencing some ruby/rails growing pains as I'm learning things like the fact that force_ssl was on in production.rb and I didn't have an SSL certificate for the digital ocean server. I am learning that heroku did a lot for me, at least in relation to this Michael Hartl tutorial that I otherwise didn't consider.

You can see the comment here.

Have another answer? Share your knowledge.