Share

Question

Hi,

I have my website running on RoR 4.0.2 and have a connection pool of capacity as 10 for my production server. On and off I get this weird error of timeout ActiveRecord::ConnectionTimeoutError on my rails app and the website hangs. I have to restart my server to get everything back to normal.

I am running unicorn to fork the Rails server.

Any help will be really appreciated.

Error:

ActiveRecord::ConnectionTimeoutError (could not obtain a database connection within 5.000 seconds (waited 5.000 seconds)):
  activerecord (4.0.2) lib/active_record/connection_adapters/abstract/connection_pool.rb:190:in `block in wait_poll'
  activerecord (4.0.2) lib/active_record/connection_adapters/abstract/connection_pool.rb:181:in `loop'
  activerecord (4.0.2) lib/active_record/connection_adapters/abstract/connection_pool.rb:181:in `wait_poll'
  activerecord (4.0.2) lib/active_record/connection_adapters/abstract/connection_pool.rb:136:in `block in poll'
  /usr/local/rvm/rubies/ruby-2.0.0-p353/lib/ruby/2.0.0/monitor.rb:211:in `mon_synchronize'
  activerecord (4.0.2) lib/active_record/connection_adapters/abstract/connection_pool.rb:146:in `synchronize'
  activerecord (4.0.2) lib/active_record/connection_adapters/abstract/connection_pool.rb:134:in `poll'
  activerecord (4.0.2) lib/active_record/connection_adapters/abstract/connection_pool.rb:423:in `acquire_connection'
  activerecord (4.0.2) lib/active_record/connection_adapters/abstract/connection_pool.rb:356:in `block in checkout'
  /usr/local/rvm/rubies/ruby-2.0.0-p353/lib/ruby/2.0.0/monitor.rb:211:in `mon_synchronize'
  activerecord (4.0.2) lib/active_record/connection_adapters/abstract/connection_pool.rb:355:in `checkout'
  activerecord (4.0.2) lib/active_record/connection_adapters/abstract/connection_pool.rb:265:in `block in connection'
  /usr/local/rvm/rubies/ruby-2.0.0-p353/lib/ruby/2.0.0/monitor.rb:211:in `mon_synchronize'
  activerecord (4.0.2) lib/active_record/connection_adapters/abstract/connection_pool.rb:264:in `connection'
  activerecord (4.0.2) lib/active_record/connection_adapters/abstract/connection_pool.rb:546:in `retrieve_connection'
  activerecord (4.0.2) lib/active_record/connection_handling.rb:79:in `retrieve_connection'
  activerecord (4.0.2) lib/active_record/connection_handling.rb:53:in `connection'
  activerecord (4.0.2) lib/active_record/query_cache.rb:51:in `restore_query_cache_settings'
  activerecord (4.0.2) lib/active_record/query_cache.rb:43:in `rescue in call'
  activerecord (4.0.2) lib/active_record/query_cache.rb:32:in `call'
  activerecord (4.0.2) lib/active_record/connection_adapters/abstract/connection_pool.rb:626:in `call'
  actionpack (4.0.2) lib/action_dispatch/middleware/callbacks.rb:29:in `block in call'
  activesupport (4.0.2) lib/active_support/callbacks.rb:373:in `_run__2161530995940529175__call__callbacks'
  activesupport (4.0.2) lib/active_support/callbacks.rb:80:in `run_callbacks'
  actionpack (4.0.2) lib/action_dispatch/middleware/callbacks.rb:27:in `call'
  actionpack (4.0.2) lib/action_dispatch/middleware/remote_ip.rb:76:in `call'
  actionpack (4.0.2) lib/action_dispatch/middleware/debug_exceptions.rb:17:in `call'
  actionpack (4.0.2) lib/action_dispatch/middleware/show_exceptions.rb:30:in `call'
  railties (4.0.2) lib/rails/rack/logger.rb:38:in `call_app'
  railties (4.0.2) lib/rails/rack/logger.rb:20:in `block in call'
  activesupport (4.0.2) lib/active_support/tagged_logging.rb:67:in `block in tagged'
  activesupport (4.0.2) lib/active_support/tagged_logging.rb:25:in `tagged'
  activesupport (4.0.2) lib/active_support/tagged_logging.rb:67:in `tagged'
  railties (4.0.2) lib/rails/rack/logger.rb:20:in `call'
  actionpack (4.0.2) lib/action_dispatch/middleware/request_id.rb:21:in `call'
  rack (1.5.2) lib/rack/methodoverride.rb:21:in `call'
  rack (1.5.2) lib/rack/runtime.rb:17:in `call'
  actionpack (4.0.2) lib/action_dispatch/middleware/static.rb:64:in `call'
  rack (1.5.2) lib/rack/sendfile.rb:112:in `call'
  railties (4.0.2) lib/rails/engine.rb:511:in `call'
  railties (4.0.2) lib/rails/application.rb:97:in `call'
  activerecord (4.0.2) lib/active_record/connection_adapters/abstract/connection_pool.rb:626:in `call'
  unicorn (4.7.0) lib/unicorn/http_server.rb:580:in `process_client'
  unicorn (4.7.0) lib/unicorn/http_server.rb:660:in `worker_loop'
  unicorn (4.7.0) lib/unicorn/http_server.rb:527:in `spawn_missing_workers'
  unicorn (4.7.0) lib/unicorn/http_server.rb:153:in `start'
  unicorn (4.7.0) bin/unicorn:126:in `<top (required)>'
  /usr/local/rvm/gems/ruby-2.0.0-p353/bin/unicorn:23:in `load'
  /usr/local/rvm/gems/ruby-2.0.0-p353/bin/unicorn:23:in `<main>'
  /usr/local/rvm/gems/ruby-2.0.0-p353/bin/ruby_executable_hooks:15:in `eval'
  /usr/local/rvm/gems/ruby-2.0.0-p353/bin/ruby_executable_hooks:15:in `<main>'

Answer 1

asb MOD October 16, 2014

The ActiveRecord::ConnectionTimeoutError error usually means that the database can not can’t accept any new connection connections as it is maxed out. As you are using Unicorn, you need to consider the number of connections that each forked worker process will attempt to make. If you have four Unicorn workers processes, they all may be trying to make 10 connections each.

If you post your database.yml and Unicorn configuration, we might be able to help further.

Report

syedsanahassan October 16, 2014
I am attaching the production part of database.yml. I have 3 worker processes under unicorn.conf and also I have updated my connection pool from 10 –> 30.

production: adapter: mysql database: database username: username password: notapassword host: localhost pool: 30 socket: /var/run/mysqld/mysqld.sock
Report
syedsanahassan December 19, 2014

@asb Is there a good documentation where I can get this error fixed appropriately. I increase the pool and after a couple of weeks when my users have increased I get to see this again.
Report
asb MOD December 19, 2014

@syedsanahassan This article in the Heroku docs is pretty helpful: Concurrency and Database Connections in Ruby with ActiveRecord
Report

Answer 2

syedsanahassan October 16, 2014
I am attaching the production part of database.yml. I have 3 worker processes under unicorn.conf and also I have updated my connection pool from 10 –> 30.

production: adapter: mysql database: database username: username password: notapassword host: localhost pool: 30 socket: /var/run/mysqld/mysqld.sock
Report
syedsanahassan December 19, 2014

@asb Is there a good documentation where I can get this error fixed appropriately. I increase the pool and after a couple of weeks when my users have increased I get to see this again.
Report
asb MOD December 19, 2014

@syedsanahassan This article in the Heroku docs is pretty helpful: Concurrency and Database Connections in Ruby with ActiveRecord
Report

Answer 3

syedsanahassan December 19, 2014

@asb Thanks for the reply I did read it, but the post on https://bibwild.wordpress.com/2014/07/17/activerecord-concurrency-in-rails4-avoid-leaked-connections/ was really helpful.

For viewers who come here from Google search (coz DO posts are high in content value :P), I was using Thread.new in many locations where I was trying to archive data that was flowing out and therefore I opened a lot of connections and never used the rails directive with_connection when trying to make a db connection and therefore AR didn’t know how to check back the connection into the pool and I exhausted the pool with my n00bness.

Report

asb MOD December 19, 2014

Thanks for sharing that post and your solution!
Report

Answer 4

asb MOD December 19, 2014

Thanks for sharing that post and your solution!
Report

Answer 5

codeminator February 16, 2015

Are you using Threads in your app dealing with the database? if so, you will have to wrap the Thread code like this

begin
#your code here
ensure
#guarantee that the thread is releasing the DB connection after it is done
ActiveRecord::Base.connectionpool.releaseconnection
end

Report

Rails app getting a ActiveRecord::ConnectionTimeoutError timeout error

Leave a Comment

Share

Share your Question

Rails app getting a ActiveRecord::ConnectionTimeoutError timeout error

Leave a Comment

Almost there!