How To Optimize Unicorn Workers in a Ruby on Rails App
If you are a Rails developer, you've probably heard of Unicorn, a HTTP server that can handle multiple requests concurrently.
Unicorn uses forked processes to achieve concurrency. Since forked processes are essentially copies of each other, this means that the Rails application need not be thread safe.
This is great because it is difficult to ensure that our own code is thread safe. If we cannot ensure that our code is thread safe, then concurrent web servers such as Puma and even alternative Ruby implementations that exploit concurrency and parallelism such as JRuby and Rubinius would be out of the question.
Therefore, Unicorn gives our Rails apps concurrency even when they are not thread safe. However, this comes at a cost. Rails apps running on Unicorn tend to consume much more memory. Without paying any heed to the memory consumption of your app, you may well find yourself with an overburdened cloud server.
In this article, we will explore a few ways to exploit Unicorn's concurrency, while at the same time control the memory consumption.
Use Ruby 2.0!
If you are using Ruby 1.9, you should seriously consider switching to Ruby 2.0. To understand why, we need to understand a little bit about forking.
Forking and Copy-on-Write (CoW)
When a child process is forked, it is the exact same copy as the parent process. However, the actual physical memory copied need not be made. Since they are exact copies, both child and parent processes can share the same physical memory. Only when a write is made-- then we copy the child process into physical memory.
So how does this relate to Ruby 1.9/2.0 and Unicorn?
Recall the Unicorn uses forking. In theory, the operating system would be able to take advantage of CoW. Unfortunately, Ruby 1.9 does not make this possible. More accurately, the garbage collection implementation of Ruby 1.9 does not make this possible. An extremely simplified version is this — when the garbage collector of Ruby 1.9 kicks in, a write would have been made, thus rendering CoW useless.
Without going into too much detail, it suffices to say that the garbage collector of Ruby 2.0 fixes this, and we can now exploit CoW.
Tuning Unicorn's Configuration
There are a few settings that we can tune in
config/unicorn.rb to squeeze as much performance as we can from Unicorn.
This sets the number of worker processes to launch. It is important to know how much memory does one process take. This is so that you can safely budget the amount of workers, in order not to exhaust the RAM of your VPS.
This should be set to a small number: usually 15 to 30 seconds is a reasonable number. This setting sets the amount of time before a worker times out. The reason you want to set a relatively low number is to prevent a long-running request from holding back other requests from being processed.
This should be set to
true. Setting this to
true reduces the start up time for starting up the Unicorn worker processes. This uses CoW to preload the application before forking other worker processes. However, there is a big gotcha. We must take special care that any sockets (such as database connections) are properly closed and reopened. We do this using
Here's an example:
before_fork do |server, worker| # Disconnect since the database connection will not carry over if defined? ActiveRecord::Base ActiveRecord::Base.connection.disconnect! end if defined?(Resque) Resque.redis.quit Rails.logger.info('Disconnected from Redis') end end after_fork do |server, worker| # Start up the database connection again in the worker if defined?(ActiveRecord::Base) ActiveRecord::Base.establish_connection end if defined?(Resque) Resque.redis = ENV['REDIS_URI'] Rails.logger.info('Connected to Redis') end end
In this example, we make sure that the connection is closed and reopened when workers are forked. In addition to database connections, we need to make sure that other connections that require sockets are treated similarly. The above includes the configuration for Resque.
Taming Your Unicorn Workers Memory Consumption
Obviously, it's not all rainbows and unicorns (pun intended!). If your Rails app is leaking memory - Unicorn will make it worse.
Each of these forked processes consume memory, since they are copies of the Rails application. Therefore, while having more workers would mean that our application could handle more incoming requests, we are bound by the amount of physical RAM we have on our system.
It is easy for a Rails application to leak memory. Even if we manage to plug all memory leaks, there is still the less than ideal garbage collector to contend with (I am referring to the MRI implementation).
The above shows a Rails application running Unicorn with memory leaks.
Over time, the memory consumption will continue growing. Using multiple Unicorn workers would simply accelerate the rate at which memory is consumed, to the point when there is no more RAM to spare. The application would then grind to a halt — leading to hordes of unhappy users and customers.
It is important to note that this is not Unicorn's fault. However, this is a problem that you would face sooner or later.
Enter the Unicorn Worker Killer
One of the easiest solutions I've come across is the unicorn-worker-killer gem.
From the README:
unicorn-worker-killergem provides automatic restart of Unicorn workers based on
1) max number of requests, and
2) process memory size (RSS), without affecting any requests.
This will greatly improve the site's stability by avoiding unexpected memory exhaustion at the application nodes.
Note that I am assuming that you already have Unicorn set up and running.
unicorn-worker-killer to your Gemfile. Put this below the
group :production do gem 'unicorn' gem 'unicorn-worker-killer' end
Here comes the fun part. Locate and open your
# --- Start of unicorn worker killer code --- if ENV['RAILS_ENV'] == 'production' require 'unicorn/worker_killer' max_request_min = 500 max_request_max = 600 # Max requests per worker use Unicorn::WorkerKiller::MaxRequests, max_request_min, max_request_max oom_min = (240) * (1024**2) oom_max = (260) * (1024**2) # Max memory size (RSS) per worker use Unicorn::WorkerKiller::Oom, oom_min, oom_max end # --- End of unicorn worker killer code --- require ::File.expand_path('../config/environment', __FILE__) run YourApp::Application
First, we check that we are in the
production environment. If so, we will go ahead and execute the code that follows.
unicorn-worker-killer kills workers given 2 conditions: Max requests and Max memory.
In this example, a worker is killed if it has handled between 500 to 600 requests. Notice that this is a range. This minimises the occurrence where more than 1 worker is terminated simultaneously.
Here, a worker is killed if it consumes between 240 to 260 MB of memory. This is a range for the same reason as above.
Every app has unique memory requirements. You should have a rough gauge of the memory consumption of your application during normal operation. That way, you could give a better estimate of the minimum and maximum memory consumption for your workers.
Once you have configured everything properly, upon deploying your app, you will notice a much lesser erratic memory behaviour:
Notice the kinks in the graph. That is the gem doing its job!
Unicorn gives your Rails application a painless way to achieve concurrency, whether it is thread safe or not. However, it comes with a cost of increased RAM consumption. Balancing RAM consumption is absolutely essential to the stability and performance of your application.
We have seen 3 ways to tune your Unicorn workers for maximum performance:
Using Ruby 2.0 gives us a much improved garbage collector that allows us to exploit copy-on-write semantics.
Tuning the various configuration options in
unicorn-worker-killerto solve the problem of gracefully by killing and restarting workers when they get too bloated.