How To Optimize Unicorn Workers in a Ruby on Rails App

If you are a Rails developer, you’ve probably heard of Unicorn, a HTTP server that can handle multiple requests concurrently.

Unicorn uses forked processes to achieve concurrency. Since forked processes are essentially copies of each other, this means that the Rails application need not be thread safe.

This is great because it is difficult to ensure that our own code is thread safe. If we cannot ensure that our code is thread safe, then concurrent web servers such as Puma and even alternative Ruby implementations that exploit concurrency and parallelism such as JRuby and Rubinius would be out of the question.

Therefore, Unicorn gives our Rails apps concurrency even when they are not thread safe. However, this comes at a cost. Rails apps running on Unicorn tend to consume much more memory. Without paying any heed to the memory consumption of your app, you may well find yourself with an overburdened cloud server.

In this article, we will explore a few ways to exploit Unicorn’s concurrency, while at the same time control the memory consumption.

If you are using Ruby 1.9, you should seriously consider switching to Ruby 2.0. To understand why, we need to understand a little bit about forking.

Forking and Copy-on-Write (CoW)

When a child process is forked, it is the exact same copy as the parent process. However, the actual physical memory copied need not be made. Since they are exact copies, both child and parent processes can share the same physical memory. Only when a write is made-- then we copy the child process into physical memory.

So how does this relate to Ruby 1.9/2.0 and Unicorn?

Recall the Unicorn uses forking. In theory, the operating system would be able to take advantage of CoW. Unfortunately, Ruby 1.9 does not make this possible. More accurately, the garbage collection implementation of Ruby 1.9 does not make this possible. An extremely simplified version is this — when the garbage collector of Ruby 1.9 kicks in, a write would have been made, thus rendering CoW useless.

Without going into too much detail, it suffices to say that the garbage collector of Ruby 2.0 fixes this, and we can now exploit CoW.

There are a few settings that we can tune in config/unicorn.rb to squeeze as much performance as we can from Unicorn.

worker_processes

This sets the number of worker processes to launch. It is important to know how much memory does one process take. This is so that you can safely budget the amount of workers, in order not to exhaust the RAM of your VPS.

timeout

This should be set to a small number: usually 15 to 30 seconds is a reasonable number. This setting sets the amount of time before a worker times out. The reason you want to set a relatively low number is to prevent a long-running request from holding back other requests from being processed.

preload_app

This should be set to true. Setting this to true reduces the start up time for starting up the Unicorn worker processes. This uses CoW to preload the application before forking other worker processes. However, there is a big gotcha. We must take special care that any sockets (such as database connections) are properly closed and reopened. We do this using before_fork and after_fork.

Here’s an example:

before_fork do |server, worker|
  # Disconnect since the database connection will not carry over
  if defined? ActiveRecord::Base
    ActiveRecord::Base.connection.disconnect!
  end
  
  if defined?(Resque)
    Resque.redis.quit
    Rails.logger.info('Disconnected from Redis')
  end
end

after_fork do |server, worker|
  # Start up the database connection again in the worker
  if defined?(ActiveRecord::Base)
    ActiveRecord::Base.establish_connection
  end
  
  if defined?(Resque)
    Resque.redis = ENV['REDIS_URI']
    Rails.logger.info('Connected to Redis')
  end
end

In this example, we make sure that the connection is closed and reopened when workers are forked. In addition to database connections, we need to make sure that other connections that require sockets are treated similarly. The above includes the configuration for Resque.

Obviously, it’s not all rainbows and unicorns (pun intended!). If your Rails app is leaking memory - Unicorn will make it worse.

Each of these forked processes consume memory, since they are copies of the Rails application. Therefore, while having more workers would mean that our application could handle more incoming requests, we are bound by the amount of physical RAM we have on our system.

It is easy for a Rails application to leak memory. Even if we manage to plug all memory leaks, there is still the less than ideal garbage collector to contend with (I am referring to the MRI implementation).

The above shows a Rails application running Unicorn with memory leaks.

Over time, the memory consumption will continue growing. Using multiple Unicorn workers would simply accelerate the rate at which memory is consumed, to the point when there is no more RAM to spare. The application would then grind to a halt — leading to hordes of unhappy users and customers.

It is important to note that this is not Unicorn’s fault. However, this is a problem that you would face sooner or later.

One of the easiest solutions I’ve come across is the unicorn-worker-killer gem.

From the README:

unicorn-worker-killer gem provides automatic restart of Unicorn workers based on

1. max number of requests, and
2. process memory size (RSS), without affecting any requests.

This will greatly improve the site’s stability by avoiding unexpected memory exhaustion at the application nodes.

Note that I am assuming that you already have Unicorn set up and running.

Step 1:

Add unicorn-worker-killer to your Gemfile. Put this below the unicorn gem.

group :production do 
  gem 'unicorn'
  gem 'unicorn-worker-killer'
end

Step 2:

Run bundle install.

Step 3:

Here comes the fun part. Locate and open your config.ru file.

# --- Start of unicorn worker killer code ---

if ENV['RAILS_ENV'] == 'production' 
  require 'unicorn/worker_killer'

  max_request_min =  500
  max_request_max =  600

  # Max requests per worker
  use Unicorn::WorkerKiller::MaxRequests, max_request_min, max_request_max

  oom_min = (240) * (1024**2)
  oom_max = (260) * (1024**2)

  # Max memory size (RSS) per worker
  use Unicorn::WorkerKiller::Oom, oom_min, oom_max
end

# --- End of unicorn worker killer code ---

require ::File.expand_path('../config/environment',  __FILE__)
run YourApp::Application

First, we check that we are in the production environment. If so, we will go ahead and execute the code that follows.

unicorn-worker-killer kills workers given 2 conditions: Max requests and Max memory.

Max Requests

In this example, a worker is killed if it has handled between 500 to 600 requests. Notice that this is a range. This minimises the occurrence where more than 1 worker is terminated simultaneously.

Max Memory

Here, a worker is killed if it consumes between 240 to 260 MB of memory. This is a range for the same reason as above.

Every app has unique memory requirements. You should have a rough gauge of the memory consumption of your application during normal operation. That way, you could give a better estimate of the minimum and maximum memory consumption for your workers.

Once you have configured everything properly, upon deploying your app, you will notice a much lesser erratic memory behaviour:

Notice the kinks in the graph. That is the gem doing its job!

Unicorn gives your Rails application a painless way to achieve concurrency, whether it is thread safe or not. However, it comes with a cost of increased RAM consumption. Balancing RAM consumption is absolutely essential to the stability and performance of your application.

We have seen 3 ways to tune your Unicorn workers for maximum performance:

1. Using Ruby 2.0 gives us a much improved garbage collector that allows us to exploit copy-on-write semantics.

2. Tuning the various configuration options in config/unicorn.rb.

3. Using unicorn-worker-killer to solve the problem of gracefully by killing and restarting workers when they get too bloated.

• A wonderful explanation of how the Ruby 2.0 garbage collector and copy-on-write semantics works.

• The full list of Unicorn Configuration options.

<div class=“author”>Submitted by: <a href=“http://benjamintanweihao.github.io/”>Benjamin Tan</div>