How To Upgrade Nginx In-Place Without Dropping Client Connections
Nginx is a powerful web server and reverse proxy that is used to serve many of the most popular sites in the world. In this guide, we'll demonstrate how to upgrade the Nginx executable in place, without losing client connections.
Before beginning this guide, you should have a non-root user on your server, configured with
sudo privileges. You will also need to have Nginx installed.
How the Upgrade Works
Nginx works by spawning a master process when the service starts. The master service, in turn, kicks on one or more worker processes that handle the actual client connections. Nginx is designed to perform certain actions when it receives specific signals from the administrator. Using these signals provides you with the opportunity to easily upgrade Nginx or its configuration in-place, without losing client connections.
Certain service scripts provided by distribution package maintainers will provide this ability for use with conventional upgrades. However, manually upgrading provides more flexibility in the approach and allows you to audit the upgrade to revert quickly if there are problems. This will also provide an option to upgrade gracefully if you have installed Nginx from source or using a method that does not provide this capability.
The following signals will be used:
USR2: This spawns a new set of master/worker processes without affecting the old set.
WINCH: This tells the Nginx master process to gracefully stop its associated worker instances.
HUP: This tells an Nginx master process to re-read its configuration files and replace worker processes with those adhering to the new configuration. If an old and new master are running, sending this to the old master will spawn workers using their original configuration.
QUIT: This shuts down a master and its workers gracefully.
TERM: This initiates a fast shutdown of the master and its workers.
KILL: This immediately kills a master and its workers without any cleanup.
Finding Nginx Process PIDs
In order to send signals to the various processes, we need to know the PID for the target process. There are two easy ways to find this.
First, you can use the
ps utility and then
grep for Nginx among the results. This is simple and allows you to see the master and worker processes:
- ps aux | grep nginx
outputroot 10846 0.0 0.3 47564 3280 ? S 13:26 0:00 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf nginx 10847 0.0 0.1 47936 1908 ? S 13:26 0:00 nginx: worker process user 10961 0.0 0.0 112640 964 pts/0 S+ 13:53 0:00 grep --color=auto nginx
The second column contains the PIDs for the selected processes. Highlighted is the PID for the process. The last column clarifies that the first result is an Nginx master process.
Another way to find the PID for the master Nginx process is to print out the contents of the
- cat /run/nginx.pid
If there are two Nginx master processes running, the old one will be moved to
Spawn a New Nginx Master/Workers Set
The first step to gracefully updating our executable is to actually update your binary. Do this using whatever method is appropriate for your Nginx installation, whether through a package manager or a source installation.
After the new binary is in place, you can spawn a second set of master/worker processes that use the new executable.
You can do this either by sending the
USR2 signal directly to the PID number you queried (make sure to substitute the PID of your own Nginx master process here):
- sudo kill -s USR2 10846
Or, you can read and substitute the value stored in your PID file directly into the command, like this:
- sudo kill -s USR2 `cat /run/nginx.pid`
If you check your current processes, you will see that you now have two sets of Nginx master/workers:
- ps aux | grep nginx
outputroot 10846 0.0 0.3 47564 3280 ? S 13:26 0:00 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf nginx 10847 0.0 0.1 47936 1908 ? S 13:26 0:00 nginx: worker process root 11003 0.0 0.3 47564 3132 ? S 13:56 0:00 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf nginx 11004 0.0 0.1 47936 1912 ? S 13:56 0:00 nginx: worker process user 11031 0.0 0.0 112640 960 pts/0 S+ 14:01 0:00 grep --color=auto nginx
You can also see that the original
/run/nginx.pid file has been moved to
/run/nginx.pid.oldbin and the newer master process's PID has been written to
- tail -n +1 /run/nginx.pid*
output==> /run/nginx.pid <== 11003 ==> /run/nginx.pid.oldbin <== 10846
You can now send signals to either of the master processes using the PIDs contained in these files.
At this point, both master/worker sets are operational and capable of serving client requests. The first set is using the original Nginx executable and configuration and the second set is using the newer versions. They can continue to operate side-by-side, but for consistency, we should start to transition to the new set.
Shut Down the First Master's Workers
In order to begin the transition to the new set, the first thing we can do is stop the original master's worker processes. The original workers will finish up handling all of their current connections and then exit.
Stop the original set's workers by issuing the
WINCH signal to their master process:
- sudo kill -s WINCH `cat /run/nginx.pid.oldbin`
This will let the new master's workers handle new client connections alone. The old master process will still be running, but with no workers:
- ps aux | grep nginx
outputroot 10846 0.0 0.3 47564 3280 ? S 13:26 0:00 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf root 11003 0.0 0.3 47564 3132 ? S 13:56 0:00 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf nginx 11004 0.0 0.1 47936 1912 ? S 13:56 0:00 nginx: worker process user 11089 0.0 0.0 112640 964 pts/0 R+ 14:13 0:00 grep --color=auto nginx
This lets you audit the new workers as they accept connections in isolation while maintaining the ability to revert to the old executable if something isn't quite right.
Evaluate the Outcome and Take the Next Steps
You should test and audit your system at this point to make sure that there are no signs of problems. You can leave your configuration in this state for as long as you wish to ensure that the new Nginx executable is bug-free and able to handle your traffic.
Your next step will depend entirely on whether you ran into problems.
If Your Upgrade was Successful, Complete the Transition
If you experienced no issues with your new set's workers, you can safely shut down the old master process. To do this, just send the old master the
sudo kill -s QUIT `cat /run/nginx.pid.oldbin`
The old master process will exit gracefully, leaving only your new set of Nginx master/workers. At this point, you've successfully performed an in-place binary update of Nginx without interrupting client connections.
If the New Workers Experience Problems, Revert to the Old Binary
If your new set of workers seem to be having problems, you can transition back to the old configuration and binary. This is possible during the same session.
The best way to do this is to restart your old master's workers by sending it the
HUP signal. Usually, when you send an Nginx master the
HUP signal, it will re-read its configuration files and start new workers. However, when the target is an older master, it will just spawn new workers using its original, working configuration:
- sudo kill -s HUP `cat /run/nginx.pid.oldbin`
You now should be back to having two sets of master/worker processes:
- ps aux | grep nginx
outputroot 10846 0.0 0.3 47564 3280 ? S 13:26 0:00 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf root 11003 0.0 0.3 47564 3132 ? S 13:56 0:00 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf nginx 11004 0.0 0.1 47936 1912 ? S 13:56 0:00 nginx: worker process nginx 19918 0.0 0.1 47936 1900 ? S 14:47 0:00 nginx: worker process user 19920 0.0 0.0 112640 964 pts/0 R+ 14:48 0:00 grep --color=auto nginx
The newest workers are associated with the old worker. Both worker sets will be accepting client connections at this point. Now, stop the newer, buggy master process and its workers by sending the
- sudo kill -s QUIT `cat /run/nginx.pid`
You should be back to your old master and workers:
- ps aux | grep nginx
outputroot 10846 0.0 0.3 47564 3280 ? S 13:26 0:00 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf nginx 19918 0.0 0.1 47936 1900 ? S 14:47 0:00 nginx: worker process user 19935 0.0 0.0 112640 964 pts/0 R+ 14:50 0:00 grep --color=auto nginx
The original master will regain the
/run/nginx.pid file for its PID.
If the above does not work for any reason, you can try just sending the new master server the
TERM signal, which should initiate a shutdown. This should stop the new master and any workers while automatically kicking over the old master to start its worker processes. If there are serious problems and the buggy workers are not exiting, you can send each of them a
KILL signal to clean up. This should be viewed as a last resort, however, as it will cut off connections.
After transitioning back to the old binary, remember that you still have the new version installed on your system. You should remove the buggy version and roll back to your previous version so that Nginx will run without issues on reboot.
By now, you should be able to seamlessly transition your machines from one Nginx binary to another. Nginx's ability to handle two master/workers sets while maintaining information about their relationships provides us with the ability to upgrade server software without taking the server machines offline.