'out of his depth' question: What are good ways to have structured, controlled patching of servers across environments?

March 30, 2016 850 views
Scaling Security Configuration Management High Availability Deployment System Tools Server Optimization

We have about 5 dozen RHEL and Centos servers at various levels. I am part of a small team of 2.2 members. My team and predecessors never addressed how to roll out patches, so every security audit becomes a panic patch session. I have been asked to fix the root of the problem and proactively patch on a monthly schedule similar to what some other OSes are doing. I have been instructed to apply patches to a Proof Of Concept bunch of servers (that i will have to build) for testing, then roll out the patches to Dev for let more testing, then the other areas. I am new to the Linux thing, so i am a very unsure of the correct direction that will not lead to administrative h***.

My current working idea is this: Host a yum repo for each Environment we need to separate. Then, puppet can be set to install the "latest" version of everything, but I control what "latest" means with the repo. I can populate the repo by having a purpose built server just download the necessary updates and dependencies, and copy these to the POC repo. These patches can be promoted to the other yum repos as tests pass and other departments give the green light. But boy, this seems like alot of work.

The backup idea is to specify the patch level of each package for each environment in puppet. This also seems like its own special level of h***. Maybe my puppet-foo is what needs to be fixed?

Yes, I talked to my team about this. They are surprisingly non-committal. Am I re-engineering the wheel? Is there an elegant way to do this that my noobness is not aware of yet? I dont want to be just a package patcher all day every day...

(ps: How is there a "chef" and "ansible" tag but not a "puppet" tag for these questions?)

  • sadly, i lack experience in some of the configuration management tools, but I looked into ansible for a brief time which may have something along the lines that you are looking for. perhaps a few lines of ansible yaml config code to run a "yum install" or a "yum update" ..., etc " ?

  • Yeah, Puppet can do the same thing (and we already use Puppet rather than Ansible), but it seems clumsy and labor intensive at best for managing dozens of packages on dozens of servers across about 5 environments. I know of a few ways to just get it done that are like this. What I am looking for is a more sophisticated and nuanced approach to the whole task.

    I am not asking something like "how do I update Linux?", not even "How do I update all my servers at once?", but rather "How do I intelligently manage my patching, rolling patches from POC to DEV to QA to STG to PRD?"

    Maybe I am wrong about Puppet and it is not clumsy and labor intensive for this task. I am hoping that if this is the case, that someone will kindly point out the "best practice" method of getting this done. Perhaps a module I overlooked?

  • Try Spacewalk, haven't tried it myself, but it looks very much suitable for your needs - http://spacewalk.redhat.com/

  • Spacewalk looks promising. Thank you.

2 Answers


Your strategy looks like the best way to proceed at this time. Sadly, configuration management tools are both the best way to handle patching, and the worst. Puppet, Ansible, and Chef can work with your private repos to promote versions of software, but none offer a great solution for this kind of staged patching. Spacewalk is a great tool for sure, but you may find that it is a little too simple and gui dependent for large deployments. I hope that one of the configuration management tools, or a new challenger steps up and creates an ideal solution for package management.

Have another answer? Share your knowledge.