After coming up with a recovery plan for the various components of your application, you should set up the backup system that is required to support it. This tutorial will focus on using Bacula as a backups solution. The benefits of using a full-fledged backup system, such as Bacula, is that it gives you full control over what you back up and restore at the individual file level, and you can schedule backups and restores according to what is best for you.
Solutions such as DigitalOcean Droplet Backups (snapshot backups of your entire Droplet) are easy to set up and may be sufficient for your needs, if you only require weekly backups. If you opt for DigitalOcean Backups, be sure to set up hot backups of your database by following the Create Hot Backups of Your Database section.
In this part of the tutorial, we will set up a Bacula to maintain daily backups of the required backups of the servers that comprise your application setup (db1, app1, app2, and lb1), defined previously in our recovery plan—essentially, this is a tutorial that shows you how to use Bacula to create backups of a LAMP stack. We will also use Percona XtraBackup to create hot backups of your MySQL database. Lastly, we will use rsync to create a copy of your backups, on a server in a remote data center. This will add two servers to your setup: backups and remotebackups (located in a separate data center).
Let’s get started.
Set up Bacula on your backups server by following this tutorial: How To Install Bacula Server on Ubuntu 14.04.
Then follow the Organize Bacula Director Configuration (Server) section of this tutorial: How To Back Up an Ubuntu 14.04 Server with Bacula. You will need the Director Name when setting up the Bacula clients (on the servers you want to back up). Stop when you reach the Install and Configure Bacula Client section.
Note that we will be using the RemoteFile pool for all of the backups jobs that we will be setting up. With that said, you may want to change some of the settings before proceeding.
Install the Bacula client on each server that you want to back up (db1, app1, app2, and lb1) by following the Install and Configure Bacula Client section of this tutorial: How To Back Up an Ubuntu 14.04 Server with Bacula. Stop when you reach the Add FileSets (Server) section.
Note that you will need the FileDaemon Name (usually the hostname appended by “-fd”) and the Director Password (the password that the Bacula server will use to connect to each client) from the bacula-fd.conf
file on each server.
On backups, the Bacula server, add a Client resource to the /etc/bacula/conf.d/clients.conf
file for each server that you installed the Bacula client on.
Open the clients.conf
file:
- sudo vi /etc/bacula/conf.d/clients.conf
Here is an example of the Client resource definition for the database server, db1. Note that the value of Name should match the the Name of the FileDaemon resource and the Password should match the Password of the Director resource, on the client server—these values can be found in /etc/bacula/bacula-fd.conf
on each Bacula client server:
Client {
Name = db1-fd
Address = db1.nyc3.example.com
FDPort = 9102
Catalog = MyCatalog
Password = "PDL47XPnjI0QzRpZVJKCDJ_xqlMOp4k46" # password for Remote FileDaemon
File Retention = 30 days # 30 days
Job Retention = 6 months # six months
AutoPrune = yes # Prune expired Jobs/Files
}
Create a similar Client resource for each of the remaining Bacula client servers. In our example, there should be four Client resources when we are finished: db1-fd, app1-fd, app2-fd, and lb1-fd. This configures the Bacula Director, on the backups server, to be able to connect to the Bacula client on each server…
Save and exit.
More details about this section can be found in the Install and Configure Bacula Client in the How To Back Up an Ubuntu Server with Bacula tutorial.
To ensure that we produce consistent (i.e. usable) backups of our active database, special care must be taken. A simple and effective way to create hot backups with MySQL is to use Percona XtraBackup.
On your database server, db1, install and configure Percona XtraBackup by following this tutorial: How To Create Hot Backups of MySQL Databases with Percona XtraBackup on Ubuntu 14.04. Stop when you reach the Perform Full Hot Backup section.
Percona XtraBackup is ready to create hot backups of your MySQL database, which will ultimately be backed up by Bacula (or DigitalOcean Backups), but the hot backups must be scheduled somehow. We will set up the simplest solution: a bash script and a cron job.
Create a bash script called run_extra_backup.sh
in /usr/local/bin
:
- sudo vi /usr/local/bin/run_xtrabackup.sh
Add the following script. Be sure to substitute the user and password with whatever you set up when you installed XtraBackup:
#!/bin/bash
# pre xtrabackup
chown -R mysql: /var/lib/mysql
find /var/lib/mysql -type d -exec chmod 770 "{}" \;
# delete existing full backup
rm -r /data/backups/full
# xtrabackup create backup
innobackupex --user=bkpuser --password=bkppassword --no-timestamp /data/backups/full
# xtrabackup prepare backup
innobackupex --apply-log /data/backups/full
Save and exit. Running this script (with superuser privileges) will delete the existing XtraBackup backup at /data/backups/full
and create a new full backup. More details about creating backups with XtraBackup can be found in the Perform Full Hot Backup section of the of the XtraBackup tutorial.
Make the script executable:
- sudo chmod +x /usr/local/bin/run_xtrabackup.sh
In order to properly backup our database, we must run (and complete) the XtraBackup script before Bacula tries to backup the database server. A good solution is to configure your Bacula backup job to run the script as a “pre-backup script”, but we will opt to use a cron job to keep it simple.
Create a cron configuration file (files in /etc/cron.d
get added to root’s crontab):
- sudo vi /etc/cron.d/xtrabackup
Add the following cron job:
30 22 * * * root /usr/local/bin/run_xtrabackup.sh
This schedules the script to run as root every day at 10:30pm (22nd hour, 30th minute). We chose this time because Bacula is currently scheduled to run its backup jobs at 11:05pm daily—we will discuss adjusting this later. This allows 35 minutes for the XtraBackup script to complete.
Now that the database hot backups are set up, let’s look at the Bacula backup FileSets.
Bacula will create backups of files that are specified in the FileSets that are associated with the backup Jobs that will be executed. This section will cover creating FileSets that include the required backups that we identified in our recovery plans. More details about adding FileSets to Bacula can be found in the Add FileSets (Server) section of the Bacula tutorial.
On your backups server, open the filesets.conf
file:
- sudo vi /etc/bacula/conf.d/filesets.conf
The required backups for our database server, according to our database server recovery plan, include:
/data/backups/full
, daily at 10:30pm/etc/mysql
We also will include the XtraBackup script: /usr/local/bin/run_xtrabackup.sh
, and the associated cron file.
With our required backups in mind, we will add this “MySQL Database” FileSet to our Bacula configuration:
FileSet {
Name = "MySQL Database"
Include {
Options {
signature = MD5
compression = GZIP
}
File = /data/backups
File = /etc/mysql/my.cnf
File = /usr/local/bin/run_xtrabackup.sh
File = /etc/cron.d/xtrabackup
}
Exclude {
File = /data/backups/exclude
}
}
Now let’s move on to the application server FileSet.
The required backups for our application servers, according to our application server recovery plan, include:
/var/www/html
in our exampleWith our required backups in mind, we will add this “Apache DocumentRoot” FileSet to our Bacula configuration:
FileSet {
Name = "Apache DocumentRoot"
Include {
Options {
signature = MD5
compression = GZIP
}
File = /var/www/html
}
Exclude {
File = /var/www/html/exclude
}
}
You may want to also include the Apache ports configuration file, but that is easily replaceable.
Now let’s move on to the load balancer server FileSet.
The required backups for our load balancer servers, according to our load balancer server recovery plan, include:
/root/certs
in our example/etc/haproxy
With our required backups in mind, we will add this “Apache DocumentRoot” FileSet to our Bacula configuration:
FileSet {
Name = "SSL Certs and HAProxy Config"
Include {
Options {
signature = MD5
compression = GZIP
}
File = /root/certs
File = /etc/haproxy
}
Exclude {
File = /root/exclude
}
}
Save and exit.
Now our FileSets are configured. Let’s move on to the creating the Bacula backup Jobs that will use these FileSets.
We will create Bacula backup Jobs that will run and create backups of our servers.
Create a jobs.conf
file in /etc/bacula/conf.d
:
- sudo vi /etc/bacula/conf.d/jobs.conf
For our database server backup job, we will create a new job named “Backup db1”. The important thing here is that we specify the correct Client (db1-fd) and FileSet (MySQL Database):
Job {
Name = "Backup db1"
JobDefs = "DefaultJob"
Client = db1-fd
Pool = RemoteFile
FileSet="MySQL Database"
}
Now we will set up the application server backup jobs.
For our application servers, we will create two backup jobs named “Backup app1” and “Backup app2”. The important thing here is that we specify the correct Clients (app1-fd and app2-fd) and FileSet (Apache DocumentRoot).
App1 job:
Job {
Name = "Backup app1"
JobDefs = "DefaultJob"
Client = app1-fd
Pool = RemoteFile
FileSet="Apache DocumentRoot"
}
App2 job:
Job {
Name = "Backup app2"
JobDefs = "DefaultJob"
Client = app2-fd
Pool = RemoteFile
FileSet="Apache DocumentRoot"
}
Now we will set up the load balancer server backup job.
For our load balancer server backup job, we will create a new job named “Backup lb1”. The important thing here is that we specify the correct Client (lb1-fd) and FileSet (SSL Certs and HAProxy Config):
Job {
Name = "Backup lb1"
JobDefs = "DefaultJob"
Client = lb1-fd
Pool = RemoteFile
FileSet="SSL Certs and HAProxy Config"
}
Save and exit.
Now our backup Jobs are configured. The last step is to restart the Bacula Director.
On the backups server, restart the Bacula Director to put all of our changes into effect:
- sudo service bacula-director restart
At this point, you will want to test your client connections and backup jobs, both of which are covered in the How To Back Up a Server with Bacula tutorial. That tutorial also covers how to restore Bacula backups. Note that restoring the MySQL database will require you to follow the Perform Backup Restoration step in the Percona XtraBackup Tutorial.
The Bacula backups schedule can be adjusted by modifying the Bacula Director configuration (/etc/bacula/bacula-dir.conf
). All of the backup Jobs that we created use the “DefaultJob” JobDef, which uses the “WeeklyCycle” schedule, which is defined as:
You can verify this by using the Bacula console to check the status of the Director. It should output all of your scheduled jobs:
Director Status — Scheduled JobsScheduled Jobs:
Level Type Pri Scheduled Name Volume
===================================================================================
Incremental Backup 10 20-May-15 23:05 BackupLocalFiles MyVolume
Incremental Backup 10 20-May-15 23:05 Backup lb1 Remote-0002
Incremental Backup 10 20-May-15 23:05 Backup app2 Remote-0002
Incremental Backup 10 20-May-15 23:05 Backup app1 Remote-0002
Incremental Backup 10 20-May-15 23:05 Backup db1 Remote-0002
Feel free to add or adjust the schedule of any of your backup jobs. It would make sense to modify the schedule of the application servers to occur at the same time that the Percona XtraBackup script is executed (10:30pm). This will prevent the application and database backups from being inconsistent with each other.
Now we’re ready to set up a remote server that will store copies of our Bacula backups. This remote server should be in a geographically separate region so you will have a copy of your critical backups even if there is a disaster in your production data center. In our example, we will use DigitalOcean’s San Francisco (SFO1) region for our remotebackups server.
We will explain a simple method to send our backups from our backups server to our remotebackups server using public SSH keys, rsync, and cron.
On the remotebackups server, create a user that will be used for the rsync login.
Next, on the backups server, generate a password-less SSH key pair as root. Install the public key on the remotebackups user that you just created. This is covered in our How To Set Up SSH Keys tutorial.
On the backups server, write up an rsync command that copies the Bacula backup data (/bacula/backup
) to somewhere on the remotebackups server. Rsync usage is covered in our How To Use Rsync tutorial. The command will probably look something like this:
- rsync -az /bacula/backup remoteuser@remotebackups_public_hostname_or_IP:/path/to/remote/backup
Add the command to a script, such as /usr/local/bin/rsync_backups.sh
and make it executable.
Lastly, you will want to set up a cron job that runs the rsync_backups.sh
script as root, after the Bacula backups jobs usually complete. This is covered in our How To Schedule Routine Tasks With Cron tutorial.
After you set all of this up, verify that there is a copy of your backups on the remotebackups server the next day.
We didn’t talk about the disk requirements for your backups. You will definitely want to review how much disk space your backups are using, and revise your setup and backups schedule based on your needs and resources.
In addition to creating backups of your application servers, you will probably want to set up backups for any other servers that are added to your setup. For example, you should configure Bacula to create backups of your monitoring and centralized logging servers once you get them up and running.
You should now have daily backups, and a remote copy of those backups, of your production application servers. Be sure to verify that you are able to restore the files, and add the steps of restoring your data to your recovery plans.
Continue to the next tutorial to start setting up the monitoring for your production server setup: Building for Production: Web Applications — Monitoring.
Thanks for learning with the DigitalOcean Community. Check out our offerings for compute, storage, networking, and managed databases.
This 6-part tutorial will show you how to build out a multi-server production application setup from scratch. The final setup will be supported by backups, monitoring, and centralized logging systems, which will help you ensure that you will be able to detect problems and recover from them. The ultimate goal of this series is to build on standalone system administration concepts, and introduce you to some of the practical considerations of creating a production server setup.
This textbox defaults to using Markdown to format your answer.
You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!
Sign up for Infrastructure as a Newsletter.
Working on improving health and education, reducing inequality, and spurring economic growth? We'd like to help.
Get paid to write technical tutorials and select a tech-focused charity to receive a matching donation.
Great tutorial, but using DigitalOcean droplet plans is impossible.
The backup server only needs a big disk, and DO plans only offer big disk with big RAM and CPUs.