Automated Backups to an Alternate Server

Automated Backups to an Alternate Server

Introduction

While there are various backup methods suggested for OpenEMR, experience shows that the following considerations are important:

• Backups should be automated. When there is a shortage of human resouces, backups are among the first things to fall by the wayside.

• The restore process must be ready to go and periodically tested. Many things can go wrong with a backup procedure and an untested restore means an untested and possibly worthless backup.

• Backups protect against events like vandalism, theft, fire, floods, equipment failure, malicious hacking and human error. The backup system must survive these events. Therefore it is important for it to be in a different physical location and to not be vulnerable to someone who hacks into the production system that is backed up.

• Because human errors can propagate to backups before being discovered, multiple historical backups should be saved.

• Because you cannot restore to a lost production server, it is ideal if the backup server can fill this role in an emergency.

• Backups should take as little time and as little bandwidth as possible.

The General Approach

What we recommend is using an off-site server that is configured with the same or a similar operating system as the production server, and able to similarly support an OpenEMR installation (i.e. with Apache/PHP/MySQL and other required packages installed). It is assumed that you have installed OpenEMR before and are aware of what these requirements are.

This document will show the steps when using Linux servers, but a similar thing can be done with Windows servers if the required software is installed (details of that are not covered here).

We will use the "rsync" utility to copy over differential backups of the MySQL and web directories. "Differential" meaning that only the files or parts of files that have changed since the last backup are copied over the network.

Furthermore the backup server will initiate all connections used for backup. Thus the production server will not have direct access to the backup server.

There will also be a process in place for restoring a backup onto the backup server to produce a fully functional duplicate of the production installation. This process will allow for restoring either the latest backup or some selected previous backup.

The backup server's "cron" utility, or its equivalent, will be used to invoke a backup script at the times that are deemed sensible. Commonly this will be weekday evenings but may easily be more often.

Preparing for Automated Connections to the Production Server

Backups will be performed over an encrypted SSH connection to the production server. Public Key technology will be used so that human intervention is not required for authentication.

To support this make sure the openssh-server package installed on the production server. Then you can generate its public/private key pair for the root user with this command:

 sudo ssh-keygen -t rsa

Be sure to just hit Enter when prompted for a passphrase.

This will create a private key in "/root/.ssh/id_rsa" and a public key in "/root/.ssh/id_rsa.pub". What you want to do is copy this public key (NOT the private key!) to the production server and put it into a file there named "/root/.ssh/authorized_keys". There are various ways to do that but here's one example:

 sudo scp /root/.ssh/id_rsa.pub admin@my.production.server:/tmp/authorized_keys
 ssh admin@my.production.server
 sudo cat /tmp/authorized_keys >> /root/.ssh/authorized_keys
 exit

At this point the root user on the backup server will be able to log in seamlessly as the root user on the production server. So make very sure that the backup server is secure! Test the root login from the backup server like this:

 sudo ssh root@my.production.server

You may be prompted for your local password for sudo, but not for any password on the remote server.

Creating a Backup Script

Next, create a /root/bin directory and a file /root/bin/backup on the backup server. This is what will be executed periodically to retrieve backups. Here's an example of what should be in it:

 #!/bin/bash
 
 # TARGET is a directory that will store your backups. It should have plenty of space
 # available. Commonly this will be a separately mounted drive.
 TARGET=/root/backups
 
 # This is the number of historical backups to retain. This will be a compromise depending
 # on available storage space.
 NUMBACKUPS=14
 
 # Here is the hostname or IP address of the server to be backed up.
 FROM=my.production.server
 
 # This is a short name representing the server to be backed up.
 # It is used to name the local target directories and files.
 SHORT=myprod
 
 cd $TARGET
 
 # Delete the oldest allowed historical backup and rename the others to make room for a new one.
 rm -f  $SHORT.$NUMBACKUPS.tar.gz
 for ((I=NUMBACKUPS; I > 1; I=I-1)); do
   J=$((I-1))
   if [ -e "$SHORT.$J.tar.gz" ]; then
     mv $SHORT.$J.tar.gz $SHORT.$I.tar.gz
   fi
 done
 
 # Make the backup directory if this is the first time.
 if [ ! -d "$SHORT" ]; then
   mkdir $SHORT
 fi
 
 # Archive the previous backup with compression. This could take a while!
 tar zcpf $SHORT.1.tar.gz $SHORT
 
 # Synchromize the production server's database and web directories to the backup copies.
 rsync -axz --numeric-ids --del root@$FROM:/var/lib/mysql $TARGET/$SHORT
 rsync -axz --numeric-ids --del root@$FROM:/var/www       $TARGET/$SHORT

Once you are done creating the backup script, make it executable and then test it:

 sudo chmod a+x /root/bin/backup
 sudo /root/bin/backup

Allow plenty of time for this first run because all of the data for the databases and web directories will be copied over. Subsequent runs will take less time.

Scheduling Automated Backups

On the backup server you can edit /etc/crontab to schedule the backup runs. Append a line to the end of it something like this:

 05  21  *  *  *  root  /root/bin/backup

That will run the backup script at 9:05 pm every night.

Here's a similar example that backs up Monday thru Friday only:

 05  21  *  *  1-5  root  /root/bin/backup

This one backs up at 12:05 pm and 11:05 pm every weekday:

 05  12,23  *  *  1-5  root  /root/bin/backup

Type "man 5 crontab" for more information as to the format of these entries.

Restoring a Backup

Restoring should be done periodically for testing and verification of the backups, and to make sure you are prepared for an emergency.

To restore the last backup directly onto the backup server, do something like this:

 sudo su -
 service mysql stop
 cd /var/lib
 mv mysql mysql.original
 cp -a /root/backups/myprod/mysql .
 chown -R mysql:mysql mysql
 service mysql start
 mysql_upgrade -p
 service apache2 stop
 cd /var
 mv www www.original
 cp -a /root/backups/myprod/www .
 service apache2 start
 exit

The "mysql_upgrade" step may be unnecessary, in which case it will fail and you can ignore it.

If you want to restore a previous backup, you can do the same thing except first extract the compressed backup to a scratch directory and be sure to copy from there.

After you are done with the restored site, you can clean up the system to its original state:

 sudo su -
 service mysql stop
 cd /var/lib
 rm -rf mysql
 mv mysql.original mysql
 service mysql start
 service apache2 stop
 cd /var
 rm -rf www
 mv www.original www
 service apache2 start
 exit

Tweaking

If you are concerned about the MySQL databases being changed while their backup is in progress, the important thing to know is that you can execute commands on the remote system with ssh.

So for example you can bring down the MySQL server before running rsync with a command like this in the backup script:

 ssh root@$FROM 'service mysql stop'

and then after the rsync command:

 ssh root@$FROM 'service mysql start'

Alternatively you could run a command to dump just the OpenEMR database to a temporary file with mysqldump, and then back that up. Of course in that case the restore procedure would be different.