View source for Automated Backups to an Alternate Server

= Automated Backups to an Alternate Server =

== Introduction ==

While there are various backup methods suggested for OpenEMR, experience shows that the following
considerations are important:

* Backups should be automated. When there is a shortage of human resouces, backups are among the first things to fall by the wayside.

* The restore process must be ready to go and periodically tested.  Many things can go wrong with a backup procedure and an untested restore means an untested and possibly worthless backup.

* Backups protect against events like vandalism, theft, fire, floods, equipment failure, malicious hacking and human error.  The backup system must survive these events.  Therefore it is important for it to be in a different physical location and to not be vulnerable to someone who hacks into the production system that is backed up.

* Because human errors can propagate to backups before being discovered, multiple historical backups should be saved.

* Because you cannot restore to a lost production server, it is ideal if the backup server can fill this role in an emergency.

* Backups should take as little time and as little bandwidth as possible.

== The General Approach ==

What we recommend is using an off-site server that is configured with the same or a similar
operating system as the production server, and able to similarly support an OpenEMR installation
(i.e. with Apache/PHP/MySQL and other required packages installed).  It is assumed that you
have installed OpenEMR before and are aware of what these requirements are.

This document will show the steps when using Linux servers, but a similar thing can be done with
Windows servers if the required software is installed (details of that are not covered here).

We will use the "rsync" utility to copy over differential backups of the MySQL and web directories.
"Differential" meaning that only the files or parts of files that have changed since the last
backup are copied over the network.

Furthermore the backup server will initiate all connections used for backup.  Thus the production
server will not have direct access to the backup server.

There will also be a process in place for restoring a backup onto the backup server to produce
a fully functional duplicate of the production installation.  This process will allow for restoring
either the latest backup or some selected previous backup.

The backup server's "cron" utility, or its equivalent, will be used to invoke a backup script
at the times that are deemed sensible.  Commonly this will be weekday evenings but may easily
be more often.

== Preparing for Automated Connections to the Production Server ==

Backups will be performed over an encrypted SSH connection to the production server.
Public Key technology will be used so that human intervention is not required for authentication.

To support this make sure the openssh-server package installed on the production server.
Then you can generate its public/private key pair for the root user with this command:

  sudo ssh-keygen -t rsa

Be sure to just hit Enter when prompted for a passphrase.

This will create a private key in "/root/.ssh/id_rsa" and a public key in "/root/.ssh/id_rsa.pub".
What you want to do is copy this public key (NOT the private key!) to the production server
and put it into a file there named "/root/.ssh/authorized_keys".  There are various ways to do
that but here's one example:

  sudo scp /root/.ssh/id_rsa.pub admin@my.production.server:/tmp/authorized_keys
  ssh admin@my.production.server
  sudo cat /tmp/authorized_keys >> /root/.ssh/authorized_keys
  exit

At this point the root user on the backup server will be able to log in seamlessly as the root
user on the production server.  So make very sure that the backup server is secure!  
Test the root login from the backup server like this:

  sudo ssh root@my.production.server

You may be prompted for your local password for sudo, but not for any password on the remote server.

== Creating a Backup Script ==

Next, create a /root/bin directory and a file /root/bin/backup on the backup server.
This is what will be executed periodically to retrieve backups.
Here's an example of what should be in it:

  #!/bin/bash
  
  # TARGET is a directory that will store your backups. It should have plenty of space
  # available. Commonly this will be a separately mounted drive.
  TARGET=/root/backups
  
  # This is the number of historical backups to retain. This will be a compromise depending
  # on available storage space.
  NUMBACKUPS=14
  
  # Here is the hostname or IP address of the server to be backed up.
  FROM=my.production.server
  
  # This is a short name representing the server to be backed up.
  # It is used to name the local target directories and files.
  SHORT=myprod
  
  cd $TARGET
  
  # Delete the oldest allowed historical backup and rename the others to make room for a new one.
  rm -f  $SHORT.$NUMBACKUPS.tar.gz
  for ((I=NUMBACKUPS; I > 1; I=I-1)); do
    J=$((I-1))
    if [ -e "$SHORT.$J.tar.gz" ]; then
      mv $SHORT.$J.tar.gz $SHORT.$I.tar.gz
    fi
  done
  
  # Make the backup directory if this is the first time.
  if [ ! -d "$SHORT" ]; then
    mkdir $SHORT
  fi
  
  # Archive the previous backup with compression. This could take a while!
  tar zcpf $SHORT.1.tar.gz $SHORT
  
  # Synchromize the production server's database and web directories to the backup copies.
  rsync -axz --numeric-ids --del root@$FROM:/var/lib/mysql $TARGET/$SHORT
  rsync -axz --numeric-ids --del root@$FROM:/var/www       $TARGET/$SHORT

Once you are done creating the backup script, make it executable and then test it:

  sudo chmod a+x /root/bin/backup
  sudo /root/bin/backup

Allow plenty of time for this first run because all of the data for the databases and web
directories will be copied over.  Subsequent runs will take less time.

== Scheduling Automated Backups ==

On the backup server you can edit /etc/crontab to schedule the backup runs.  Append a line to
the end of it something like this:

  05  21  *  *  *  root  /root/bin/backup

That will run the backup script at 9:05 pm every night.

Here's a similar example that backs up Monday thru Friday only:

  05  21  *  *  1-5  root  /root/bin/backup

This one backs up at 12:05 pm and 11:05 pm every weekday:

  05  12,23  *  *  1-5  root  /root/bin/backup

Type "man 5 crontab" for more information as to the format of these entries.

== Restoring a Backup ==

Restoring should be done periodically for testing and verification of the backups, and to
make sure you are prepared for an emergency.

To restore the last backup directly onto the backup server, do something like this:

  sudo su -
  service mysql stop
  cd /var/lib
  mv mysql mysql.original
  cp -a /root/backups/myprod/mysql .
  chown -R mysql:mysql mysql
  service mysql start
  mysql_upgrade -p
  service apache2 stop
  cd /var
  mv www www.original
  cp -a /root/backups/myprod/www .
  service apache2 start
  exit

The "mysql_upgrade" step may be unnecessary, in which case it will fail and you can ignore it.

If you want to restore a previous backup, you can do the same thing except first extract the
compressed backup to a scratch directory and be sure to copy from there.

After you are done with the restored site, you can clean up the system to its original state:

  sudo su -
  service mysql stop
  cd /var/lib
  rm -rf mysql
  mv mysql.original mysql
  service mysql start
  service apache2 stop
  cd /var
  rm -rf www
  mv www.original www
  service apache2 start
  exit

== Tweaking ==

If you are concerned about the MySQL databases being changed while their backup is in progress, the important thing to know is that you can execute commands on the remote system with ssh.

So for example you can bring down the MySQL server before running rsync with a command like this in the backup script:

  ssh root@$FROM 'service mysql stop'

and then after the rsync command:

  ssh root@$FROM 'service mysql start'

Alternatively you could run a command to dump just the OpenEMR database to a temporary file with mysqldump, and then back that up.  Of course in that case the restore procedure would be different.