Changes RSS

====== Differences ====== This shows you the differences between two versions of the page.

Link to this comparison view

guides:bacup:bacula [2010/11/05 20:32]
fishy Windows client backed up! 		guides:bacup:bacula [2010/11/06 16:28] (current)
fishy Untested change to the Windows FileSet.
Line 588: Line 588:
     # WorkingDirectory for the File Daemon.     # WorkingDirectory for the File Daemon.
     File = /var/lib/bacula     File = /var/lib/bacula
 +    
 +    # For mysql servers, it is better to dump and restore
 +    # and/or do binlog replay. So, instead of dumping the
 +    # MySQL data directory, do dull database dumps to
 +    # a different location, and use binlogs in addition if
 +    # real incremental MySQL backups are needed/wanted.
 +    File = /var/lib/mysql
 +
     # Excluding files/directories that do not exist     # Excluding files/directories that do not exist
     # has no effect other than making the FileSet generic..     # has no effect other than making the FileSet generic..
Line 1035: Line 1043:
       WildFile = "[A-Z]:/pagefile.sys"       WildFile = "[A-Z]:/pagefile.sys"
  
 +      # bacula-fd > 3.0 on win32/64 should be able to use VSS,
 +      # so we want to avoid copying the VSS metadata during the backup
 +      File = "\"C:/System Volume Information\""
 +      
       # Bacula working directory       # Bacula working directory
       WildFile = "C:/Program Files/Bacula/working/*"       WildFile = "C:/Program Files/Bacula/working/*"
Line 1063: Line 1075:
 } }
 </code> </code>
 +
 +A word of caution. My Windows client normally runs F-Secure antivirus. With its realtime scanner active, my backups were crazy slow (<600kbit/sec) making my backups take forever. Disabling the scanner boosted the speed up to normal speeds. I am investigating ways to either "white-list" the bacula-fd process from the scanner, or temporarily disabling it during backups.
  
 Use a bconsole to reload the configuration, and test it by running an ''estimate''. And that is all there is to it. Windows-clients are really not that different from Unix clients :) Use a bconsole to reload the configuration, and test it by running an ''estimate''. And that is all there is to it. Windows-clients are really not that different from Unix clients :)
 +
 +===== The MySQL server =====
 +
 +As my Catalog is stored on a remote MySQL server, the parts concering the Catalog dumps in the default configuration files shipped with the Debian packages make little sense. Why pull the data across the wire to the Director server for backup, when the database server needs to be backed up anyway?
 +
 +The backup of the database server will introduce another important, and useful, feature of Bacula backups. Pre- and post backup commands. Basically, to create a backup of the MySQL server and its databases, we want to do some database maintenance and dumping, and then back up the files on the server.
 +
 +A pre-script is a command that will be started on the client that is to be backed up, before any file evaluation is done. If the script fails (returns non-zero), the backup will fail. A post-script may similarly be used to clean up after backups.
 +
 +So, I'll be using a pre-script to dump databases before running the actual backup.
 +But, I'm using a mixed method for dumping. I'm using MySQL's [[http://dev.mysql.com/doc/refman/5.0/en/binary-log.html|binary logging]], allowing atomic change-sets and restores, and backing those up on Increments. On Full and Differential backups, I flush the bin-log, and do a full backup of the databases.
 +
 +First, a note on my mysql binary logging setup. I have created a separate directory for the binary logs:
 +<code>
 +mkdir -p /var/lib/mysql_binlog
 +chown mysql.mysql /var/lib/mysql_binlog
 +</code>
 +
 +The binary log also needs to be configured, by default there is no binary logging enabled on Debian installatoions. So in the [mysqld] section of ''/etc/mysql/my.cnf'', I have the followin lines:
 +<code>
 +log_bin                 = /var/lib/mysql_binlog/database
 +log_bin_index           = /var/lib/mysql_binlog/index
 +expire_logs_days        = 10
 +max_binlog_size         = 500M
 +</code>
 +
 +So, with binary logging available on the MySQL server, it is time to present the script that will be run on the client before data-transfer. This script is written so it contains the MySQL root password, so make sure you place it safe, and set applicable permissions on it. Alternatively, remove any password (and related command options) from the file, and set up a suitable my.cnf instead.
 +
 +<code>
 +#!/bin/sh
 +# Filename: /etc/bacula/scripts/mysqldump.sh
 +# Based on a mix of MySQL recommendations, example scripts in the
 +# Bacula Wiki and personal experience.
 +#
 +# This script is not provided with ANY warranty.
 +# It is provided as an example, by Jon Langseth (fishy at defcon dot no)
 +# and is supplied as-is to the public domain.
 +
 +# Just a security measure...
 +cd /tmp
 +
 +# The backup level is supplied as an argument to the script..
 +LEVEL=$1
 +
 +# Bail out if no Level is given...
 +if [ "${LEVEL}" == "" ];
 +then
 +        echo "No backup level given. Supply Full, Differential or Incremental"
 +        exit 1;
 +fi
 +
 +# Directory to store backups in
 +DST=/var/local/backup
 +
 +# A regex, passed to egrep -v, for which databases to ignore
 +IGNREG='^snort$'
 +
 +# The MySQL username and password
 +DBUSER=root
 +DBPASS=databaserootpassword
 +
 +# Any backups older than KEEPDAYS will be deleted first
 +# Cleanup is only done on Full or Differential backups
 +KEEPDAYS=14
 +
 +if [ "${LEVEL}" == "Full" -o "${LEVEL}" == "Differential" ]
 +then
 +  echo "Full/Differential"
 +
 +        # Clean up in the destination directory, to avoid clogging the filesystem
 +        find ${DST} -type f -mtime +${KEEPDAYS} -exec rm -f {} \;
 +        rmdir $DST/* 2>/dev/null
 +
 +        # Create a new directory for the dump, in a DTG-format.
 +        DATE=$(date  +%Y-%m-%d-%H%M%S)
 +        mkdir -p ${DST}/${DATE}
 +
 +        # Use the 'show databases;' command using mysql in silent mode
 +        # to get the names of all databases on the server. Iterate
 +        # over all the databases with a simple for-loop.
 +        # WARNING: This, and the later dump, may choke on "esoteric" database names.
 +        for db in $(echo 'show databases;' | mysql -s -u ${DBUSER} -p${DBPASS} | egrep -v ${IGNREG}) ; do
 +                echo -n "Backing up ${db}... "
 +                # Dump the databases individually, flushing all logs in the process.
 +                mysqldump \
 +                        --single-transaction \
 +                        --add-drop-table \
 +                        --add-drop-database \
 +                        --add-locks \
 +                        --allow-keywords \
 +                        --complete-insert \
 +                        --create-options \
 +                        --flush-logs \
 +                        --master-data=2 \
 +                        --delete-master-logs \
 +                        --user=${DBUSER} \
 +                        --password=${DBPASS} \
 +                        --databases $db | gzip -c > ${DST}/${DATE}/${db}.txt.gz
 +                echo "Done."
 +        done
 +else
 +        echo "Incremental"
 +        # Simply flush the binary logs. An incremental restore can then
 +        # be performed by replaying the log files.
 +        mysqladmin --user=${DBUSER} --password=${DBPASS} flush-logs
 +fi
 +
 +# Exit cleanly.
 +exit 0
 +</code>
 +
 +So, binary logging is enabled on the MySQL server, and the backup script in place, install the bacula-fd package on the database server, and start setting up a
 +Client definition like usual.
 +
 +<code>
 +Client {
 +  Name = database.example.com;
 +  Address = database.example.com;
 +  FDPort = 9102;
 +  Catalog = StandardCatalog;
 +  Password = "random-password-for-fd-director";
 +  AutoPrune = yes;
 +}
 +</code>
 +
 +What is more interesting, is the one line added to the Job definition:
 +
 +<code>
 +Job {
 +  Name = "database.example.com Default"
 +  FileSet = "Full Set"
 +  JobDefs = "DefaultJob";
 +  Client = database.example.com;
 +
 +  Client Run Before Job = "/etc/bacula/scripts/mysqldump.sh %l";
 +
 +  Write Bootstrap = "/var/lib/bacula/bactank.bsr";
 +}
 +</code>
 +
 +There are several ways to run scripts/programs from the Job definition
 +  * **RunScript** A section-definition for scripts to run on the **Director**
 +  * **Run Before Job** Path and arguments to a script to run on the **Director** before the data-part of the Job is started.
 +  * **Run After Job** Path and arguments to a script to run on the **Director** after the data-part of the Job is done
 +  * **Client Run Before Job** Path and arguments to a script to run on the **Client** before the data-part of the Job is started.
 +  * **Client Run After Job** Path and arguments to a script to run on the **Client** after the data-part of the Job is done
 +  * **Run After Failed Job** Path and arguments to a script to run on the **Director** after a Job is detected as Failed.
 +
 +To these scripts, there are several variables that can be used as arguments with the commands. Most importantly for Client Pre commands:
 +<code>
 +    %% = %
 +    %c = Client's name
 +    %d = Director's name
 +    %i = JobId
 +    %j = Unique Job id
 +    %l = Job Level
 +    %n = Job name
 +    %s = Since time
 +    %t = Job type (Backup, ...)
 +</code>  
 +
 +As seen in my Job definition, after running the Client-Run-Before script, the rest of the Job is simply doing a file-based backup. 
 +
 +If you remember the "Full Set" FileSet, I have Excluded ''/var/lib/mysql'' from the FileSet. This means that for a full Restore of this database server, I will have to start the Restore by pulling in the last Full, then merging inn a Differential if that has run after the Full. After that, I will have to reinstall the MySQL server, tank in the SQL files from ''/var/local/backups''. Finally, to get the database up to the latest increment, I'll have to replay all needed log-files.
 +
 +Documentation related to the pre- and post-scripts are located in the [[http://bacula.org/5.0.x-manuals/en/main/main/Configuring_Director.html#SECTION001830000000000000000|Job definition documentation]]
  
 ===== Links, references, scratch ===== ===== Links, references, scratch =====
 
You are here: start » guides » bacup » bacula
Trace:
guides/bacup/bacula.1288985547.txt.gz · Last modified: 2010/11/05 20:32 by fishy