Changes RSS

====== Differences ====== This shows you the differences between two versions of the page.

Link to this comparison view

guides:bacup:bacula [2010/11/01 16:47]
fishy Next client definition! 		guides:bacup:bacula [2010/11/06 16:28] (current)
fishy Untested change to the Windows FileSet.
Line 173: Line 173:
 Since Bacula is separated into different components that can live completely Since Bacula is separated into different components that can live completely
 separate, configuration of these components are split into respecitve configuration files. Needless to say, these configuration files will relate to each other, enabling communication between the components. Here is an attempt at visualizing the relations: separate, configuration of these components are split into respecitve configuration files. Needless to say, these configuration files will relate to each other, enabling communication between the components. Here is an attempt at visualizing the relations:
 +
 {{:guides:bacup:bacula-config-relations.png?500|}} {{:guides:bacup:bacula-config-relations.png?500|}}
  
Line 193: Line 194:
   * I will be using one FileSet for each client   * I will be using one FileSet for each client
   * On the host __bactank__ I will be excluding /opt completely, and store VTL files and File-based "volumes" under that directory.   * On the host __bactank__ I will be excluding /opt completely, and store VTL files and File-based "volumes" under that directory.
 +
 +:!: **NOTE**
 +
 +In the Debian configuration files, the passwords seem like they are auto-generated. **They are not**. Please generate new passwords wherever appropriate. To be relatively random, you can use pwgen like this:
 +<code>
 +pwgen -s 32 1
 +AwAdLG9doWqJuQgZ0BmnMViX8lCgAH7S 
 +</code>
 ==== Storage Daemon ==== ==== Storage Daemon ====
  
Line 361: Line 370:
 === A read-only console for monitoring === === A read-only console for monitoring ===
  
 +<code>
 Console { Console {
   Name = bactank-mon   Name = bactank-mon
Line 366: Line 376:
   CommandACL = status, .status; # Allow only status-reading.   CommandACL = status, .status; # Allow only status-reading.
 } }
 +</code>
  
 === Catalog === === Catalog ===
Line 444: Line 455:
 </code> </code>
  
-The comments say "N.B. Use a fully qualified name here", but in reality, anything that ends up with the IP that the Storage Daemon listens on can be used. Note that the Storage definition for the autochanger references the Autochanger as Device, not the individual tape drives. It is the responsibility of the Storage Daemon to represent the changer correctly. Also, remember the notes about Media Type when we configured the Storage Daemon.+The comments say "N.B. Use a fully qualified name here", but in reality, anything that ends up with the IP that the Storage Daemon listens on can be used. Observe that the name used here, will be reported to File Servers, and they will use it to communicate with the Storage Daemon. So all Clients will also need to be able to get an IP from what you enter here:!: 
 + 
 +Note that the Storage definition for the autochanger references the Autochanger as Device, not the individual tape drives. It is the responsibility of the Storage Daemon to represent the changer correctly. Also, remember the notes about Media Type when we configured the Storage Daemon.
  
 === Pools === === Pools ===
Line 575: Line 588:
     # WorkingDirectory for the File Daemon.     # WorkingDirectory for the File Daemon.
     File = /var/lib/bacula     File = /var/lib/bacula
 +    
 +    # For mysql servers, it is better to dump and restore
 +    # and/or do binlog replay. So, instead of dumping the
 +    # MySQL data directory, do dull database dumps to
 +    # a different location, and use binlogs in addition if
 +    # real incremental MySQL backups are needed/wanted.
 +    File = /var/lib/mysql
 +
     # Excluding files/directories that do not exist     # Excluding files/directories that do not exist
     # has no effect other than making the FileSet generic..     # has no effect other than making the FileSet generic..
Line 611: Line 632:
 <code> <code>
 Job { Job {
-  Name = "bactank.ecample.com Default"+  Name = "bactank.example.com Default"
   JobDefs = "DefaultJob";   JobDefs = "DefaultJob";
   Client = bactank.example.com;   Client = bactank.example.com;
Line 634: Line 655:
   Pool = Default   Pool = Default
   Type = Restore   Type = Restore
-  Client=bactank.home.defcon.no+  Client=bactank.example.com
   FileSet="Full Set"   FileSet="Full Set"
   Messages = Standard   Messages = Standard
Line 640: Line 661:
 } }
 </code> </code>
 +
 ==== Console ==== ==== Console ====
  
Line 721: Line 743:
 <code> <code>
 Run Backup job Run Backup job
-JobName:  bactank.home.defcon.no Default+JobName:  bactank.example.com Default
 Level:    Incremental Level:    Incremental
-Client:   bactank.home.defcon.no+Client:   bactank.example.com
 FileSet:  Full Set FileSet:  Full Set
 Pool:     Default (From Job resource) Pool:     Default (From Job resource)
Line 749: Line 771:
 Select level (1-5): 1 Select level (1-5): 1
 Run Backup job Run Backup job
-JobName:  bactank.home.defcon.no Default+JobName:  bactank.example.com Default
 Level:    Full Level:    Full
-Client:   bactank.home.defcon.no+Client:   bactank.example.com
 FileSet:  Full Set FileSet:  Full Set
 Pool:     Default (From Job resource) Pool:     Default (From Job resource)
Line 769: Line 791:
 | JobId | Name                           | StartTime           | Type | Level | JobFiles | JobBytes | JobStatus | | JobId | Name                           | StartTime           | Type | Level | JobFiles | JobBytes | JobStatus |
 +-------+--------------------------------+---------------------+------+-------+----------+----------+-----------+ +-------+--------------------------------+---------------------+------+-------+----------+----------+-----------+
-|     1 | bactank.home.defcon.no Default | 2010-10-31 23:58:22 | B    | F     |        0 |        0 | R         |+|     1 | bactank.example.com Default | 2010-10-31 23:58:22 | B    | F     |        0 |        0 | R         |
 +-------+--------------------------------+---------------------+------+-------+----------+----------+-----------+ +-------+--------------------------------+---------------------+------+-------+----------+----------+-----------+
 </code> </code>
Line 888: Line 910:
 +----------+------------------------+---------------+--------------+ +----------+------------------------+---------------+--------------+
 </code> </code>
 +===== Adding a remote windows client =====
  
 +Setup of a Windows Client starts out by downloading the appropriate installer from the Win32_64 section of http://bacula.org/en/?page=downloads
  
-===== Adding a remote windows client =====+When you run the installer, you can choose to install the Client, Consoles, Documentation and any combination thereof. Naturally, since we want to take backups of our Windows system, make sure Client is checked. 
 + 
 +If you chose to also install Consoles (bconsole/bat), you will be prompted for Director configuration for the console. 
 + 
 +:!: You will not be prompted for information about the File Service, however! 
 + 
 +After installation, an example Client definition for your system will be placed at the root of your ''C:'' drive. There is not much in that file that you could not have figured out on your own by now :) 
 + 
 +You will, however, want to have a look at the ''bacula-fd.conf'' file. The easiest way to get at that file, is to open your Start meny, locate the Bacula program group, where you'll find ''Configuration / Edit Client Configuration''. Choosing this will open the conf file using Wordpad/Write. 
 + 
 +<code> 
 +FileDaemon {                            # this is me 
 +  Name = windows.example.com 
 +  FDport = 9102                # where we listen for the director 
 +  WorkingDirectory = "C:\\Program Files\\Bacula\\working" 
 +  Pid Directory = "C:\\Program Files\\Bacula\\working" 
 +# Plugin Directory = "C:\\Program Files\\Bacula\\plugins" 
 +  Maximum Concurrent Jobs = 10 
 +} 
 + 
 +# List Directors who are permitted to contact this File daemon 
 +Director { 
 +  Name = bactank-dir 
 +  Password = "random-generated-string-replace-this" 
 +} 
 + 
 +# Restricted Director, used by tray-monitor to get the 
 +#   status of the file daemon 
 +# This is a local system tray application on Windows. 
 +Director { 
 +  Name = windows.example.com-mon 
 +  Password = "random-generated-string" 
 +  Monitor = yes 
 +} 
 + 
 +# Send all messages except skipped files back to Director 
 +Messages { 
 +  Name = Standard 
 +  director = bactank-dir = all, !skipped, !restored 
 +} 
 +</code> 
 + 
 +So, as you probably notice, absolutely nothing really magical about that. The most magical-looking may be the Windows specific path-specifications, and the hint "Plugin Directory". All Client platforms support the concept of Plugins, but perhaps the platform where this is most relevant to mention is Windows. Simply because a lot of Windows servers out there exist to run Microsoft Exchange. Bacula has a plugin that handles Exchange way better than just doing file-level dumps. The plugin is shipped in the Win32/Win64 installer package, so if you are interested in this functionality, read up on the relevant [[http://www.bacula.org/manuals/en/concepts/concepts/New_Features.html#SECTION003200000000000000000|Bacula New Features section]]. 
 + 
 +We move on to adding the client to the Director, by adding the file 
 +''/etc/bacula/clients.d/windows.example.com.conf''. First, we put in the Client definition: 
 +<code> 
 +Client { 
 +  Name = windows.example.com; 
 +  Address = windows.example.com; 
 +  FDPort = 9102;  
 +  Catalog = StandardCatalog; 
 +  Password = "random-generated-string-replace-this"; 
 +  AutoPrune = yes;  # Prune expired Jobs/Files 
 +}    
 +</code> 
 + 
 +Nothing new here either, a simple Client definition like the ones we have seen before. The FileSet, however, is going to be a lot different. I have only one Windows system that I want backups of, so I put my windows-specific FileSet in 
 +the clients.d/client-name.conf file. If you have multiple Windows computers 
 +that you want to do, with the same requirements for the FileSet, you should 
 +put it in the global Director file instead. 
 + 
 +<code> 
 + 
 +FileSet { 
 +  Name = "Windows Full"; 
 +  # With Portable = no, a Windows backup will only be possible to 
 +  # restore on WinNT4+, but saves and restores full NTFS ACL's 
 +  Include { 
 +    Options { 
 +      Signature = MD5 
 +       
 +      # Windows has no concept of sane case handling in filenames ;) 
 +      IgnoreCase = yes 
 +       
 +      # Setting Portable to No means this backup cannot be restored to 
 +      # any operating system other than the Windows NT family (WinNT/XP/2K3/2K8), 
 +      # but it also means that VSS can be used and that NTFS ACL will be 
 +      # backed up and restored correctly. 
 +      Portable = no; 
 +      Checkfilechanges=yes 
 + 
 +      # The 'Exclude = yes' statement means that any WildFile, 
 +      # WildDir, File or other path spec will be excluded from 
 +      # the paths defined in _this_ include section. 
 +      # Exclude is used in the Options part of the Include section 
 +      # because we want to use wildcards... 
 +      Exclude = yes; 
 + 
 +      # Exclude Mozilla-based programs' file caches 
 +      WildDir = "[A-Z]:/Documents and Settings/*/Application Data/*/Profiles/*/*/Cache" 
 +      WildDir = "[A-Z]:/Documents and Settings/*/Application Data/*/Profiles/*/*/Cache.Trash" 
 + 
 +      # Exclude user's registry files - they're always in use anyway. 
 +      WildFile = "[A-Z]:/Documents and Settings/*/Local Settings/Application Data/Microsoft/Windows/usrclass.*" 
 +      WildFile = "[A-Z]:/Documents and Settings/*/ntuser.*" 
 +      # Exclude directories full of lots and lots of useless little per-user files 
 +      WildDir = "[A-Z]:/Documents and Settings/*/Cookies" 
 +      WildDir = "[A-Z]:/Documents and Settings/*/Recent" 
 +      WildDir = "[A-Z]:/Documents and Settings/*/My Documents/Downloads" 
 +      WildDir = "[A-Z]:/Documents and Settings/*/Local Settings/History" 
 +      WildDir = "[A-Z]:/Documents and Settings/*/Local Settings/Temp" 
 +      WildDir = "[A-Z]:/Documents and Settings/*/Local Settings/Temporary Internet Files" 
 +    
 +      # These are always open and unable to be backed up 
 +      WildFile = "[A-Z]:/Documents and Settings/All Users/Application Data/Microsoft/Network/Downloader/qmgr[01].dat" 
 +    
 +      # Some random bits of Windows we want to ignore 
 +      WildFile = "[A-Z]:/windows/security/logs/scepol.log" 
 +      WildDir = "[A-Z]:/windows/system32/config" 
 +      WildDir = "[A-Z]:/windows/msdownld.tmp" 
 +      WildDir = "[A-Z]:/windows/Download*" 
 +      WildDir = "[A-Z]:/windows/Internet Logs" 
 +      WildDir = "[A-Z]:/windows/$Nt*Uninstall*" 
 +      WildDir = "[A-Z]:/windows/sysvol" 
 +      WildDir = "[A-Z]:/windows/KB*.log" 
 +    
 +      # Temporary directories & files 
 +      WildDir = "[A-Z]:/windows/Temp" 
 +      WildDir = "[A-Z]:/temp"  
 +      WildFile = "*.tmp" 
 +      WildDir = "[A-Z]:/tmp" 
 + 
 +      # Recycle bins 
 +      WildDir = "[A-Z]:/RECYCLER" 
 + 
 +      # Swap files 
 +      WildFile = "[A-Z]:/pagefile.sys" 
 + 
 +      # bacula-fd > 3.0 on win32/64 should be able to use VSS, 
 +      # so we want to avoid copying the VSS metadata during the backup 
 +      File = "\"C:/System Volume Information\"" 
 +       
 +      # Bacula working directory 
 +      WildFile = "C:/Program Files/Bacula/working/*" 
 +    } 
 +    # Include at least the most common root drive 
 +    # Note that all drives must be specified separately, 
 +    # just like different filesystems on Unix. 
 +    # Also note that bacula expects the Unix directory separator, / 
 +    File = "C:/" 
 +  } 
 + 
 +} 
 +</code> 
 + 
 +That ought to have a lot of self-documentation. In this FileSet you also see 
 +the use of Wildcard-definitions. A WildDir is a wildcard for a directory whose 
 +content are to be included/excluded, a WildFile names precicely that, a wildcard for files. Note that Windows FileSets use the Unix directory separator. 
 + 
 +Finally, we add the Job for this client: 
 +<code> 
 +# The full backup for this client. 
 +Job { 
 +  Name = "windows.example.com Default" 
 +  FileSet = "Windows Full" 
 +  JobDefs = "DefaultJob"; 
 +  Client = windows.example.com; 
 +  Write Bootstrap = "/var/lib/bacula/bactank.bsr"; 
 +} 
 +</code> 
 + 
 +A word of caution. My Windows client normally runs F-Secure antivirus. With its realtime scanner active, my backups were crazy slow (<600kbit/sec) making my backups take forever. Disabling the scanner boosted the speed up to normal speeds. I am investigating ways to either "white-list" the bacula-fd process from the scanner, or temporarily disabling it during backups. 
 + 
 +Use a bconsole to reload the configuration, and test it by running an ''estimate''. And that is all there is to it. Windows-clients are really not that different from Unix clients :) 
 + 
 +===== The MySQL server ===== 
 + 
 +As my Catalog is stored on a remote MySQL server, the parts concering the Catalog dumps in the default configuration files shipped with the Debian packages make little sense. Why pull the data across the wire to the Director server for backup, when the database server needs to be backed up anyway? 
 + 
 +The backup of the database server will introduce another important, and useful, feature of Bacula backups. Pre- and post backup commands. Basically, to create a backup of the MySQL server and its databases, we want to do some database maintenance and dumping, and then back up the files on the server. 
 + 
 +A pre-script is a command that will be started on the client that is to be backed up, before any file evaluation is done. If the script fails (returns non-zero), the backup will fail. A post-script may similarly be used to clean up after backups. 
 + 
 +So, I'll be using a pre-script to dump databases before running the actual backup. 
 +But, I'm using a mixed method for dumping. I'm using MySQL's [[http://dev.mysql.com/doc/refman/5.0/en/binary-log.html|binary logging]], allowing atomic change-sets and restores, and backing those up on Increments. On Full and Differential backups, I flush the bin-log, and do a full backup of the databases. 
 + 
 +First, a note on my mysql binary logging setup. I have created a separate directory for the binary logs: 
 +<code> 
 +mkdir -p /var/lib/mysql_binlog 
 +chown mysql.mysql /var/lib/mysql_binlog 
 +</code> 
 + 
 +The binary log also needs to be configured, by default there is no binary logging enabled on Debian installatoions. So in the [mysqld] section of ''/etc/mysql/my.cnf'', I have the followin lines: 
 +<code> 
 +log_bin                 = /var/lib/mysql_binlog/database 
 +log_bin_index           = /var/lib/mysql_binlog/index 
 +expire_logs_days        = 10 
 +max_binlog_size         = 500M 
 +</code> 
 + 
 +So, with binary logging available on the MySQL server, it is time to present the script that will be run on the client before data-transfer. This script is written so it contains the MySQL root password, so make sure you place it safe, and set applicable permissions on it. Alternatively, remove any password (and related command options) from the file, and set up a suitable my.cnf instead. 
 + 
 +<code> 
 +#!/bin/sh 
 +# Filename: /etc/bacula/scripts/mysqldump.sh 
 +# Based on a mix of MySQL recommendations, example scripts in the 
 +# Bacula Wiki and personal experience. 
 +# 
 +# This script is not provided with ANY warranty. 
 +# It is provided as an example, by Jon Langseth (fishy at defcon dot no) 
 +# and is supplied as-is to the public domain. 
 + 
 +# Just a security measure... 
 +cd /tmp 
 + 
 +# The backup level is supplied as an argument to the script.. 
 +LEVEL=$1 
 + 
 +# Bail out if no Level is given... 
 +if [ "${LEVEL}" == "" ]; 
 +then 
 +        echo "No backup level given. Supply Full, Differential or Incremental" 
 +        exit 1; 
 +fi 
 + 
 +# Directory to store backups in 
 +DST=/var/local/backup 
 + 
 +# A regex, passed to egrep -v, for which databases to ignore 
 +IGNREG='^snort$' 
 + 
 +# The MySQL username and password 
 +DBUSER=root 
 +DBPASS=databaserootpassword 
 + 
 +# Any backups older than KEEPDAYS will be deleted first 
 +# Cleanup is only done on Full or Differential backups 
 +KEEPDAYS=14 
 + 
 +if [ "${LEVEL}" == "Full" -o "${LEVEL}" == "Differential" ] 
 +then 
 +  echo "Full/Differential" 
 + 
 +        # Clean up in the destination directory, to avoid clogging the filesystem 
 +        find ${DST} -type f -mtime +${KEEPDAYS} -exec rm -f {} \; 
 +        rmdir $DST/* 2>/dev/null 
 + 
 +        # Create a new directory for the dump, in a DTG-format. 
 +        DATE=$(date  +%Y-%m-%d-%H%M%S) 
 +        mkdir -p ${DST}/${DATE} 
 + 
 +        # Use the 'show databases;' command using mysql in silent mode 
 +        # to get the names of all databases on the server. Iterate 
 +        # over all the databases with a simple for-loop. 
 +        # WARNING: This, and the later dump, may choke on "esoteric" database names. 
 +        for db in $(echo 'show databases;' | mysql -s -u ${DBUSER} -p${DBPASS} | egrep -v ${IGNREG}) ; do 
 +                echo -n "Backing up ${db}... " 
 +                # Dump the databases individually, flushing all logs in the process. 
 +                mysqldump \ 
 +                        --single-transaction \ 
 +                        --add-drop-table \ 
 +                        --add-drop-database \ 
 +                        --add-locks \ 
 +                        --allow-keywords \ 
 +                        --complete-insert \ 
 +                        --create-options \ 
 +                        --flush-logs \ 
 +                        --master-data=2 \ 
 +                        --delete-master-logs \ 
 +                        --user=${DBUSER} \ 
 +                        --password=${DBPASS} \ 
 +                        --databases $db | gzip -c > ${DST}/${DATE}/${db}.txt.gz 
 +                echo "Done." 
 +        done 
 +else 
 +        echo "Incremental" 
 +        # Simply flush the binary logs. An incremental restore can then 
 +        # be performed by replaying the log files. 
 +        mysqladmin --user=${DBUSER} --password=${DBPASS} flush-logs 
 +fi 
 + 
 +# Exit cleanly. 
 +exit 0 
 +</code> 
 + 
 +So, binary logging is enabled on the MySQL server, and the backup script in place, install the bacula-fd package on the database server, and start setting up a 
 +Client definition like usual. 
 + 
 +<code> 
 +Client { 
 +  Name = database.example.com; 
 +  Address = database.example.com; 
 +  FDPort = 9102; 
 +  Catalog = StandardCatalog; 
 +  Password = "random-password-for-fd-director"; 
 +  AutoPrune = yes; 
 +} 
 +</code> 
 + 
 +What is more interesting, is the one line added to the Job definition: 
 + 
 +<code> 
 +Job { 
 +  Name = "database.example.com Default" 
 +  FileSet = "Full Set" 
 +  JobDefs = "DefaultJob"; 
 +  Client = database.example.com; 
 + 
 +  Client Run Before Job = "/etc/bacula/scripts/mysqldump.sh %l"; 
 + 
 +  Write Bootstrap = "/var/lib/bacula/bactank.bsr"; 
 +} 
 +</code> 
 + 
 +There are several ways to run scripts/programs from the Job definition 
 +  * **RunScript** A section-definition for scripts to run on the **Director** 
 +  * **Run Before Job** Path and arguments to a script to run on the **Director** before the data-part of the Job is started. 
 +  * **Run After Job** Path and arguments to a script to run on the **Director** after the data-part of the Job is done 
 +  * **Client Run Before Job** Path and arguments to a script to run on the **Client** before the data-part of the Job is started. 
 +  * **Client Run After Job** Path and arguments to a script to run on the **Client** after the data-part of the Job is done 
 +  * **Run After Failed Job** Path and arguments to a script to run on the **Director** after a Job is detected as Failed. 
 + 
 +To these scripts, there are several variables that can be used as arguments with the commands. Most importantly for Client Pre commands: 
 +<code> 
 +    %% = % 
 +    %c = Client's name 
 +    %d = Director's name 
 +    %i = JobId 
 +    %j = Unique Job id 
 +    %l = Job Level 
 +    %n = Job name 
 +    %s = Since time 
 +    %t = Job type (Backup, ...) 
 +</code>   
 + 
 +As seen in my Job definition, after running the Client-Run-Before script, the rest of the Job is simply doing a file-based backup.  
 + 
 +If you remember the "Full Set" FileSet, I have Excluded ''/var/lib/mysql'' from the FileSet. This means that for a full Restore of this database server, I will have to start the Restore by pulling in the last Full, then merging inn a Differential if that has run after the Full. After that, I will have to reinstall the MySQL server, tank in the SQL files from ''/var/local/backups''. Finally, to get the database up to the latest increment, I'll have to replay all needed log-files.
  
-FIXME Missing.+Documentation related to the pre- and post-scripts are located in the [[http://bacula.org/5.0.x-manuals/en/main/main/Configuring_Director.html#SECTION001830000000000000000|Job definition documentation]]
  
 ===== Links, references, scratch ===== ===== Links, references, scratch =====
 
You are here: start » guides » bacup » bacula
Trace:
guides/bacup/bacula.1288626440.txt.gz · Last modified: 2010/11/01 16:47 by fishy