Changes RSS

====== Differences ====== This shows you the differences between two versions of the page.

Link to this comparison view

guides:bacup:bacula [2010/10/31 22:29]
fishy 		guides:bacup:bacula [2010/11/06 16:28] (current)
fishy Untested change to the Windows FileSet.
Line 173: Line 173:
 Since Bacula is separated into different components that can live completely Since Bacula is separated into different components that can live completely
 separate, configuration of these components are split into respecitve configuration files. Needless to say, these configuration files will relate to each other, enabling communication between the components. Here is an attempt at visualizing the relations: separate, configuration of these components are split into respecitve configuration files. Needless to say, these configuration files will relate to each other, enabling communication between the components. Here is an attempt at visualizing the relations:
 +
 {{:guides:bacup:bacula-config-relations.png?500|}} {{:guides:bacup:bacula-config-relations.png?500|}}
  
Line 193: Line 194:
   * I will be using one FileSet for each client   * I will be using one FileSet for each client
   * On the host __bactank__ I will be excluding /opt completely, and store VTL files and File-based "volumes" under that directory.   * On the host __bactank__ I will be excluding /opt completely, and store VTL files and File-based "volumes" under that directory.
 +
 +:!: **NOTE**
 +
 +In the Debian configuration files, the passwords seem like they are auto-generated. **They are not**. Please generate new passwords wherever appropriate. To be relatively random, you can use pwgen like this:
 +<code>
 +pwgen -s 32 1
 +AwAdLG9doWqJuQgZ0BmnMViX8lCgAH7S 
 +</code>
 ==== Storage Daemon ==== ==== Storage Daemon ====
  
Line 358: Line 367:
 Messages sections in the other files. in the other files, the Messages section Messages sections in the other files. in the other files, the Messages section
 typically describes what director should recieve what messages. On the Director, we'll be using Messages sections to actually do something with those messages. typically describes what director should recieve what messages. On the Director, we'll be using Messages sections to actually do something with those messages.
 +
 +=== A read-only console for monitoring ===
 +
 +<code>
 +Console {
 +  Name = bactank-mon
 +  Password = "shared-secret-password-used-in-console-config"
 +  CommandACL = status, .status; # Allow only status-reading.
 +}
 +</code>
  
 === Catalog === === Catalog ===
Line 436: Line 455:
 </code> </code>
  
-The comments say "N.B. Use a fully qualified name here", but in reality, anything that ends up with the IP that the Storage Daemon listens on can be used. Note that the Storage definition for the autochanger references the Autochanger as Device, not the individual tape drives. It is the responsibility of the Storage Daemon to represent the changer correctly. Also, remember the notes about Media Type when we configured the Storage Daemon.+The comments say "N.B. Use a fully qualified name here", but in reality, anything that ends up with the IP that the Storage Daemon listens on can be used. Observe that the name used here, will be reported to File Servers, and they will use it to communicate with the Storage Daemon. So all Clients will also need to be able to get an IP from what you enter here:!: 
 + 
 +Note that the Storage definition for the autochanger references the Autochanger as Device, not the individual tape drives. It is the responsibility of the Storage Daemon to represent the changer correctly. Also, remember the notes about Media Type when we configured the Storage Daemon.
  
 === Pools === === Pools ===
Line 453: Line 474:
   AutoPrune = yes              # Prune expired volumes   AutoPrune = yes              # Prune expired volumes
   Volume Retention = 4 months  # 1/3 year   Volume Retention = 4 months  # 1/3 year
 +  File Retention = 1 months
 +  Job Retention = 2 months
   RecyclePool = Scratch        # Move to this pool when markec Recyclable   RecyclePool = Scratch        # Move to this pool when markec Recyclable
   Cleaning Prefix = CLN        # If cleaning tapes are available, they have this pfx.   Cleaning Prefix = CLN        # If cleaning tapes are available, they have this pfx.
Line 464: Line 487:
   AutoPrune = yes              # Prune expired volumes   AutoPrune = yes              # Prune expired volumes
   Volume Retention = 24 months # 2 years   Volume Retention = 24 months # 2 years
 +  File Retention = 9 months
 +  Job Retention = 12 months
   RecyclePool = Scratch        # Move to this pool when markec Recyclable   RecyclePool = Scratch        # Move to this pool when markec Recyclable
   Cleaning Prefix = CLN        # If cleaning tapes are available, they have this pfx.   Cleaning Prefix = CLN        # If cleaning tapes are available, they have this pfx.
Line 477: Line 502:
 </code> </code>
  
-Volume Retention needs to be set fairly high, at least higher than any File or Job retention set in later sections. The retention periods define how long data about a given Volume/Job/File is to be kept in the Catalog, and as such, how much time will pass before a Volume Expires...+Volume Retention needs to be set fairly high, at least higher than any File or Job retention. The retention periods define how long data about a given Volume/Job/File is to be kept in the Catalog, and as such, how much time will pass before a Volume Expires...
  
 Please look at the [[http://www.bacula.org/5.0.x-manuals/en/main/main/Catalog_Maintenance.html#SECTION004510000000000000000|Setting Retention Periods]] section of the Bacula manual for an explanation. Please look at the [[http://www.bacula.org/5.0.x-manuals/en/main/main/Catalog_Maintenance.html#SECTION004510000000000000000|Setting Retention Periods]] section of the Bacula manual for an explanation.
Line 526: Line 551:
  
 This sets up a "template", where any job using this definition will get: This sets up a "template", where any job using this definition will get:
- * A job type of Backup (other types are: Resore, Verify, Admin) +  * A job type of Backup (other types are: Resore, Verify, Admin) 
- * Uses the "Full Set" FileSet definition (will be described later) +  * Uses the "Full Set" FileSet definition (will be described later) 
- * Runs according to the "Weekly Schedule" described above. +  * Runs according to the "WeeklySchedule" described above. 
- * Uses the Standard message-handler. +  * Uses the Standard message-handler. 
- * Uses media from the pool named Default. +  * Uses media from the pool named Default. 
- * Has a Priority of 10 (Higher values means jobs will run later) +  * Has a Priority of 10 (Higher values means jobs will run later) 
- * Will Upgrade the next job to the Type of a job that prevoiusly failed.+  * Will Upgrade the next job to the Type of a job that prevoiusly failed.
  
 +=== FileSet ===
 +
 +I add generic FileSets in the common configuration, before Clients and Jobs, simply because they are meant to be just that: generic. If I need to specify what and how to do the files of a given Client or Job more precicely, I put that specific definition along with the Client and/or Job definition.
 +
 +Here, I'll list my sort-of generic Unix-related FileSet that will try it's best at doing a backup of a complete Unix filesystem, as long as the Client using this in a Job has all files in a single partition.
 +
 +<code>
 +FileSet {
 +  Name = "Full Set"
 +#  Note: / backs up everything on the root partition.
 +#    if you have other partitons such as /usr or /home
 +#    you will probably want to add them too.
 +  Include {
 +    Options {
 +      signature = MD5
 +    }
 +    File = /
 +    File = /boot
 +  }
 +  Exclude {
 +    File = /proc
 +    File = /tmp
 +    File = /sys
 +    File = /dev
 +    File = /.journal
 +    File = /.fsck
 +    # I typically define /var/lib/bacula as the
 +    # WorkingDirectory for the File Daemon.
 +    File = /var/lib/bacula
 +    
 +    # For mysql servers, it is better to dump and restore
 +    # and/or do binlog replay. So, instead of dumping the
 +    # MySQL data directory, do dull database dumps to
 +    # a different location, and use binlogs in addition if
 +    # real incremental MySQL backups are needed/wanted.
 +    File = /var/lib/mysql
 +
 +    # Excluding files/directories that do not exist
 +    # has no effect other than making the FileSet generic..
 +    File = /opt/vtl
 +    File = /opt/bacula-filestore
 +  }
 +}
 +</code>
 +
 +FileSets can be buildt VERY complex, this is an attempt at a fairly manageable base-definition. For more details, and more complex examples, look at the relatively long sections in the [[http://www.bacula.org/5.0.x-manuals/en/main/main/Configuring_Director.html#SECTION001870000000000000000|Bacula Director manual]]
 +
 +=== Client and Job in bacula-dir.conf ===
 +
 +Now we actually have a very-close-to useful configuration. The only parts missing are Clients and Jobs. If you remember from far up in the document, One Job may only reference One Client, so even though there is a One-to-Many relationship between Clients and Jobs (one Client may have multiple Jobs associated) each Job is directly tied to one, and only one, Client.
 +
 +This means that grouping Client definitions and Job definitions together is very natural. So natural in fact, that I'll use file-inclusion features in Bacula configrations to create a separate configuration file for each client.
 +
 +But, there is one Client, with its associated Jobs, that is natural to include in the bacula-dir.conf file, and that is the Client definition for the bacula server, or localhost if you wish.
 +
 +Remember from the bacula-fd.conf that we set up, I used the Name ''bactank.example.com'', and used the random-generated Password directive. Using those two strings:
 +
 +<code>
 +# Client (File Services) to backup
 +Client {
 +  Name = bactank.example.com;
 +  Address = bactank;
 +  FDPort = 9102;
 +  Catalog = StandardCatalog;
 +  Password = "random-generated-password-identical-to-fd-conf";
 +  AutoPrune = yes;  # Prune expired Jobs/Files
 +}
 +</code>
 +
 +Now, we'll add the absoutely basic Job for this Client: a full backup.
 +
 +<code>
 +Job {
 +  Name = "bactank.example.com Default"
 +  JobDefs = "DefaultJob";
 +  Client = bactank.example.com;
 +  Write Bootstrap = "/var/lib/bacula/bactank.bsr";
 +}
 +</code>
 +
 +As this uses the DefaultJob JobDef/template, this will use the ''Full Set'' standard for Unixes as its FileSet, it will run using the ''WeeklySchedule'', backing up data to the ''Default'' pool, and reporting messages to the ''Standard'' message facility.
 +
 +FIXME Needs a FileSet and Job definition for dumping the Catalog, and backing that up.
 +
 +=== One common Job ===
 +
 +We'll add one Job that gets tied to the first Client, but will in actuallity be modified in the Console applications when it gets run. This Job definition is the
 +''RestoreFiles'' job. Because all jobs must be defined before they can be used, and all Jobs must be tied to a Client, this gets put in the global config.
 +
 +But, this Job will (should) never be run as-is. This job exists as a template to be used when starting a restore job. In the console, You'll modify each and every spec in this Job, to match the Restore that will be done.
 +
 +<code>
 +Job {
 +  Name = "RestoreFiles"
 +  Pool = Default
 +  Type = Restore
 +  Client=bactank.example.com
 +  FileSet="Full Set"
 +  Messages = Standard
 +  Where = /nonexistant/path/to/file/archive/dir/bacula-restores
 +}
 +</code>
 +
 +==== Console ====
 +
 +There is one more configuration file to take a look at, before we are done with the basic configuration, and that is the configuration of the Bacula Console locally on the Director server.
 +
 +This file is named ''/etc/bacula/bconsole.conf'' and is very simple:
 +<code>
 +Director {
 +  Name = localhost-dir;
 +  DIRport = 9101;
 +  address = localhost;
 +  Password = "random-generated-password-from-director-section-in-dir-conf";
 +}
 +</code>
 +===== Starting the components =====
 +
 +What order you start the components in should be absolutely irrelevant, because the individual Components will not try to communicate before they need to. E.g. the Director will not contact the SD before a Storage operation needs to be done, or a FD before a Job needs to communicate with the Client.
 +
 +But to be on the safe side:
 +<code>
 +/etc/init.d/bacula-sd start
 +/etc/init.d/bacula-fd start
 +/etc/init.d/bacula-director start
 +</code>
 ===== Tapes/volumes in the Media database / pools from an autochanger ===== ===== Tapes/volumes in the Media database / pools from an autochanger =====
  
 +To get some Tape Volumes to work with, we start by doing a load-and read query to the autochanger, to initialize and inventory the changer.
 +
 +Start the Bacula console on the Director server host
 <code> <code>
-label pool=Scratch slots=1-22 barcodes+bconsole
 </code> </code>
 +
 +Run the update/inventory:
 +<code>
 +update slots storage=MHVTL drive=0 scan
 +</code>
 +
 +The ''update slots'' command will output a whole bunch of "Read error" errormessages. This is normal, the VTL simulates an unlabeled/uninitialized tape,
 +and that is what we want when running this command against a fresh VTL.
 +
 +So, now that we are sure that we have a correctly initialized library with a
 +fresh inventory, 
 +
 +To be sure everything is fresh, and note the slots that we want to add to our Media index, the next step becomes:
 +<code>
 +status slots storage=MHVTL drive=0
 +</code>
 +
 +As long as none of the Slots come up with an assigned Pool (or status/media for that matter), we can safely Label the Volumes. To automatically Label Volumes using their "barcode", use:
 +
 +<code>
 +label storage=MHVTL drive=0 pool=Scratch slots=1-22 barcodes
 +</code>
 +
 +Notice that this adds the Volumes to the Scratch pool, where they will reside until they are needed.
 +===== Run your first job =====
 +
 +Start bconsole (if you exited it earlier)
 +<code>
 +bconsole
 +</code>
 +
 +Do the absolutely simplest ''run'' command possible:
 +<code>
 +run
 +</code>
 +
 +This will show you a list of availabe Job Resources to run. If you haven't added anything beyond my example here, you'll get something like:
 +<code>
 +A job name must be specified.
 +The defined Job resources are:
 +     1: bactank.example.com Default
 +     2: RestoreFiles
 +Select Job resource (1-2): 1
 +</code>
 +
 +I selected the first resource, as I want to run the first backup. Doin a restore at this point makes no sense ;)
 +
 +Now, this will list the settings for the job. I'll simply show how I modified the setings from an Incremental to a Full job:
 +<code>
 +Run Backup job
 +JobName:  bactank.example.com Default
 +Level:    Incremental
 +Client:   bactank.example.com
 +FileSet:  Full Set
 +Pool:     Default (From Job resource)
 +Storage:  MHVTL (From Pool resource)
 +When:     2010-10-31 23:53:48
 +Priority: 10
 +OK to run? (yes/mod/no): mod
 +Parameters to modify:
 +     1: Level
 +     2: Storage
 +     3: Job
 +     4: FileSet
 +     5: Client
 +     6: When
 +     7: Priority
 +     8: Pool
 +     9: Plugin Options
 +Select parameter to modify (1-9): 1
 +Levels:
 +     1: Full
 +     2: Incremental
 +     3: Differential
 +     4: Since
 +     5: VirtualFull
 +Select level (1-5): 1
 +Run Backup job
 +JobName:  bactank.example.com Default
 +Level:    Full
 +Client:   bactank.example.com
 +FileSet:  Full Set
 +Pool:     Default (From Job resource)
 +Storage:  MHVTL (From Pool resource)
 +When:     2010-10-31 23:53:48
 +Priority: 10
 +OK to run? (yes/mod/no): yes
 +Job queued. JobId=1
 +</code>
 +
 +We can now see that it is indeed running:
 +<code>
 +list jobs
 +</code>
 +<code>
 ++-------+--------------------------------+---------------------+------+-------+----------+----------+-----------+
 +| JobId | Name                           | StartTime           | Type | Level | JobFiles | JobBytes | JobStatus |
 ++-------+--------------------------------+---------------------+------+-------+----------+----------+-----------+
 +|     1 | bactank.example.com Default | 2010-10-31 23:58:22 | B    | F     |        0 |        0 | R         |
 ++-------+--------------------------------+---------------------+------+-------+----------+----------+-----------+
 +</code>
 +
 +In this first dump, I simply assumed that communication worked, and that I would have enough storage space for the backup. In a more proper scenario, you should have used the command ''estimate'':
 +
 +<code>
 +estimate job="bactank.example.com Default"
 +</code>
 +<code>
 +Using Catalog "StandardCatalog"
 +Connecting to Client bactank.example.com at bactank:9102
 +2000 OK estimate files=41,763 bytes=855,995,113
 +</code>
 +
 +I knew that I had set MHVTL up to use tape-files of 15GB size, and that I had 390GB available on the LVM-volume where this gets stored, so handling ~850MB of data would be no problem. I also added Compression to the mhvtl setup, so the resulting storage use was:
 +
 +<code>
 +bactank:~# ls -lh /opt/vtl/TAPE01L4 
 +-rw-rw---- 1 vtl vtl 512M 2010-11-01 00:01 /opt/vtl/TAPE01L4
 +</code>
 +
 +So, 850MB got compressed down to 512MB. I could push this way further down,
 +I have used compression level 1 of 9 in my MHVTL configuration.
 +
 +===== Adding a remote linux client =====
 +
 +Configuration and control of bacula clients / File Servers are done at the Director. But before we configure the Director, we'll start by installing the software.
 +
 +==== On the client ====
 +
 +What we need to do at the client, is installing and configuring the Bacula File Daemon. On a Debian system, we pull the package in using apt-get((Notice that I skipped setting up lenny-backports. A higher-level director is compatible with an older file daemon)).
 +
 +<code>
 +sudo apt-get -y install bacula-fd
 +</code>
 +
 +Next, set up the ''/etc/bacula/bacula-fd.conf'' file on the Client host:
 +
 +<code>
 +Director {
 +  Name = bactank-dir
 +  Password = "random-password-to-use-in-director-client-def"
 +}
 +
 +# "Global" File daemon configuration specifications
 +FileDaemon {                          # this is me
 +  Name = linuxclient.example.com;
 +  FDport = 9102                  # where we listen for the director
 +  WorkingDirectory = /var/lib/bacula
 +  Pid Directory = /var/run/bacula
 +  Maximum Concurrent Jobs = 20
 +}
 +
 +# Send all messages except skipped files back to Director
 +Messages {
 +  Name = Standard
 +  Director = bactank-dir = all, !skipped, !restored
 +}
 +</code>
 +
 +Remember to update the Director = references to the name of your Director!
 +
 +==== On the Director ====
 +
 +Now, we will start using a wee bit of Bacula configuration file magic! We want to keep the main Director configuration as clean as possible. This may in part be achieved by splitting individual client definitions (Client, Job and possibly FileSet) out into separate perr-client files. Bacula configurations support inclusion of external files, and even inclusion of configuration generated by commands!
 +
 +Since Bacula 2.2.0 you can include the output of a command within a configuration file with the ”@|” syntax. We use this to create a "dot-dee" directory for client configurations. Added to the bottom of the ''/etc/bacula/bacula-dir.conf'' on the Director server:
 +<code>
 +# Include subfiles associated with configuration of clients.
 +# They define the bulk of the Clients, Jobs, and FileSets.
 +# Remember to "reload" the Director after adding a client file.
 +#
 +@|"sh -c 'for f in /etc/bacula/clients.d/*.conf ; do echo @${f} ; done'"
 +</code>
 +
 +The reason we have not added this yet, is because loading the configuration
 +will fail if the directory is empty or non-existant.
 +
 +Before we reload the configuration, we want to add the client in. I prefer to use client-hostname based file-names, so I create the file \\
 +''/etc/bacula/clients.d/linuxclient.example.com.conf'':
 +
 +<code>
 +Client {
 +  Name = linuxclient.example.com;
 +  Address = linuxclient.example.com;
 +  FDPort = 9102;
 +  Catalog = StandardCatalog;
 +  Password = "random-password-to-use-in-director-client-def";
 +  AutoPrune = yes;  # Prune expired Jobs/Files
 +}
 +
 +# The full backup for this client.
 +Job {
 +  Name = "linuxclient.example.com Default"
 +  JobDefs = "DefaultJob";
 +  Client = linuxclient.example.com;
 +  Write Bootstrap = "/var/lib/bacula/bactank.bsr";
 +}
 +</code>
 +
 +More or less identical to the first Client, the "localhost" definition...
 +
 +With that configuration bit in place, we are ready to load up the configuration, and use it. On a bacula console controlling the director:
 +
 +<code>
 +reload
 +list clients
 +</code>
 +<code>
 +Automatically selected Catalog: StandardCatalog
 +Using Catalog "StandardCatalog"
 ++----------+------------------------+---------------+--------------+
 +| ClientId | Name                   | FileRetention | JobRetention |
 ++----------+------------------------+---------------+--------------+
 +|        1 | bactank.example.com    |     5,184,000 |   15,552,000 |
 +|        2 | web.example.com        |             0 |            0 |
 ++----------+------------------------+---------------+--------------+
 +</code>
 +===== Adding a remote windows client =====
 +
 +Setup of a Windows Client starts out by downloading the appropriate installer from the Win32_64 section of http://bacula.org/en/?page=downloads
 +
 +When you run the installer, you can choose to install the Client, Consoles, Documentation and any combination thereof. Naturally, since we want to take backups of our Windows system, make sure Client is checked.
 +
 +If you chose to also install Consoles (bconsole/bat), you will be prompted for Director configuration for the console.
 +
 +:!: You will not be prompted for information about the File Service, however!
 +
 +After installation, an example Client definition for your system will be placed at the root of your ''C:'' drive. There is not much in that file that you could not have figured out on your own by now :)
 +
 +You will, however, want to have a look at the ''bacula-fd.conf'' file. The easiest way to get at that file, is to open your Start meny, locate the Bacula program group, where you'll find ''Configuration / Edit Client Configuration''. Choosing this will open the conf file using Wordpad/Write.
 +
 +<code>
 +FileDaemon {                            # this is me
 +  Name = windows.example.com
 +  FDport = 9102                # where we listen for the director
 +  WorkingDirectory = "C:\\Program Files\\Bacula\\working"
 +  Pid Directory = "C:\\Program Files\\Bacula\\working"
 +# Plugin Directory = "C:\\Program Files\\Bacula\\plugins"
 +  Maximum Concurrent Jobs = 10
 +}
 +
 +# List Directors who are permitted to contact this File daemon
 +Director {
 +  Name = bactank-dir
 +  Password = "random-generated-string-replace-this"
 +}
 +
 +# Restricted Director, used by tray-monitor to get the
 +#   status of the file daemon
 +# This is a local system tray application on Windows.
 +Director {
 +  Name = windows.example.com-mon
 +  Password = "random-generated-string"
 +  Monitor = yes
 +}
 +
 +# Send all messages except skipped files back to Director
 +Messages {
 +  Name = Standard
 +  director = bactank-dir = all, !skipped, !restored
 +}
 +</code>
 +
 +So, as you probably notice, absolutely nothing really magical about that. The most magical-looking may be the Windows specific path-specifications, and the hint "Plugin Directory". All Client platforms support the concept of Plugins, but perhaps the platform where this is most relevant to mention is Windows. Simply because a lot of Windows servers out there exist to run Microsoft Exchange. Bacula has a plugin that handles Exchange way better than just doing file-level dumps. The plugin is shipped in the Win32/Win64 installer package, so if you are interested in this functionality, read up on the relevant [[http://www.bacula.org/manuals/en/concepts/concepts/New_Features.html#SECTION003200000000000000000|Bacula New Features section]].
 +
 +We move on to adding the client to the Director, by adding the file
 +''/etc/bacula/clients.d/windows.example.com.conf''. First, we put in the Client definition:
 +<code>
 +Client {
 +  Name = windows.example.com;
 +  Address = windows.example.com;
 +  FDPort = 9102; 
 +  Catalog = StandardCatalog;
 +  Password = "random-generated-string-replace-this";
 +  AutoPrune = yes;  # Prune expired Jobs/Files
 +}   
 +</code>
 +
 +Nothing new here either, a simple Client definition like the ones we have seen before. The FileSet, however, is going to be a lot different. I have only one Windows system that I want backups of, so I put my windows-specific FileSet in
 +the clients.d/client-name.conf file. If you have multiple Windows computers
 +that you want to do, with the same requirements for the FileSet, you should
 +put it in the global Director file instead.
 +
 +<code>
 +
 +FileSet {
 +  Name = "Windows Full";
 +  # With Portable = no, a Windows backup will only be possible to
 +  # restore on WinNT4+, but saves and restores full NTFS ACL's
 +  Include {
 +    Options {
 +      Signature = MD5
 +      
 +      # Windows has no concept of sane case handling in filenames ;)
 +      IgnoreCase = yes
 +      
 +      # Setting Portable to No means this backup cannot be restored to
 +      # any operating system other than the Windows NT family (WinNT/XP/2K3/2K8),
 +      # but it also means that VSS can be used and that NTFS ACL will be
 +      # backed up and restored correctly.
 +      Portable = no;
 +      Checkfilechanges=yes
 +
 +      # The 'Exclude = yes' statement means that any WildFile,
 +      # WildDir, File or other path spec will be excluded from
 +      # the paths defined in _this_ include section.
 +      # Exclude is used in the Options part of the Include section
 +      # because we want to use wildcards...
 +      Exclude = yes;
 +
 +      # Exclude Mozilla-based programs' file caches
 +      WildDir = "[A-Z]:/Documents and Settings/*/Application Data/*/Profiles/*/*/Cache"
 +      WildDir = "[A-Z]:/Documents and Settings/*/Application Data/*/Profiles/*/*/Cache.Trash"
 +
 +      # Exclude user's registry files - they're always in use anyway.
 +      WildFile = "[A-Z]:/Documents and Settings/*/Local Settings/Application Data/Microsoft/Windows/usrclass.*"
 +      WildFile = "[A-Z]:/Documents and Settings/*/ntuser.*"
 +      # Exclude directories full of lots and lots of useless little per-user files
 +      WildDir = "[A-Z]:/Documents and Settings/*/Cookies"
 +      WildDir = "[A-Z]:/Documents and Settings/*/Recent"
 +      WildDir = "[A-Z]:/Documents and Settings/*/My Documents/Downloads"
 +      WildDir = "[A-Z]:/Documents and Settings/*/Local Settings/History"
 +      WildDir = "[A-Z]:/Documents and Settings/*/Local Settings/Temp"
 +      WildDir = "[A-Z]:/Documents and Settings/*/Local Settings/Temporary Internet Files"
 +   
 +      # These are always open and unable to be backed up
 +      WildFile = "[A-Z]:/Documents and Settings/All Users/Application Data/Microsoft/Network/Downloader/qmgr[01].dat"
 +   
 +      # Some random bits of Windows we want to ignore
 +      WildFile = "[A-Z]:/windows/security/logs/scepol.log"
 +      WildDir = "[A-Z]:/windows/system32/config"
 +      WildDir = "[A-Z]:/windows/msdownld.tmp"
 +      WildDir = "[A-Z]:/windows/Download*"
 +      WildDir = "[A-Z]:/windows/Internet Logs"
 +      WildDir = "[A-Z]:/windows/$Nt*Uninstall*"
 +      WildDir = "[A-Z]:/windows/sysvol"
 +      WildDir = "[A-Z]:/windows/KB*.log"
 +   
 +      # Temporary directories & files
 +      WildDir = "[A-Z]:/windows/Temp"
 +      WildDir = "[A-Z]:/temp" 
 +      WildFile = "*.tmp"
 +      WildDir = "[A-Z]:/tmp"
 +
 +      # Recycle bins
 +      WildDir = "[A-Z]:/RECYCLER"
 +
 +      # Swap files
 +      WildFile = "[A-Z]:/pagefile.sys"
 +
 +      # bacula-fd > 3.0 on win32/64 should be able to use VSS,
 +      # so we want to avoid copying the VSS metadata during the backup
 +      File = "\"C:/System Volume Information\""
 +      
 +      # Bacula working directory
 +      WildFile = "C:/Program Files/Bacula/working/*"
 +    }
 +    # Include at least the most common root drive
 +    # Note that all drives must be specified separately,
 +    # just like different filesystems on Unix.
 +    # Also note that bacula expects the Unix directory separator, /
 +    File = "C:/"
 +  }
 +
 +}
 +</code>
 +
 +That ought to have a lot of self-documentation. In this FileSet you also see
 +the use of Wildcard-definitions. A WildDir is a wildcard for a directory whose
 +content are to be included/excluded, a WildFile names precicely that, a wildcard for files. Note that Windows FileSets use the Unix directory separator.
 +
 +Finally, we add the Job for this client:
 +<code>
 +# The full backup for this client.
 +Job {
 +  Name = "windows.example.com Default"
 +  FileSet = "Windows Full"
 +  JobDefs = "DefaultJob";
 +  Client = windows.example.com;
 +  Write Bootstrap = "/var/lib/bacula/bactank.bsr";
 +}
 +</code>
 +
 +A word of caution. My Windows client normally runs F-Secure antivirus. With its realtime scanner active, my backups were crazy slow (<600kbit/sec) making my backups take forever. Disabling the scanner boosted the speed up to normal speeds. I am investigating ways to either "white-list" the bacula-fd process from the scanner, or temporarily disabling it during backups.
 +
 +Use a bconsole to reload the configuration, and test it by running an ''estimate''. And that is all there is to it. Windows-clients are really not that different from Unix clients :)
 +
 +===== The MySQL server =====
 +
 +As my Catalog is stored on a remote MySQL server, the parts concering the Catalog dumps in the default configuration files shipped with the Debian packages make little sense. Why pull the data across the wire to the Director server for backup, when the database server needs to be backed up anyway?
 +
 +The backup of the database server will introduce another important, and useful, feature of Bacula backups. Pre- and post backup commands. Basically, to create a backup of the MySQL server and its databases, we want to do some database maintenance and dumping, and then back up the files on the server.
 +
 +A pre-script is a command that will be started on the client that is to be backed up, before any file evaluation is done. If the script fails (returns non-zero), the backup will fail. A post-script may similarly be used to clean up after backups.
 +
 +So, I'll be using a pre-script to dump databases before running the actual backup.
 +But, I'm using a mixed method for dumping. I'm using MySQL's [[http://dev.mysql.com/doc/refman/5.0/en/binary-log.html|binary logging]], allowing atomic change-sets and restores, and backing those up on Increments. On Full and Differential backups, I flush the bin-log, and do a full backup of the databases.
 +
 +First, a note on my mysql binary logging setup. I have created a separate directory for the binary logs:
 +<code>
 +mkdir -p /var/lib/mysql_binlog
 +chown mysql.mysql /var/lib/mysql_binlog
 +</code>
 +
 +The binary log also needs to be configured, by default there is no binary logging enabled on Debian installatoions. So in the [mysqld] section of ''/etc/mysql/my.cnf'', I have the followin lines:
 +<code>
 +log_bin                 = /var/lib/mysql_binlog/database
 +log_bin_index           = /var/lib/mysql_binlog/index
 +expire_logs_days        = 10
 +max_binlog_size         = 500M
 +</code>
 +
 +So, with binary logging available on the MySQL server, it is time to present the script that will be run on the client before data-transfer. This script is written so it contains the MySQL root password, so make sure you place it safe, and set applicable permissions on it. Alternatively, remove any password (and related command options) from the file, and set up a suitable my.cnf instead.
 +
 +<code>
 +#!/bin/sh
 +# Filename: /etc/bacula/scripts/mysqldump.sh
 +# Based on a mix of MySQL recommendations, example scripts in the
 +# Bacula Wiki and personal experience.
 +#
 +# This script is not provided with ANY warranty.
 +# It is provided as an example, by Jon Langseth (fishy at defcon dot no)
 +# and is supplied as-is to the public domain.
 +
 +# Just a security measure...
 +cd /tmp
 +
 +# The backup level is supplied as an argument to the script..
 +LEVEL=$1
 +
 +# Bail out if no Level is given...
 +if [ "${LEVEL}" == "" ];
 +then
 +        echo "No backup level given. Supply Full, Differential or Incremental"
 +        exit 1;
 +fi
 +
 +# Directory to store backups in
 +DST=/var/local/backup
 +
 +# A regex, passed to egrep -v, for which databases to ignore
 +IGNREG='^snort$'
 +
 +# The MySQL username and password
 +DBUSER=root
 +DBPASS=databaserootpassword
 +
 +# Any backups older than KEEPDAYS will be deleted first
 +# Cleanup is only done on Full or Differential backups
 +KEEPDAYS=14
 +
 +if [ "${LEVEL}" == "Full" -o "${LEVEL}" == "Differential" ]
 +then
 +  echo "Full/Differential"
 +
 +        # Clean up in the destination directory, to avoid clogging the filesystem
 +        find ${DST} -type f -mtime +${KEEPDAYS} -exec rm -f {} \;
 +        rmdir $DST/* 2>/dev/null
 +
 +        # Create a new directory for the dump, in a DTG-format.
 +        DATE=$(date  +%Y-%m-%d-%H%M%S)
 +        mkdir -p ${DST}/${DATE}
 +
 +        # Use the 'show databases;' command using mysql in silent mode
 +        # to get the names of all databases on the server. Iterate
 +        # over all the databases with a simple for-loop.
 +        # WARNING: This, and the later dump, may choke on "esoteric" database names.
 +        for db in $(echo 'show databases;' | mysql -s -u ${DBUSER} -p${DBPASS} | egrep -v ${IGNREG}) ; do
 +                echo -n "Backing up ${db}... "
 +                # Dump the databases individually, flushing all logs in the process.
 +                mysqldump \
 +                        --single-transaction \
 +                        --add-drop-table \
 +                        --add-drop-database \
 +                        --add-locks \
 +                        --allow-keywords \
 +                        --complete-insert \
 +                        --create-options \
 +                        --flush-logs \
 +                        --master-data=2 \
 +                        --delete-master-logs \
 +                        --user=${DBUSER} \
 +                        --password=${DBPASS} \
 +                        --databases $db | gzip -c > ${DST}/${DATE}/${db}.txt.gz
 +                echo "Done."
 +        done
 +else
 +        echo "Incremental"
 +        # Simply flush the binary logs. An incremental restore can then
 +        # be performed by replaying the log files.
 +        mysqladmin --user=${DBUSER} --password=${DBPASS} flush-logs
 +fi
 +
 +# Exit cleanly.
 +exit 0
 +</code>
 +
 +So, binary logging is enabled on the MySQL server, and the backup script in place, install the bacula-fd package on the database server, and start setting up a
 +Client definition like usual.
 +
 +<code>
 +Client {
 +  Name = database.example.com;
 +  Address = database.example.com;
 +  FDPort = 9102;
 +  Catalog = StandardCatalog;
 +  Password = "random-password-for-fd-director";
 +  AutoPrune = yes;
 +}
 +</code>
 +
 +What is more interesting, is the one line added to the Job definition:
 +
 +<code>
 +Job {
 +  Name = "database.example.com Default"
 +  FileSet = "Full Set"
 +  JobDefs = "DefaultJob";
 +  Client = database.example.com;
 +
 +  Client Run Before Job = "/etc/bacula/scripts/mysqldump.sh %l";
 +
 +  Write Bootstrap = "/var/lib/bacula/bactank.bsr";
 +}
 +</code>
 +
 +There are several ways to run scripts/programs from the Job definition
 +  * **RunScript** A section-definition for scripts to run on the **Director**
 +  * **Run Before Job** Path and arguments to a script to run on the **Director** before the data-part of the Job is started.
 +  * **Run After Job** Path and arguments to a script to run on the **Director** after the data-part of the Job is done
 +  * **Client Run Before Job** Path and arguments to a script to run on the **Client** before the data-part of the Job is started.
 +  * **Client Run After Job** Path and arguments to a script to run on the **Client** after the data-part of the Job is done
 +  * **Run After Failed Job** Path and arguments to a script to run on the **Director** after a Job is detected as Failed.
 +
 +To these scripts, there are several variables that can be used as arguments with the commands. Most importantly for Client Pre commands:
 +<code>
 +    %% = %
 +    %c = Client's name
 +    %d = Director's name
 +    %i = JobId
 +    %j = Unique Job id
 +    %l = Job Level
 +    %n = Job name
 +    %s = Since time
 +    %t = Job type (Backup, ...)
 +</code>  
 +
 +As seen in my Job definition, after running the Client-Run-Before script, the rest of the Job is simply doing a file-based backup. 
 +
 +If you remember the "Full Set" FileSet, I have Excluded ''/var/lib/mysql'' from the FileSet. This means that for a full Restore of this database server, I will have to start the Restore by pulling in the last Full, then merging inn a Differential if that has run after the Full. After that, I will have to reinstall the MySQL server, tank in the SQL files from ''/var/local/backups''. Finally, to get the database up to the latest increment, I'll have to replay all needed log-files.
 +
 +Documentation related to the pre- and post-scripts are located in the [[http://bacula.org/5.0.x-manuals/en/main/main/Configuring_Director.html#SECTION001830000000000000000|Job definition documentation]]
  
 ===== Links, references, scratch ===== ===== Links, references, scratch =====
 
You are here: start » guides » bacup » bacula
Trace:
guides/bacup/bacula.1288560547.txt.gz · Last modified: 2010/10/31 22:29 by fishy