Defcon Kaklkaklkekl (Wiki)

====== Virtual Tape Library using linuxvtl2/mhvtl on Debian ====== ===== About VTL ===== This segment is stolen from the [[http://en.wikipedia.org/wiki/Virtual_tape_library|Wikipedia article on VTL]]: > A virtual tape library (VTL) is a data storage virtualization technology used typically for backup and recovery purposes. A VTL presents a storage component (usually hard disk storage) as tape libraries or tape drives for use with existing backup software. > Virtualizing the disk storage as tape allows integration of VTLs with existing backup software and existing backup and recovery processes and policies. The benefits of such virtualization include storage consolidation and faster data restore processes. For linux, we have an experimental implementation of file-based VTL in the form of [[http://sites.google.com/site/linuxvtl2/|linuxvtl2]], thanks to the work of Mark Harvey. It is based on the work done with the [[http://sg.danny.cz/sg/sdebug26.html|scsi_debug]] adapter driver, and consists of a combination of a kernel character device driver and userspace tools. I will in general reference this implementation by its "official" name, linuxvtl2, and more often by its in-use name, mhvtl (obviously named after Mark Harvey). ===== About this install walk-through ===== There is no package for linuxvtl2 in the official package trees, and linuxvtl2 is to be considered "experimental" as data-formats may change between versions. This means that it will have to be installed from source. In this how-I-did blurb, I was working as 'root' (shrug), and as such, I have not used 'sudo' to perform commands that need root-like permissions. If you are doing this properly, add 'sudo' to your commands as needed. ===== Pre-requisites for compiling and "tape" administration ===== After installing a base Debian system, pull in essential packages for compiling C/C++ and kernel related code. The shortest route: <code> apt-get -y install build-essential apt-get -y install linux-headers-`uname -r` # Because of brainfart in either mhvtl or the XenServer package: ln -s /usr/src/linux-headers-`uname -r` /lib/modules/`uname -r`/build apt-get -y install zlib1g-dev </code> Prepare the environment for tape-library use, by installing the two most essential packages for handling SCSI and autochangers. <code> apt-get -y install lsscsi mtx mt-st scsitools sg3-utils --no-install-recommends </code> ===== Downloading and compiling ===== Pull down the source for linuxvtl2/mhvtl, and unpack it. <code> wget http://sites.google.com/site/linuxvtl2/mhvtl-2010-05-09.tgz?attredirects=0 -O mhvtl-2010-05-09.tgz tar zxvf mhvtl-2010-05-09.tgz </code> MHVTL really-really wants to run as the user 'vtl', having membership in the group 'vtl', and having the users home directory as '/opt/vtl'. So we should satisfy this before compiling the code. The username, group and location can be changed bu going through the Makefiles and changing it. Or, you can just live with the defaults, and mount appropriate storage space as either /opt or /opt/vtl <code> groupadd vtl useradd --system -c "Vitrual Tape Library" -d /opt/vtl -g vtl -m vtl </code> Now, simply compile and install. <code> cd mhvtl-0.16 make make install cd kernel make make install </code> ===== Configuration ===== Configuration files are not __installed__ by default. Rather, they get __generated__ when mhvtl is first started. So, before progressing, start and then stop the "service". After this, I started up the daemon, and ran some testing commands: <code> /etc/init.d/mhvtl start sleep 10 /etc/init.d/mhvtl shutdown rm /opt/vtl/* </code> The default configuration file contained a lot more devices and stuff than I wanted, so I stripped down the config a bit. First, this is my resulting /etc/mhvtl/device.conf file: <code> VERSION: 4 # VPD page format: # <page #> <Length> <x> <x+1>... <x+n> # NAA format is an 8 hex byte value seperated by ':' # Note: NAA is part of inquiry VPD 0x83 # # Each 'record' is separated by one (or more) blank lines. # Each 'record' starts at column 1 # Serial num max len is 10. # Compression: factor X enabled 0|1 # Where X is zlib compression factor 1 = Fastest compression # 9 = Best compression # enabled 0 == off, 1 == on Library: 10 CHANNEL: 03 TARGET: 00 LUN: 00 Vendor identification: SPECTRA Product identification: PYTHON Product revision level: 550V Unit serial number: XYZZY_A NAA: 10:22:33:44:ab:00:00:00 Drive: 11 CHANNEL: 03 TARGET: 01 LUN: 00 Library ID: 10 Slot: 01 Vendor identification: IBM Product identification: ULT3580-TD4 Product revision level: 550V Unit serial number: XYZZY_A1 NAA: 10:22:33:44:ab:00:01:00 Compression: factor 1 enabled 1 Drive: 12 CHANNEL: 03 TARGET: 02 LUN: 00 Library ID: 10 Slot: 02 Vendor identification: IBM Product identification: ULT3580-TD4 Product revision level: 550V Unit serial number: XYZZY_A2 NAA: 10:22:33:44:ab:00:02:00 Compression: factor 1 enabled 1 Drive: 13 CHANNEL: 03 TARGET: 03 LUN: 00 Library ID: 10 Slot: 03 Vendor identification: IBM Product identification: ULT3580-TD4 Product revision level: 550V Unit serial number: XYZZY_A3 NAA: 10:22:33:44:ab:00:03:00 Compression: factor 1 enabled 1 Drive: 14 CHANNEL: 03 TARGET: 04 LUN: 00 Library ID: 10 Slot: 04 Vendor identification: IBM Product identification: ULT3580-TD4 Product revision level: 550V Unit serial number: XYZZY_A4 NAA: 10:22:33:44:ab:00:04:00 Compression: factor 1 enabled 1 </code> I then changed the "contents" of the defined library (/etc/mhvtl/library_contents.10): <code> Drive 1: Drive 2: Drive 3: Drive 4: Picker 1: MAP 1: MAP 2: MAP 3: MAP 4: # Slot 1 - ?, no gaps # Slot N: [barcode] # [barcode] # a barcode is comprised of three fields: [Leading] [identifier] [Trailing] # Leading "CLN" -- cleaning tape # Leading "W" -- WORM tape # Leading "NOBAR" -- will appear to have no barcode # If the barcode is at least 8 character long, then the last two characters are Trailing # Trailing "S3" - SDLT600 # Trailing "X4" - AIT-4 # Trailing "L1" - LTO 1 # Trailing "TA" - T10000+ # Trailing "JA" - 3592+ # Trailing "JB" - 3592E05+ # Trailing "JW" - WORM 3592+ # Trailing "JX" - WORM 3592E05+ # Slot 1: TAPE01L4 Slot 2: TAPE02L4 Slot 3: TAPE03L4 Slot 4: TAPE04L4 Slot 5: TAPE05L4 Slot 6: TAPE06L4 Slot 7: TAPE07L4 Slot 8: TAPE08L4 Slot 9: TAPE09L4 Slot 10: TAPE10L4 Slot 11: TAPE11L4 Slot 12: TAPE12L4 Slot 13: TAPE13L4 Slot 14: TAPE14L4 Slot 15: TAPE15L4 Slot 16: TAPE16L4 Slot 17: TAPE17L4 Slot 18: TAPE18L4 Slot 19: TAPE19L4 Slot 20: TAPE20L4 Slot 21: CLN000L4 Slot 22: CLN001L4 </code> ===== Testing the setup ===== After this, I started up the daemon, and ran some testing commands: <code> /etc/init.d/mhvtl start </code> <code> vtltape: version 0.16.13 vtltape: version 0.16.13 vtltape: version 0.16.13 vtltape: version 0.16.13 vtllibrary: version 0.16.13 vtllibrary process PID is 3638 </code> Checking that the config had created the SCSI devices expected: <code> lsscsi -g </code> <code> [2:3:0:0] mediumx SPECTRA PYTHON 550V /dev/sch0 /dev/sg4 [2:3:1:0] tape IBM ULT3580-TD4 550V /dev/st0 /dev/sg0 [2:3:2:0] tape IBM ULT3580-TD4 550V /dev/st1 /dev/sg1 [2:3:3:0] tape IBM ULT3580-TD4 550V /dev/st2 /dev/sg2 [2:3:4:0] tape IBM ULT3580-TD4 550V /dev/st3 /dev/sg3 </code> Verifying that I could use 'mtx' to do standard library operations: <code> mtx -f /dev/sg4 status </code> <code> Storage Changer /dev/sg4:4 Drives, 26 Slots ( 4 Import/Export ) Data Transfer Element 0:Empty Data Transfer Element 1:Empty Data Transfer Element 2:Empty Data Transfer Element 3:Empty Storage Element 1:Full :VolumeTag=TAPE01L4 Storage Element 2:Full :VolumeTag=TAPE02L4 Storage Element 3:Full :VolumeTag=TAPE03L4 Storage Element 4:Full :VolumeTag=TAPE04L4 Storage Element 5:Full :VolumeTag=TAPE05L4 Storage Element 6:Full :VolumeTag=TAPE06L4 Storage Element 7:Full :VolumeTag=TAPE07L4 Storage Element 8:Full :VolumeTag=TAPE08L4 Storage Element 9:Full :VolumeTag=TAPE09L4 Storage Element 10:Full :VolumeTag=TAPE10L4 Storage Element 11:Full :VolumeTag=TAPE11L4 Storage Element 12:Full :VolumeTag=TAPE12L4 Storage Element 13:Full :VolumeTag=TAPE13L4 Storage Element 14:Full :VolumeTag=TAPE14L4 Storage Element 15:Full :VolumeTag=TAPE15L4 Storage Element 16:Full :VolumeTag=TAPE16L4 Storage Element 17:Full :VolumeTag=TAPE17L4 Storage Element 18:Full :VolumeTag=TAPE18L4 Storage Element 19:Full :VolumeTag=TAPE19L4 Storage Element 20:Full :VolumeTag=TAPE20L4 Storage Element 21:Full :VolumeTag=CLN000L4 Storage Element 22:Full :VolumeTag=CLN001L4 Storage Element 23 IMPORT/EXPORT:Empty Storage Element 24 IMPORT/EXPORT:Empty Storage Element 25 IMPORT/EXPORT:Empty Storage Element 26 IMPORT/EXPORT:Empty </code> Checking the resulting files created by mhvtl: <code> ls -l /opt/vtl/ </code> <code> total 88 -rw-rw---- 1 vtl vtl 2048 2010-10-30 15:11 CLN000L4 -rw-rw---- 1 vtl vtl 2048 2010-10-30 15:11 CLN001L4 -rw-rw---- 1 vtl vtl 2048 2010-10-30 15:27 TAPE01L4 -rw-rw---- 1 vtl vtl 2048 2010-10-30 15:11 TAPE02L4 -rw-rw---- 1 vtl vtl 2048 2010-10-30 15:11 TAPE03L4 -rw-rw---- 1 vtl vtl 2048 2010-10-30 15:11 TAPE04L4 -rw-rw---- 1 vtl vtl 2048 2010-10-30 15:11 TAPE05L4 -rw-rw---- 1 vtl vtl 2048 2010-10-30 15:11 TAPE06L4 -rw-rw---- 1 vtl vtl 2048 2010-10-30 15:11 TAPE07L4 -rw-rw---- 1 vtl vtl 2048 2010-10-30 15:11 TAPE08L4 -rw-rw---- 1 vtl vtl 2048 2010-10-30 15:11 TAPE09L4 -rw-rw---- 1 vtl vtl 2048 2010-10-30 15:11 TAPE10L4 -rw-rw---- 1 vtl vtl 2048 2010-10-30 15:11 TAPE11L4 -rw-rw---- 1 vtl vtl 2048 2010-10-30 15:11 TAPE12L4 -rw-rw---- 1 vtl vtl 2048 2010-10-30 15:11 TAPE13L4 -rw-rw---- 1 vtl vtl 2048 2010-10-30 15:11 TAPE14L4 -rw-rw---- 1 vtl vtl 2048 2010-10-30 15:11 TAPE15L4 -rw-rw---- 1 vtl vtl 2048 2010-10-30 15:11 TAPE16L4 -rw-rw---- 1 vtl vtl 2048 2010-10-30 15:11 TAPE17L4 -rw-rw---- 1 vtl vtl 2048 2010-10-30 15:11 TAPE18L4 -rw-rw---- 1 vtl vtl 2048 2010-10-30 15:11 TAPE19L4 -rw-rw---- 1 vtl vtl 2048 2010-10-30 15:11 TAPE20L4 </code> ===== Finalizing ===== The startup-script gets added to /etc/init.d, but is not added to any runlevels during installation. So, so make it start on boot, you'll have to add it yourself, preferablu using update-rc.d <code> update-rc.d mhvtl defaults </code> ===== Note ===== In this walk-through, I did not touch the mhvtl.conf file. As a result, the "tapes" will be limited to the default "volume-size" of 500MB per tape-file, regardless of what tape-type and drive-type you describe in your devices.conf and library_contents.conf. If you want to simulate "realistic" volume-sizes, take a look at the mhvtl.conf file, and be aware that the tape-images are stored as continous files of the size you set up. Your filesystem may choke on huge files, your kernel may choke on huge files, and finally, mhvtl may even do the same.