====== Virtual Tape Library using linuxvtl2/mhvtl on Debian ======

===== About VTL =====

This segment is stolen from the [[http://en.wikipedia.org/wiki/Virtual_tape_library|Wikipedia article on VTL]]:

> A virtual tape library (VTL) is a data storage virtualization technology used typically for backup and recovery purposes. A VTL presents a storage component (usually hard disk storage) as tape libraries or tape drives for use with existing backup software.

> Virtualizing the disk storage as tape allows integration of VTLs with existing backup software and existing backup and recovery processes and policies. The benefits of such virtualization include storage consolidation and faster data restore processes.

For linux, we have an experimental implementation of file-based VTL in the form of [[http://sites.google.com/site/linuxvtl2/|linuxvtl2]], thanks to the work of Mark Harvey. It is based on the work done with the [[http://sg.danny.cz/sg/sdebug26.html|scsi_debug]] adapter driver, and consists of a combination of a kernel character device driver and userspace tools.

I will in general reference this implementation by its "official" name, linuxvtl2, and more often by its in-use name, mhvtl (obviously named after Mark Harvey).

===== About this install walk-through =====

There is no package for linuxvtl2 in the official package trees, and
linuxvtl2 is to be considered "experimental" as data-formats may change
between versions. This means that it will have to be installed from source.

In this how-I-did blurb, I was working as 'root' (shrug), and as such,
I have not used 'sudo' to perform commands that need root-like permissions.
If you are doing this properly, add 'sudo' to your commands as needed.

===== Pre-requisites for compiling and "tape" administration =====

After installing a base Debian system, pull in essential packages
for compiling C/C++ and kernel related code. The shortest route:

<code>
apt-get -y install build-essential
apt-get -y install linux-headers-`uname -r`
# Because of brainfart in either mhvtl or the XenServer package:
ln -s /usr/src/linux-headers-`uname -r` /lib/modules/`uname -r`/build
apt-get -y install zlib1g-dev
</code> 

Prepare the environment for tape-library use, by installing the two most
essential packages for handling SCSI and autochangers.
<code>
apt-get -y install lsscsi mtx mt-st scsitools sg3-utils --no-install-recommends
</code> 

===== Downloading and compiling =====

Pull down the source for linuxvtl2/mhvtl, and unpack it.
<code> 
wget http://sites.google.com/site/linuxvtl2/mhvtl-2010-05-09.tgz?attredirects=0 -O mhvtl-2010-05-09.tgz
tar zxvf mhvtl-2010-05-09.tgz
</code> 

MHVTL really-really wants to run as the user 'vtl', having membership in the group 'vtl', and having the users home directory as '/opt/vtl'. So we should satisfy this before compiling the code. The username, group and location can be changed
bu going through the Makefiles and changing it. Or, you can just live with the defaults, and mount appropriate storage space as either /opt or /opt/vtl
<code>
groupadd vtl
useradd --system -c "Vitrual Tape Library" -d /opt/vtl -g vtl -m vtl
</code>

Now, simply compile and install.
<code>
cd mhvtl-0.16
make
make install
cd kernel
make
make install
</code>

===== Configuration =====

Configuration files are not __installed__ by default. Rather, they get __generated__ when mhvtl is first started. So, before progressing, start and then stop the "service".

After this, I started up the daemon, and ran some testing commands:
<code>
/etc/init.d/mhvtl start
sleep 10
/etc/init.d/mhvtl shutdown
rm /opt/vtl/*
</code> 

The default configuration file contained a lot more devices and stuff than I wanted, so I stripped down the config a bit. First, this is my resulting
/etc/mhvtl/device.conf file:
<code>
VERSION: 4

# VPD page format:
# <page #> <Length> <x> <x+1>... <x+n>
# NAA format is an 8 hex byte value seperated by ':'
# Note: NAA is part of inquiry VPD 0x83
#
# Each 'record' is separated by one (or more) blank lines.
# Each 'record' starts at column 1
# Serial num max len is 10.
# Compression: factor X enabled 0|1
#     Where X is zlib compression factor        1 = Fastest compression
#                                               9 = Best compression
#     enabled 0 == off, 1 == on

Library: 10 CHANNEL: 03 TARGET: 00 LUN: 00
 Vendor identification: SPECTRA
 Product identification: PYTHON
 Product revision level: 550V
 Unit serial number: XYZZY_A
 NAA: 10:22:33:44:ab:00:00:00

Drive: 11 CHANNEL: 03 TARGET: 01 LUN: 00
 Library ID: 10 Slot: 01
 Vendor identification: IBM
 Product identification: ULT3580-TD4
 Product revision level: 550V
 Unit serial number: XYZZY_A1
 NAA: 10:22:33:44:ab:00:01:00
 Compression: factor 1 enabled 1

Drive: 12 CHANNEL: 03 TARGET: 02 LUN: 00
 Library ID: 10 Slot: 02
 Vendor identification: IBM
 Product identification: ULT3580-TD4
 Product revision level: 550V
 Unit serial number: XYZZY_A2
 NAA: 10:22:33:44:ab:00:02:00
 Compression: factor 1 enabled 1

Drive: 13 CHANNEL: 03 TARGET: 03 LUN: 00
 Library ID: 10 Slot: 03
 Vendor identification: IBM
 Product identification: ULT3580-TD4
 Product revision level: 550V
 Unit serial number: XYZZY_A3
 NAA: 10:22:33:44:ab:00:03:00
 Compression: factor 1 enabled 1

Drive: 14 CHANNEL: 03 TARGET: 04 LUN: 00
 Library ID: 10 Slot: 04
 Vendor identification: IBM
 Product identification: ULT3580-TD4
 Product revision level: 550V
 Unit serial number: XYZZY_A4
 NAA: 10:22:33:44:ab:00:04:00
 Compression: factor 1 enabled 1
</code>

I then changed the "contents" of the defined library (/etc/mhvtl/library_contents.10):
<code>

Drive 1:
Drive 2:
Drive 3:
Drive 4:

Picker 1:

MAP 1:
MAP 2:
MAP 3:
MAP 4:

# Slot 1 - ?, no gaps
# Slot N: [barcode]
# [barcode]
# a barcode is comprised of three fields: [Leading] [identifier] [Trailing]
# Leading "CLN" -- cleaning tape
# Leading "W" -- WORM tape
# Leading "NOBAR" -- will appear to have no barcode
# If the barcode is at least 8 character long, then the last two characters are Trailing
# Trailing "S3" - SDLT600
# Trailing "X4" - AIT-4
# Trailing "L1" - LTO 1
# Trailing "TA" - T10000+
# Trailing "JA" - 3592+
# Trailing "JB" - 3592E05+
# Trailing "JW" - WORM 3592+
# Trailing "JX" - WORM 3592E05+
#
Slot 1: TAPE01L4
Slot 2: TAPE02L4
Slot 3: TAPE03L4
Slot 4: TAPE04L4
Slot 5: TAPE05L4
Slot 6: TAPE06L4
Slot 7: TAPE07L4
Slot 8: TAPE08L4
Slot 9: TAPE09L4
Slot 10: TAPE10L4
Slot 11: TAPE11L4
Slot 12: TAPE12L4
Slot 13: TAPE13L4
Slot 14: TAPE14L4
Slot 15: TAPE15L4
Slot 16: TAPE16L4
Slot 17: TAPE17L4
Slot 18: TAPE18L4
Slot 19: TAPE19L4
Slot 20: TAPE20L4
Slot 21: CLN000L4
Slot 22: CLN001L4
</code>

===== Testing the setup =====

After this, I started up the daemon, and ran some testing commands:
<code>
/etc/init.d/mhvtl start
</code>
<code>
vtltape: version 0.16.13
vtltape: version 0.16.13
vtltape: version 0.16.13
vtltape: version 0.16.13
vtllibrary: version 0.16.13
vtllibrary process PID is 3638
</code> 

Checking that the config had created the SCSI devices expected:
<code>
lsscsi -g
</code>
<code>
[2:3:0:0]    mediumx SPECTRA  PYTHON           550V  /dev/sch0  /dev/sg4
[2:3:1:0]    tape    IBM      ULT3580-TD4      550V  /dev/st0  /dev/sg0
[2:3:2:0]    tape    IBM      ULT3580-TD4      550V  /dev/st1  /dev/sg1
[2:3:3:0]    tape    IBM      ULT3580-TD4      550V  /dev/st2  /dev/sg2
[2:3:4:0]    tape    IBM      ULT3580-TD4      550V  /dev/st3  /dev/sg3
</code>

Verifying that I could use 'mtx' to do standard library operations:
<code>
mtx -f /dev/sg4 status
</code>
<code>
  Storage Changer /dev/sg4:4 Drives, 26 Slots ( 4 Import/Export )
Data Transfer Element 0:Empty
Data Transfer Element 1:Empty
Data Transfer Element 2:Empty
Data Transfer Element 3:Empty
      Storage Element 1:Full :VolumeTag=TAPE01L4
      Storage Element 2:Full :VolumeTag=TAPE02L4
      Storage Element 3:Full :VolumeTag=TAPE03L4
      Storage Element 4:Full :VolumeTag=TAPE04L4
      Storage Element 5:Full :VolumeTag=TAPE05L4
      Storage Element 6:Full :VolumeTag=TAPE06L4
      Storage Element 7:Full :VolumeTag=TAPE07L4
      Storage Element 8:Full :VolumeTag=TAPE08L4
      Storage Element 9:Full :VolumeTag=TAPE09L4
      Storage Element 10:Full :VolumeTag=TAPE10L4
      Storage Element 11:Full :VolumeTag=TAPE11L4
      Storage Element 12:Full :VolumeTag=TAPE12L4
      Storage Element 13:Full :VolumeTag=TAPE13L4
      Storage Element 14:Full :VolumeTag=TAPE14L4
      Storage Element 15:Full :VolumeTag=TAPE15L4
      Storage Element 16:Full :VolumeTag=TAPE16L4
      Storage Element 17:Full :VolumeTag=TAPE17L4
      Storage Element 18:Full :VolumeTag=TAPE18L4
      Storage Element 19:Full :VolumeTag=TAPE19L4
      Storage Element 20:Full :VolumeTag=TAPE20L4
      Storage Element 21:Full :VolumeTag=CLN000L4
      Storage Element 22:Full :VolumeTag=CLN001L4
      Storage Element 23 IMPORT/EXPORT:Empty
      Storage Element 24 IMPORT/EXPORT:Empty
      Storage Element 25 IMPORT/EXPORT:Empty
      Storage Element 26 IMPORT/EXPORT:Empty
</code>

Checking the resulting files created by mhvtl:
<code>
ls -l /opt/vtl/
</code>
<code>
total 88
-rw-rw---- 1 vtl vtl 2048 2010-10-30 15:11 CLN000L4
-rw-rw---- 1 vtl vtl 2048 2010-10-30 15:11 CLN001L4
-rw-rw---- 1 vtl vtl 2048 2010-10-30 15:27 TAPE01L4
-rw-rw---- 1 vtl vtl 2048 2010-10-30 15:11 TAPE02L4
-rw-rw---- 1 vtl vtl 2048 2010-10-30 15:11 TAPE03L4
-rw-rw---- 1 vtl vtl 2048 2010-10-30 15:11 TAPE04L4
-rw-rw---- 1 vtl vtl 2048 2010-10-30 15:11 TAPE05L4
-rw-rw---- 1 vtl vtl 2048 2010-10-30 15:11 TAPE06L4
-rw-rw---- 1 vtl vtl 2048 2010-10-30 15:11 TAPE07L4
-rw-rw---- 1 vtl vtl 2048 2010-10-30 15:11 TAPE08L4
-rw-rw---- 1 vtl vtl 2048 2010-10-30 15:11 TAPE09L4
-rw-rw---- 1 vtl vtl 2048 2010-10-30 15:11 TAPE10L4
-rw-rw---- 1 vtl vtl 2048 2010-10-30 15:11 TAPE11L4
-rw-rw---- 1 vtl vtl 2048 2010-10-30 15:11 TAPE12L4
-rw-rw---- 1 vtl vtl 2048 2010-10-30 15:11 TAPE13L4
-rw-rw---- 1 vtl vtl 2048 2010-10-30 15:11 TAPE14L4
-rw-rw---- 1 vtl vtl 2048 2010-10-30 15:11 TAPE15L4
-rw-rw---- 1 vtl vtl 2048 2010-10-30 15:11 TAPE16L4
-rw-rw---- 1 vtl vtl 2048 2010-10-30 15:11 TAPE17L4
-rw-rw---- 1 vtl vtl 2048 2010-10-30 15:11 TAPE18L4
-rw-rw---- 1 vtl vtl 2048 2010-10-30 15:11 TAPE19L4
-rw-rw---- 1 vtl vtl 2048 2010-10-30 15:11 TAPE20L4
</code> 

===== Finalizing =====

The startup-script gets added to /etc/init.d, but is not added
to any runlevels during installation. So, so make it start on boot,
you'll have to add it yourself, preferablu using update-rc.d

<code>
update-rc.d mhvtl defaults
</code> 

===== Note =====

In this walk-through, I did not touch the mhvtl.conf file. As a result,
the "tapes" will be limited to the default "volume-size" of 500MB per tape-file,
regardless of what tape-type and drive-type you describe in your devices.conf and library_contents.conf. If you want to simulate "realistic" volume-sizes, take a look at the mhvtl.conf file, and be aware that the tape-images are stored as continous files of the size you set up. Your filesystem may choke on huge files, your kernel may choke on huge files, and finally, mhvtl may even do the same.