+++ date = "2021-08-04T12:10:05+01:00" description = "" title = "LTO-4 Backup with Bacula" categories = [ "Backup", "Bacula", "Tape", "LTO-4" ] thumbnail = "/images/blog/bacula-lto4-backup/lto4.jpg" +++ ## Intro With Archlinux32 reaching some terabytes of data to backup I needed something "modern", like a tape. Now, big tapes like LTO-8 are close to unaffordable, LTO-4 drives and tapes on the other hand can be aqcuired on the cheap. They get thrown out of servers rooms at the moment. An LTO-4 tape can take 800 GB uncompressed data and the drive can be bought on Ebay for 200 to 300 CHF. Media is affordable at ca. 40 CHF per tape. # Tape Drive My first drive I ordered was advertised as working, which proved to be more the kind of drive only able to produce squealing noises and to be really hungry for tapes (and killing them). Well, my plan was to be under 1000 CHF for a backup solution, so I simply ordered a second one, keeping the first one for spare parts. Both drives are a HP Ultrium 1840. The second drive turned out to work just fine. But now, trying to connect, it showed some issues. The tape drive comes in a noisy black box, which I definitely don't want to run 24 hours a day. So I decided to remove the drive and squeeze it (quite literaly) into a machine. # Tape Media After ordering quite the wrong tape (LTO-4 WORM, which costs more and can be written only once, but has "smartness" built in to be tamperfree, oh well), I got boxes and boxes of old tapes from Ricardo from somebody desperately trying to get rid of them. Which is cool with me. The price per tape dropped to around 20 CHF this way and I have more tapes than I could ever have wished for. # Connectivity Issues I tried several SCSI cards to connect to the drive. The drive uses the last generation of parallel SCSI, which is quite a nuisance to find cards for. Either SCSI cards are server-grade (PCI-X) or they are not fast enough. Some cards (like dedicated backup SCA host adapters) work fine in some machines, but not in others. The SCSI cables are prone to transmition errors, especially a 4 meter long external SCSI cable with 320 MHz (external or internal 68-pin LVDS) is somehow not really reliable at this high speed. I went with a short shielded internal SCSI cable and putting the drive as close to the SCA host adapter as possible. This provided the best results. The result looks like this: {{< figure src="/images/blog/bacula-lto4-backup/fitting.jpg" alt="LTO-4 drive fitting the machine" >}} # Manual Backups and Tools ## tar Most people nowadays don't know anymore what the 't' in 'tar' stands for - you guessed it: '***t***ape ***ar***chive'. :-) There are other formats but usually the "rule-of-least-surprise" applies here, the simpler the command line parameters used and the more widespread the format, the more likely somebody else (or even you yourself) is able to actually read and restore the data. ## mt The old magnetic tape tool is no longer available as binary package on Archlinux, but there is an AUR package 'mt-st-git' providing the 'mt-st' binary. This tool you need to do basic operations on the tape like positioning, ejecting, setting compression levels, etc. ## Some use cases ### Rewind and eject ``` mt-st -f /dev/nst0 rewoffl ``` ### Erasing tapes ``` mt-st -f /dev/nst0 defcompression 0 mt-st -f /dev/nst0 compression 0 mt-st -f /dev/nst0 rewind tar -cvf /dev/nst0 /dev/null mt-st -f /dev/nst0 rewind ``` ***Note*** I'm disabling compression on the tapes for several reasons: * with compression on I'm not able to deliver enough data, resuling in shoe-shining * the remaining size of a tape is so much more predictable * I have enough tapes anyway. :-) ### Append to end of data ``` mt-st -f /dev/nst0 eom tar zcvf /dev/nst0 * ``` ### Status of the drive, current position of the tape ``` mt-st -f /dev/nst0 status ``` ## sg_logs This tool can give you all kind of internal information like temperature, I/O errors of the drive, media information. The first page of information serves as sort of an index, of what the drive can report: ``` shell> sg_logs /dev/nst0 -p 0 HP Ultrium 4-SCSI B32D Supported log pages [0x0]: 0x00 Supported log pages [sp] 0x02 Write error [we] 0x03 Read error [re] 0x0c Sequential access device [sad] 0x0d Temperature [temp] 0x11 DT Device status [dtds] 0x12 Tape alert response [tar] 0x13 Requested recovery [rr] 0x18 Protocol specific port [psp] 0x2e Tape alert [ta] 0x30 Tape usage (lto-5, 6) [tu_] 0x31 Tape capacity (lto-5, 6) [tc_] 0x32 Data compression (lto-5) [dc_] 0x33 Write errors (lto-5) [we_] 0x34 Read forward errors (lto-5) [rfe_] 0x35 DT Device Error (lto-5, 6) [dtde_] 0x3e Device Status (lto-5, 6) [ds_] ``` For instance I can get the temperature of the drive with: ``` shell> sg_logs /dev/nst0 -p 13 HP Ultrium 4-SCSI B32D Temperature page [0xd] Current temperature = 47 C Reference temperature = ``` This could be meshed into a nagios check script, checking the sanity of the drive, but then I have to manually unmount the tape pool in bacula-sd before each check. ## socat socat is like netcat and more. It allows to build tunnels between machines, so that the 'tar' command can pack files on one machine and send them to another machine, where the tape write command is attached to a listening socat. ``` # on the machine with the files to backup tar cvf - * | socat - TCP4::8080 # on the machine where the tape is socat TCP4-LISTEN:8080 - | dd of=/dev/nst0 bs=10240 status=progress ``` ***Note*** If using dd I set the blocksize manually to 20*512=10240, this seems to be the standard blocksize of 'tar' on Linux. ## mbuffer Writting directly to the tape has some drawbacks as the tape drive is very fast and you cannot deliver data fast enough over a 1GBit/s network. So here 'mbuffer' helps to at least buffer data for some time and then flash it in one burst to the tape drive. This avoids the dreadful "shoe-shining" which not only drives you crazy (the sounds of it), but also reduces the lifetime of the components (or at least of the mechanics of the tape drive): ``` tar cvf - * | mbuffer -m 2G -P100% | \ socat - TCP4::8080 socat TCP4-LISTEN:8080 - | mbuffer -m 2G -P100% | \ dd of=/dev/nst0 bs=10240 status=progress ``` Buffering on either side is possible, not sure if having a buffer on both sides improves anything. # Use Cases ## Full Backup I did a full backup of everything onto 10 tapes with the 'tar/socat/mbuffer/dd' method. This is data which is quite stable and never changes, so I'll just keep it on some tapes with the write protection label on. It doesn't make much sense to put them into a bacula job, as the retention period is basically 30 years or so - or till the tape dies. The index of the tape is a simple text file, noting the kind of data, the size, the tape number, the file number (offset on tape) and the date of the backup: ``` doc 946M 1 0 17.4.2021 Attic 13G 1 1 17.4.2021 bilder 19G 1 2 17.4.2021 projects 29G 1 3 18.4.2021 ARCHIVE 16G 1 5 18.4.2021 BACKUPS 122G 1 6 18.4.2021 ... music 154G 4 0 19.4.2021 movies part1 547G 5 0 19.4.2021 movies part2 785G 6 0 20.4.2021 ``` ## Bacula I use bacula for the daily incremental and full backups now for tape and offline cloud storage. bacula-sd just works fine and integrates with the rest of my backup system (the master bacula-dir is still living on an old Raspberry Pi). The only thing I was missing is to be able to copy a bacula job to two different media, one being the remote cloud storage and the other one the tape. Sort of a bacula 'tee' would be nice to have. # References * http://cdrtools.sourceforge.net/private/portability-of-tar-features.html: on tar formats and compatibility * https://copyconstruct.medium.com/socat-29453e9fc8a6: blog about socat * https://www.commandlinefu.com/commands/view/13582/backup-to-lto-tape-with-progress-checksums-and-buffering * https://aur.archlinux.org/packages/mt-st-git/