Warning Livedoc is no longer being updated and will be deprecated shortly. Please refer to https://documentation.tjhsst.edu.

Difference between revisions of "OpenAFS/backup system"

From Livedoc - The Documentation Repository
Jump to: navigation, search
m (Robot: Changing Category:Obsolete)
(updated page for the new afs backup system)
Line 1: Line 1:
 
==Overview==
 
==Overview==
The '''OpenAFS backup system''' consists of three main programs: '''buserver''', '''backup''', and '''butc'''. Currently, the system is being used to backup all web, user, and service volumes to tapes on the [[Exabyte EXB-480 Tape Drive]].
+
The '''OpenAFS backup system''' consists of three main programs: '''buserver''', '''backup''', and '''butc'''. Currently, the system is being used to backup all web, active user, and staff volumes to files on Openafs3, a zone on '''Agni'''.
  
 
==buserver==
 
==buserver==
The buserver is a daemon which keeps track of the "backup database," which contains all information about what dumps have been performed, the dumping schedule, and what volumes go on what dumps. It also knows what dumps are on what tapes, and keeps track of what port offsets are on which server. This is just added in BosConfig; no configuration necessary just for the server.
+
The buserver is a daemon which keeps track of the "backup database," which contains all information about what dumps have been performed, the dumping schedule, and what volumes go on what dumps. It also knows what dumps are on what tapes, and keeps track of what port offsets are on which server. The buserver should be run on all of the DB Servers (currently haafs1, haafs2, and openafs1) by running:
 +
  bos create -server <servername> -instance buserver -type simple -cmd </path/to/buserver>
 +
For example:
 +
bos create -server openafs1 -instance buserver -type simple -cmd /usr/lib/openafs/buserver
  
 
==butc==
 
==butc==
To write a backup dump to a tape, there must be a butc process running for that specific tape device. The butc process coordinates obtaining the correct tape for writing, and getting volume dumps, etc. as well as writing to the tape. They also can display information about what's going on, so it's generally a good idea to run butc where you can observer output. Right now, butc processes for [[emperor]] are run in a screen session, so we can check up on them at any time. The butc process will also prompt an operator to insert a certain tape manually if automatically getting the correct tape fails for some reason.
+
To write a backup dump to a tape, there must be a butc process running for that specific tape device. The butc process coordinates obtaining the correct tape for writing, and getting volume dumps, etc. as well as writing to the tape. It will also print information about what is currently happening and prompt for user input if it gets confused. Currently, the butc processes are run in a screen on openafs3.
  
 
==backup==
 
==backup==
The '''backup''' command suite has several commands to run.
+
The '''backup''' command suite has several commands to run.  It has a nice built-in help system (like most AFS commands) so that information will not be duplicated here.  It is generally advised to run many backup commands through interactive mode (run backup by itself and then run the commands).  This allows multiple commands to be queued up and then monitored.  Currently, an interactive backup session with local authentication tokens is kept running in the same screen as the butc processes for primary backup scheduling.  NOTE: closing this session will cancel any jobs that are currently scheduled through it (you can see these with the jobs command) so please make note of anything that is scheduled before closing the session and reschedule them ASAP.
  
 
===Dump scheduling===
 
===Dump scheduling===
  adddump        add dump schedule
+
A dump schedule isn't really a "schedule" per se. A dump "schedule" basically just consists of a dump level, and long until it expires. For example, one of our dump schedules is:
  deldump        delete dump schedule
+
   /sunday1 expires in  27d
  listdumps      list dump schedules
+
    /monday1 expires in  13d
  setexp          set/clear dump expiration dates
+
    /tuesday1 expires in  13d
A dump schedule isn't really a "schedule" per se. A dump "schedule" basically just consists of a dump level, and long until it expires. For example, some of our dump schedules include:
+
    /wednesday1 expires in  13d
   /sunday0 expires in  27d
+
    /thursday1 expires in  13d
      /Mon expires in  6d
+
    /friday1 expires in  13d
      /Tue expires in  6d
+
    /saturday1 expires in  13d
      /Wed expires in  6d
+
 
      /Thu expires in  6d
+
Which are named pretty intuitively. The '/sunday1' schedule is a full dump, and usually occurs every four weeks (as you can see, it expires in 27 days, making it so the backup system prevents anything from overwriting a /sunday0 backup dump until 27 days have passed). The other levels (which are actually /sunday1/Monday1 and so on; backup just doesn't repeat all of that) are incremental dumps, each one only recording the information changed since the last /sunday1 dump. These dump schedules only expire in 13 days, so it's possible to overwrite them in two weeks (which is what we do).  Currently, we keep weekly dumps for one month (Sunday1-Sunday4) and daily dumps for two weeks (Monday1-Saturday1 for odd Sundays and Monday2-Saturday2 for even Sundays).  This gives us the ability to restore a volume to any day in the past two weeks and to any Sunday in the past month.
      /Fri expires in  6d
 
      /Sat expires in  6d
 
Which are named pretty intuitively. The '/sunday0' schedule is a full dump, and usually occurs every four weeks (as you can see, it expires in 27 days, making it so the backup system prevents anything from overwriting a /sunday0 backup dump until 27 days have passed). The other levels (which are actually /sunday0/Mon and so on; backup just doesn't repeat all of that) are incremental dumps, ecah one only recording the information changed since the last /sunday0 dump. These dump schedules only expire in 6 days, so it's possible to overwrite them in just a week (which is what we do).
 
  
 
===Hosts and port offsets===
 
===Hosts and port offsets===
  addhost        add host to config
+
When performing a backup dump, a "port offset" must be specified. A [[port offset]] determines both the host to contact, and what tape device that host uses. (See '''tapeconfig''' furthur down on this page.)
  delhost        delete host to config
 
  listhosts      list config hosts
 
When performing a backup dump, a "port offset" must be specified. A [[port offset]] determines both the host to contact, and what tape device that host uses. (See '''tapeconfig''' furthur down on this page.) Use these commands to specify which host corresponds to a certain port offset.
 
  
 
===Volume sets===
 
===Volume sets===
  addvolentry    add a new volume entry
+
A volume set is what it sounds like: just a set of volumes. A volume set is defined by a series of volume entries. A volume entry is an entry specifying what volume to backup, and on what server and partition. The volume name, partition, and server can all be simple regular expressions (the only special characters are brackets, dots, backslashes, and asterisks), so you can specify large amounts of volumes with a single volume entry. For example, the following volume set encompasses all current students for the 2009-2010 school year:
  addvolset      create a new volume set
+
   Volume set currentstudents:
  delvolentry    delete a volume set sub-entry
+
    Entry   1: server .*, partition .*, volumes: 2010\..*\.backup
  delvolset      delete a volume set
+
    Entry   2: server .*, partition .*, volumes: 2011\..*\.backup
  listvolsets    list volume sets
+
    Entry   3: server .*, partition .*, volumes: 2012\..*\.backup
A volume set is what it sounds like: just a set of volumes. A volume set is defined by a series of volume entries. A volume entry is an entry specifying what volume to backup, and on what server and partition. The volume name, partition, and server can all be simple regular expressions (the only special characters are brackets, dots, backslashes, and asterisks), so you can specify large amounts of volumes with a single volume entry.
+
    Entry  4: server .*, partition .*, volumes: 2013\..*\.backup
 
 
===Dumping and restoring===
 
   deletedump      delete dumps from the database
 
   diskrestore    restore partition
 
  dump            start dump
 
  restoredb      restore backup database
 
   savedb          save backup database
 
   scantape        dump information recovery from tape
 
  volrestore      restore volume
 
  volsetrestore  restore a set of volumes
 
The "dump" command starts a backup dump for the specified volume set and dump schedule and port offset. If you use the "-append" switch, instead of overwriting the data on the tape (which it would normally do if the data was expired), it just appends to it.
 
 
 
The *restore commands just restore volumes from tape from a few different methods (either specifying the specific volume, or restoring a whole disk, or restoring a whole volume set).
 
  
The *db commands deal with backing up the backup database itself. If somehow all of the backup database is lost (which seems rather unlikely, since it's retained across all of the dbservers via ubik), you can restore it from tape with those commands. You can also get the database information for a single dump by using 'scantape' on a tape with the dump on it. (So you can then restore it afterwards.)
+
Note that the . is a wildcard character so we need to escape it (\.) to indicate a literal .. The wildcard .* matches any server or partition.
  
"deletedump" just deletes the information from the backup database about a certain dump. The actual data on the tape remains intact, so you can still restore it later with "scantape".
+
In general, we do not put the actual volumes into the volume sets.  Instead we add the .backup volumes that are created nightly as snapshots of the volume. The reason for this is that when AFS actually dumps the volume to tape, it puts a lock on the volume for the duration of the dump and so by dumping the backup volume, we only interrupt someone's yesterday folder instead of their homedir.
 
 
===Miscellaneous===
 
  apropos        search by help text
 
  dbverify        check ubik database integrity
 
  dumpinfo        provide information about a dump in the database
 
  help            get help on commands
 
  interactive    enter interactive mode
 
  jobs            list running jobs
 
  kill            kill running job
 
  labeltape      label a tape
 
  quit            leave the program
 
  readlabel      read the label on tape
 
  status          get tape coordinator status
 
  volinfo        query the backup database
 
Most of these either have obvious functions or aren't used very often. The "jobs" refer to something like a backup dump running. "readlabel" can possibly tell you what data is on a tape, if any, and "labeltape" can be used to assign your own label to a tape. Reasons for why you'd want to do that are either nonexistant or just for advanced users; check the OpenAFS documentation.
 
  
 
==Configuration==
 
==Configuration==
Under Debian, configuration for the backup system occurs in <tt>/var/lib/openafs/backup/</tt> (as well as logging). Here are the various files that live there:
+
On openafs3, the backup system configuration and logging takes place in /usr/local/var/openafs/backup. The files that live there are:
  
 
===tapeconfig===
 
===tapeconfig===
This is a very simple file, which maps tape devices to [[port offset]]s, optionally with tape sizes and filemark sizes. As of the time of this writing, it contains the following (on emperor):
+
This is a very simple file, which maps tape devices to [[port offset]]s, optionally with tape sizes and filemark sizes. As of the time of this writing, it contains the following (on openafs3):
  
   /dev/st0       0
+
   500G    0      /tapes/faketapea       0
   /dev/st1       1
+
   500G    0      /tapes/faketapeb       1
   /dev/st2       2
+
   500G    0      /tapes/faketapec       2
   /dev/st3       3
+
   500G    0      /tapes/faketaped       3
  
Since the [[Exabyte]] has four tape drives in it, this just maps them to port offsets 0-4. As of the time of this writing, only port offsets 1 and 2 are used for backups; the others are left for fiddling around with tapes, or possible for use with [[AMANDA]] in the future. (Some AMANDA tape changers only use the first drive, which is why we're not using it for the OpenAFS backups.)
+
This specifies four files to backup to (actually symlinks) which are mapped to port offsets 0-3. The reason for creating four is so that each of our primary volsets can be mapped to a different port offset so that a problem with one backup will not affect the others.  Note that normally, with modern tape drives, it is recommended to omit the tape size and filemark size but with files, the former is highly recommended to avoid running out of space on the partition so both should be specified (if you give one you have to give both).
 
 
Optionally, there can be a third and fourth column with are the tape size and filemark size, but it works fine (and is even recommended in the manual) to leave them blank to their defaults. The only time it would be advisable to set the size is when the tapes are very small (<2GB), so butc doesn't spend all of it's time writing a volume, only to find that it wasn't large enough to fit on the tape. Our tapes are 20 GB (more with compression, obviously), so we don't need to worry about this.
 
  
 
===CFG_device_name===
 
===CFG_device_name===
This is the configuration file used for the tape device device_name. If the tape device was /dev/foo/bar, then the configuration filename would be CFG_foo_bar. As of now, we're only using /dev/st0 through /dev/st3, so we have the config files CFG_st0 through CFG_st3. Since they are all [[Exabyte]] drives and are all used the same way, they're all symlinked to the same configuration file: CFG.exabyte.
+
This is the configuration file used for the tape device device_name. If the tape device was /dev/foo/bar, then the configuration filename would be CFG_foo_bar and for a backup file in /tapes/faketapea the configuration filename would be CFG_tapes_faketapea. As of now, we're using /tapes/faketapea through /tapes/faketaped so we have the config files CFG_tapes_faketapea through CFG_tapes_faketaped. Since they are all basically the same and are all used the same way, they're all symlinked to the same configuration file: CFG_faketapes
  
===CFG.exabyte===
+
===CFG_faketapes===
The configuration file for exabyte drives. Right now it just contains the following:
+
The configuration file for the backup files. Right now it just contains the following:
  
   AUTOQUERY YES
+
   FILE YES
   MOUNT /usr/local/lib/openafs/exabyte-mount
+
   MOUNT /root/scripts/afsmountscript.sh
  UNMOUNT /usr/local/lib/openafs/exabyte-unmount
 
  
The first line makes sure that butc asks for a tape the first time it tries to load one in a dump set. If AUTOQUERY is set to NO, then it assumes the correct tape is already in there (which is useful for manual loading of tapes, but we don't do that, since we have the [[Exabyte]]).
+
The first line tells butc that it will be backing up to a file instead of to a tapedrive. The second line specifies a mount script that will be called to load the appropriate tape into the drive. In this case it links to a script from the OpenAFS docs that symlinks a filename of the form <volset>.<dumplevel>.<tapenumber> to the given device name.
 
 
The second and third lines specify the location of the mounting and unmounting scripts to use for load/unloading tapes from the tape changer. Documentation for these scripts can be found at [[exabyte-mount]]; they just call '''mtx''' and '''mt''' to load/unload the requested tape.
 
  
 
===TE_device_name===
 
===TE_device_name===
Line 106: Line 70:
 
===TL_device_name===
 
===TL_device_name===
 
Using the same device_name convention as CFG, these files are where all butc operations are logged.
 
Using the same device_name convention as CFG, these files are where all butc operations are logged.
 +
 +
==External Links==
 +
* Openafs - configuring the AFS backup system: http://docs.openafs.org/AdminGuide/ch06.html
 +
* Openafs - dumping data to a backup file: http://docs.openafs.org/AdminGuide/ch06s10.html#HDRWQ282
  
 
[[Category:Obsolete Page]]
 
[[Category:Obsolete Page]]

Revision as of 03:01, 4 May 2010

Overview

The OpenAFS backup system consists of three main programs: buserver, backup, and butc. Currently, the system is being used to backup all web, active user, and staff volumes to files on Openafs3, a zone on Agni.

buserver

The buserver is a daemon which keeps track of the "backup database," which contains all information about what dumps have been performed, the dumping schedule, and what volumes go on what dumps. It also knows what dumps are on what tapes, and keeps track of what port offsets are on which server. The buserver should be run on all of the DB Servers (currently haafs1, haafs2, and openafs1) by running:

 bos create -server <servername> -instance buserver -type simple -cmd </path/to/buserver>

For example:

bos create -server openafs1 -instance buserver -type simple -cmd /usr/lib/openafs/buserver

butc

To write a backup dump to a tape, there must be a butc process running for that specific tape device. The butc process coordinates obtaining the correct tape for writing, and getting volume dumps, etc. as well as writing to the tape. It will also print information about what is currently happening and prompt for user input if it gets confused. Currently, the butc processes are run in a screen on openafs3.

backup

The backup command suite has several commands to run. It has a nice built-in help system (like most AFS commands) so that information will not be duplicated here. It is generally advised to run many backup commands through interactive mode (run backup by itself and then run the commands). This allows multiple commands to be queued up and then monitored. Currently, an interactive backup session with local authentication tokens is kept running in the same screen as the butc processes for primary backup scheduling. NOTE: closing this session will cancel any jobs that are currently scheduled through it (you can see these with the jobs command) so please make note of anything that is scheduled before closing the session and reschedule them ASAP.

Dump scheduling

A dump schedule isn't really a "schedule" per se. A dump "schedule" basically just consists of a dump level, and long until it expires. For example, one of our dump schedules is:

 /sunday1  expires in  27d
   /monday1  expires in  13d
   /tuesday1  expires in  13d
   /wednesday1  expires in  13d
   /thursday1  expires in  13d
   /friday1  expires in  13d
   /saturday1  expires in  13d

Which are named pretty intuitively. The '/sunday1' schedule is a full dump, and usually occurs every four weeks (as you can see, it expires in 27 days, making it so the backup system prevents anything from overwriting a /sunday0 backup dump until 27 days have passed). The other levels (which are actually /sunday1/Monday1 and so on; backup just doesn't repeat all of that) are incremental dumps, each one only recording the information changed since the last /sunday1 dump. These dump schedules only expire in 13 days, so it's possible to overwrite them in two weeks (which is what we do). Currently, we keep weekly dumps for one month (Sunday1-Sunday4) and daily dumps for two weeks (Monday1-Saturday1 for odd Sundays and Monday2-Saturday2 for even Sundays). This gives us the ability to restore a volume to any day in the past two weeks and to any Sunday in the past month.

Hosts and port offsets

When performing a backup dump, a "port offset" must be specified. A port offset determines both the host to contact, and what tape device that host uses. (See tapeconfig furthur down on this page.)

Volume sets

A volume set is what it sounds like: just a set of volumes. A volume set is defined by a series of volume entries. A volume entry is an entry specifying what volume to backup, and on what server and partition. The volume name, partition, and server can all be simple regular expressions (the only special characters are brackets, dots, backslashes, and asterisks), so you can specify large amounts of volumes with a single volume entry. For example, the following volume set encompasses all current students for the 2009-2010 school year:

 Volume set currentstudents:
   Entry   1: server .*, partition .*, volumes: 2010\..*\.backup
   Entry   2: server .*, partition .*, volumes: 2011\..*\.backup
   Entry   3: server .*, partition .*, volumes: 2012\..*\.backup
   Entry   4: server .*, partition .*, volumes: 2013\..*\.backup

Note that the . is a wildcard character so we need to escape it (\.) to indicate a literal .. The wildcard .* matches any server or partition.

In general, we do not put the actual volumes into the volume sets. Instead we add the .backup volumes that are created nightly as snapshots of the volume. The reason for this is that when AFS actually dumps the volume to tape, it puts a lock on the volume for the duration of the dump and so by dumping the backup volume, we only interrupt someone's yesterday folder instead of their homedir.

Configuration

On openafs3, the backup system configuration and logging takes place in /usr/local/var/openafs/backup. The files that live there are:

tapeconfig

This is a very simple file, which maps tape devices to port offsets, optionally with tape sizes and filemark sizes. As of the time of this writing, it contains the following (on openafs3):

 500G    0       /tapes/faketapea        0
 500G    0       /tapes/faketapeb        1
 500G    0       /tapes/faketapec        2
 500G    0       /tapes/faketaped        3

This specifies four files to backup to (actually symlinks) which are mapped to port offsets 0-3. The reason for creating four is so that each of our primary volsets can be mapped to a different port offset so that a problem with one backup will not affect the others. Note that normally, with modern tape drives, it is recommended to omit the tape size and filemark size but with files, the former is highly recommended to avoid running out of space on the partition so both should be specified (if you give one you have to give both).

CFG_device_name

This is the configuration file used for the tape device device_name. If the tape device was /dev/foo/bar, then the configuration filename would be CFG_foo_bar and for a backup file in /tapes/faketapea the configuration filename would be CFG_tapes_faketapea. As of now, we're using /tapes/faketapea through /tapes/faketaped so we have the config files CFG_tapes_faketapea through CFG_tapes_faketaped. Since they are all basically the same and are all used the same way, they're all symlinked to the same configuration file: CFG_faketapes

CFG_faketapes

The configuration file for the backup files. Right now it just contains the following:

 FILE YES
 MOUNT /root/scripts/afsmountscript.sh

The first line tells butc that it will be backing up to a file instead of to a tapedrive. The second line specifies a mount script that will be called to load the appropriate tape into the drive. In this case it links to a script from the OpenAFS docs that symlinks a filename of the form <volset>.<dumplevel>.<tapenumber> to the given device name.

TE_device_name

Using the same device_name convention as CFG, these files are where errors are logged by butc.

TL_device_name

Using the same device_name convention as CFG, these files are where all butc operations are logged.

External Links