Todo - Summer 2014

From Livedoc - The Documentation Repository
Jump to: navigation, search

The following list of items should ideally be accomplished over the summer. If you are interested in working on a task below, please add your name next to the system name, see iodine-ldap for an example. Items in red need downtime notification provided to users and the Sysadmins mailing list at least 24 hours in advance of maintenance. All other items should have a downtime notice provided to the Sysadmins mailing list at least 6 (preferably 24) hours in advance. Maintenance times should be posted here and loaded into Nagios as soon as they are available for general awareness.

General Notes

Downtime notifications for items highlighted in Red should be posted via Iodine at least 24 hours in advance (but ideally as soon as possible). They should include both a start time and estimated end time. Be generous when estimating end-times :).

In general, a good pattern to follow when updating systems is: Reboot, Update, Reboot again, Verify. This way you are sure that the system is in working order before beginning work. The Verify step is also very important to make sure that you leave systems in working order :). When running updates, check for unexpected downgrades in the package list before starting an emerge; these can indicate packages that need a later version unmasked or keyworded. Always run updates from within a screen on the host system in case of an unexpected disconnection.

You should also make sure you have a current backup (If the system is running Guardian, check /root/scripts/backup.log to make sure the last backup is current and successful).

For paired/redundant systems (eg: casey/smith or ns1/ns2), there should be at least 24 hours between the maintenance windows for the two servers to allow time for any subtle problems to surface.

Please do not claim tasks unless you intend to start working on them shortly. Claiming a bunch of jobs right off the bat leaves other people looking for things to do.

Updates

VM Servers

Antipodes

  • update system software
  • install and configure salt for configuration management (Already Done) Samuel Damashek 01:42, 1 August 2014 (EDT)

Galapagos

  • update system software
  • install and configure salt for configuration management (Already Done) Samuel Damashek 01:42, 1 August 2014 (EDT)

Littleblue

  • update system software
  • install and configure salt for configuration management (Already Done) Samuel Damashek 01:42, 1 August 2014 (EDT)

Vega

  • update system software
  • install and configure salt for configuration management (Already Done) Samuel Damashek 01:42, 1 August 2014 (EDT)

Waitaha

  • update system software
  • install and configure salt for configuration management (Already Done) Samuel Damashek 01:42, 1 August 2014 (EDT)

VMs

bugs

  • update system software Samuel Damashek 18:24, 31 July 2014 (EDT)
  • install and configure salt for configuration management Samuel Damashek 20:48, 31 July 2014 (EDT)

casey

  • update system software
  • install and configure salt for configuration management

cups2

  • update system software Samuel Damashek 18:51, 31 July 2014 (EDT)
  • install and configure salt for configuration management Samuel Damashek 20:48, 31 July 2014 (EDT)

fcpsapps

(coordinate system downtime with Mr. Brandon Kosatka and Samuel Damashek)

  • update system software
  • install and configure salt for configuration management Samuel Damashek 00:58, 2 August 2014 (EDT)

gitlab

  • update system software Samuel Damashek 18:51, 31 July 2014 (EDT)
  • install and configure salt for configuration management Samuel Damashek 20:48, 31 July 2014 (EDT)
  • Fix gitlab software to start properly on boot

haimageserver

  • update system software Samuel Damashek 18:24, 31 July 2014 (EDT)
  • install and configure salt for configuration management Samuel Damashek 23:45, 31 July 2014 (EDT)

iodine

  • update system software
  • install and configure salt for configuration management
  • Transfer from fryingpan to sonic

iodine-ldap (Andrew Hamilton)

  • update system software
  • install and configure salt for configuration management
  • Transfer from fryingpan to sonic

ion

(Updates should be coordinated with the ion development team)

  • update system software
  • install and configure salt for configuration management Samuel Damashek 00:58, 2 August 2014 (EDT)
  • Transfer from apocalypse to sonic

license

  • update system software Samuel Damashek 17:57, 31 July 2014 (EDT)
  • install and configure salt for configuration management Samuel Damashek 20:48, 31 July 2014 (EDT)

lists

  • update system software
  • install and configure salt for configuration management

mysql1

  • update system software
  • install and configure salt for configuration management (Already Done) Samuel Damashek 01:42, 1 August 2014 (EDT)

ns1

  • update system software Samuel Damashek 18:24, 31 July 2014 (EDT)
  • install and configure salt for configuration management Samuel Damashek 20:48, 31 July 2014 (EDT)

ns2

  • update system software
  • install and configure salt for configuration management Samuel Damashek 00:58, 2 August 2014 (EDT)
  • Transfer from fryingpan to apocalypse

openafs1

  • update system software
  • install and configure salt for configuration management (Already Done) Samuel Damashek 01:42, 1 August 2014 (EDT)
  • Transfer from apocalypse to sonic

openafs2/openafs6

  • update system software
  • install and configure salt for configuration management
  • Rename openafs6 to openafs2

openafs4

  • update system software
  • install and configure salt for configuration management
  • Transfer from apocalypse to sonic

openafs5

  • update system software
  • install and configure salt for configuration management
  • Transfer from apocalypse to sonic

openldap1

  • update system software Samuel Damashek 17:57, 31 July 2014 (EDT)
  • install and configure salt for configuration management Samuel Damashek 20:48, 31 July 2014 (EDT)

openldap2

  • update system software
  • install and configure salt for configuration management (Already Done) Samuel Damashek 01:42, 1 August 2014 (EDT)
  • Transfer from fryingpan to apocalypse

openvpn

  • update system software
  • install and configure salt for configuration management

smith

  • update system software
  • install and configure salt for configuration management (Already Done) Samuel Damashek 01:42, 1 August 2014 (EDT)

stage64

  • update system software Samuel Damashek 18:24, 31 July 2014 (EDT)
  • install and configure salt for configuration management (make sure this configuration is either imageable or excluded in the newvm-excludes file) Samuel Damashek 20:48, 31 July 2014 (EDT)

steeltoe

  • update system software Samuel Damashek 18:24, 31 July 2014 (EDT)
  • install and configure salt for configuration management Samuel Damashek 20:48, 31 July 2014 (EDT)

www

  • update system software
  • install and configure salt for configuration management

Other Servers

Crate

  • update system software
  • install and configure salt for configuration management

Barrel

  • update system software
  • install and configure salt for configuration management

Guardian

  • update system software
  • install and configure salt for configuration management

openafs3

  • update system software Peter Foley 20:41, 2 Aug 2014 (PDT)
  • install and configure salt for configuration management

mirror

  • update system software
  • install and configure salt for configuration management

Infrastructure Changes

Salt

Nagios/NRPE

Add saltstate to configure Nagios/NRPE. This will likely need to use an accumulator to permit the overriding or addition of checks (such as diskspace and dovecot checks on casey/smith). New saltstate should ideally also fix Bug 878, Bug 1063, and Bug 1066.

LDAP Configuration

NSS LDAP configurations should be pushed to all servers by Salt. This includes making sure the necessary software is installed and pushing appropriate config files. Friendly warning, casey and smith are NOT setup with LDAP accounts. Samuel Damashek 23:45, 31 July 2014 (EDT)

Kerberos Configuration

Kerberos configurations should be pushed to all servers by Salt. This should include everything except the keytab and .k5login files. Samuel Damashek 23:45, 31 July 2014 (EDT)

Multipath Configuration

multipath.conf should be unified (Bug 983) and deployed via Salt.

.bashrc configurations

Push out root .bashrc files to all servers/VMs. Also a good opportunity to resolve Bug 1062. Samuel Damashek 23:45, 31 July 2014 (EDT)

NTP configurations

Push out ntp.conf configurations to all servers. Pushed configuration should make sure to restrict all operations to localhost for security. Samuel Damashek 23:45, 31 July 2014 (EDT)

LDAP configuration changes

All systems should be configured to point to an LDAP master IP (currently ldap-sun.tjhsst.edu, a new name would be good) and the two OpenLDAP IPs. ldap1 and ldap2 should be removed from configuration (as they are redundant with openldap1 and openldap2. This should ideally be done with Salt.

Graduates (Samuel Damashek)

  • Archive 2014 AFS home directories to openafs2/6 Samuel Damashek 10:36, 31 July 2014 (EDT)
  • Deactivate 2014 email accounts (excluding graduated sysadmins) after August 1
  • Archive deactivated 2014 maildirs to Apocalypse
  • generate 2014 maildir archive for transfer to tape
  • generate 2014 AFS archive backup for transfer to tape Samuel Damashek 13:05, 31 July 2014 (EDT)

OpenStack (Andrew Hamilton and Samuel Damashek)

Continue to work on the openstack cluster. Networking remains the major blocking point.