Royal and Fiordland are the lab's storage servers. As of November 2010 HA-iSCSI is down indefinitely.
Royal and Fiordland each have 2 QLA2200 HBAs in them. Each server has a single path to each Fibre Channel Switch (Powervault 51F). Each switch then has a connection to one of the 224Fs controllers. This provides two completely separate Fibre Channel fabrics from the hard drive itself to the server. Any switch, IO module, or HBA can explode and HA-iSCSI will still be up.
Currently, 3 Dell 224F JBODs are being used. The 660Fs with their smart controllers were giving problems: trying to reset the 224Fs, not liking to be configured and such. Thumbtack and Soupspoon have 14x73GB drives and Bigfish has 14x146GB drives.
Currently, the "iSCSI Enterprise Target" is used (http://iscsitarget.sourceforge.net/). This is the iscsitarget package in portage and is also known as iet. IETD provides two IO methods for targets, BlockIO and FileIO. FileIO uses the Linux page cache (filesystem cache) allowing reads to be cached, but everything must be copied twice. BlockIO does not use the page cache and thus does not have to copy data twice, resulting in faster write speeds, but slightly reduced read speeds. Since VMs can cache reads, but not writes, it was decided to go for faster write speeds and use BlockIO.
In order to make iSCSI HA, Pacemaker is used (on top of OpenAIS/Corosync) (http://clusterlabs.org/wiki/Main_Page). Under normal conditions, one server runs haicsci1 and the other haicsi2. If one server is unable to serve haiscsi for any reason, then the other server takes over the failed servers' haiscsi. This consists of starting a Linux software raid array, hot-adding the iSCSI target, and bringing up the service IP.
In addition to HA-iSCSI, the storage servers provide imageserver (not yet HA) and HA-NFS used for /usr/portage on vms and vm configs and locks. To keep locks in place, locks are stored on the shared array instead of on local storage.
Bringing up a storage server
After booting a storage server (royal or fiord), perform these steps to allow it to run services again:
- modprobe qla2xxx #loads the hba modules, takes a while to load and detect all disk drives
- multipath #creates multipath devices for each drive
- /etc/init.d/multipathd start #Starts the multipath path checker daemon (this may not be needed)
- /etc/init.d/ietd start #Start the iSCSI target daemon
- /etc/init.d/corosync start #Start the corosync (which automatically starts pacemaker)
Monitoring Cluster Status
Run 'crm_mon' on a cluster node to real time view resource status.