VM storage plan
This is a proposal for a new VM system, discuss here. Sign with four ~ . Here's how the plan goes:
- Some sort of disks hooked up to a server/redundant pair that then export it over the network so that all vm servers have access (iSCSI or NFS)
- Priority is getting some sort of reliable storage
- One option is zvols exported as iSCSI devices offering rollback and snapshots, as well as compression
- HA-NFS backed vm storage is also a possibility since initial tests report excellent performance
- This also limits the single point of failure (HA-NFS is easier than HA-iSCSI)
- Live-migration is as simple as taking the vm down on one server and bringing it up on the other, since it's a block device on every server it's exported to.
- Dom0s run Gentoo. We will have a standardized install procedure and will likely have a portage overlay with custom xen bits (hypervisor/tools/kernel).
- DomU's are Debian except for ltsp. Debian is simpler and we can't keep an image for every VM -- the software varies too much. Compiling on the VM to update is definitely not ideal. Thus, Debian is a better option, and it's arguably more stable.
There are a lot of ways you could create a central management system for this sort of VM setup. You could have a script that brings down VM's on one server and brings them up another, periodically pings hosts, etc. If a host goes down, you could tell iSCSI to stop sharing with that server until further notice and switch to another server. This is just one of several possibilities. There's also always the option of manual management.
All vm servers will have access to a shared nfs filesystem which will contain all the xen vm configurations and lockfiles for the vms preventing simultaneous starting (ask Thomas to elaborate on locking).
Problems so far
- Writes are really slow, reads are pretty fast. Writing: zeros 3.1MB/s random 2.5 MB/s Reading: zeros 82.9 MB/s random 59.9MB/s. Need to figure out why and fix it. Peter Godofsky 17:20, 16 June 2009 (EDT)
- Update: It's not iscsi, it's the DomU. Same horrible disk write speeds are observed when running on sovereign, with the disk on sovereign. Peter Godofsky 17:43, 16 June 2009 (EDT)
- Update also: The same domu on sovereign experiences the same writing slowness. 1.5MB/s write, 67MB/s read to local disks. It appears that t is an issue with the software on the vm servers since both royal and sovereign use lenny dom0 with etch domu kernel. Thomas Georgiou 17:43, 16 June 2009 (EDT)
- Initial NFS testing appears to be getting nice performance: for a 6G file, 73MB/s writes and 101MB/s reads