Gentoo Workstation Image in Xen
A persistent annoyance the lab has dealt with is that of maintaining two (or more) separate user-visible systems -- the workstation image plus a remote access, LTSP, or high performance server -- and keeping them reasonably consistent and updated. In the 2005-2006 year, the remote server (which ran on robustus most of the time) was plagued by a deficiency of desired software, especially libraries for building graphical programs. In the 2006-2007 year, LTSP was used to give a second life to the lab's aging desktops (under half of the machines). Unfortunately this resulted in two competing environments which tended to break if users switched between them.
The lab's currently ongoing move to Xen-based virtualization of services has opened the possibility of a clean, low-maintenance solution to this problem. By encapsulating a copy of the workstation image in a virtual machine (DomU in Xen speak), the workstation system can run on a server with minimal modification, while still allowing the server to be used for a multitude of other services.
- Create a new partition (or logical volume, etc.) on the Xen host (or shared storage)
- Mount it somewhere in the Dom0 (xen host) filesystem
- Use rsync to copy the image from the image server
- Configure the hostname -- this can't happen automatically as it's not booted from dhcp
- Manually run the post-imaging scripts -- be mindful of ssh host keys and Kerberos keytabs
- Unmount, find a kernel, set up the Xen config file (/etc/xen/<name>.cfg)
- Start the new VM and watch it boot via "xm create <name> -c"
- A few further tweaks will be necessary, especially with kernel-related stuff. Those necessary for a Gentoo image are described below.
- The local X server will of course be unable to start, but gdm can still accept XDMCP connections. Ideally gdm would be configured not to start a local X server, but if not, its XKeepsCrashing script will kick in and everything will be fine.
One issue with Xen VMs in general is that glibc uses some processor feature that somehow conflicts with the hypervisor, so Xen emulates it, causing a slowdown. Debian's solution is to have a separate "libc6-xen" package. In Gentoo, add "-mno-tls-direct-seg-refs" to your CFLAGS in /etc/make.conf, and rebuild glibc. See  for lots more info.
Most of the following will depend on what kernel the VM is booted with. It will require one compiled with the Xen patch set; Portage currently has xen-sources up to 2.6.16. But easier is to use one of Debian's prebuild xen kernels, though this leads to a few glitches with Gentoo.
- Debian's initrd apparently mounts /proc and /sys. Gentoo is hardcoded to do this itself (during "rc sysinit", as called from /etc/inittab). My quick solution was to comment out the relevant lines in /sbin/rc (which is a bash script). There may be a cleaner way to fix this.
- The next difficulty comes in getting the OpenAFS kernel module rebuild for the new kernel. Since we're using a Debian kernel, the best way is to copy /usr/src/linux-headers-blah from a Debian VM with the kernel-headers-blah package installed. Of course update the /usr/src/linux symlink. This is almost enough to coax the openafs-kernel ebuild into working.
- One more glitch -- Debian's kernel header scripts somewhere have "gcc-4.1" hardcoded, but that doesn't currently exist on gentoo (August 2007 -- there's either gcc, or gcc-4.1.2). The solution is to change the offending line in /usr/src/linux-headers-blah-xen-686/.kernelvariables
- Once the AFS module is build, you will want to add /lib/modules to the exclude list for updating the VM image, as well as whatever other tweaks you've made.