Gentoo Server Install

Summary

This article will cover the basic process used to install Gentoo Linux onto a CSL 64bit Server System using LVM for disk space management and 802.3ad link aggregation for network redundancy.

Preparation

Backups, Backups, Backups

Make sure that you have a current backup of all data that is stored on the server. In particular, make sure you have a copy of the SSH keys (/etc/ssh/ssh_host_*) as losing these is BAD(TM). The only reason for skipping this step is when you are setting up a brand new server.

Necessary Information

You will also need access to the following information at some point during the install (the values below are what the rest of this guide will be using as example values, be sure to substitute your own in):

 * Processor Architecture: x86_64/amd64/64 bit
 * Hostname: fiordland
 * Domain: csl.tjhsst.edu
 * FQDN: fiordland.csl.tjhsst.edu
 * Server VLAN: 1600
 * IPv4 Address/Netmask: 198.38.17.42/23
 * IPv4 Gateway: 198.38.17.254
 * IPv6 Address Assignment: Stateless Address Autoconfiguration
 * IPv6 Gateway Assignment: Stateless Address Autoconfiguration
 * DNS Server IPs: 198.38.16.40, 198.38.16.41, and 151.188.14.2

Network Hardware Configuration

If the server is currently setup to use 802.3ad bonding or 802.1Q VLAN tagging, you will need to disable these temporarily.

First we need to find out which switchports the server is connected to. On core0 run:

 TJHSST-4500MSL#sh run | inc Fiordland
  description Port8 Fiordland-1,2
  description Gi2/3 Fiordland-1
  description Gi2/4 Fiordland-2
  description Fa5/19 Fiordland-ilo

Then we look at the configuration of the individual switchports.

 TJHSST-4500MSL#sh run inter Gi2/3
 Building configuration...
 
 Current configuration : 305 bytes
 !
 interface GigabitEthernet2/3
  description Gi2/3 Fiordland-1
  switchport access vlan 1600
  switchport trunk encapsulation dot1q
  switchport trunk allowed vlan 16,1600
  switchport mode trunk ! <--This line indicates that vlan tagging is enabled
  logging event link-status
  channel-protocol lacp
  channel-group 8 mode active ! <--This line indicates that bonding is enabled
  spanning-tree portfast trunk
 end

Disable bonding and VLAN tagging on at least one interface:

 TJHSST-4500MSL#conf t
 Enter configuration commands, one per line.  End with CNTL/Z.
 TJHSST-4500MSL(config)#inter Gi2/3
 TJHSST-4500MSL(config-if)#no channel-group 8 mode active
 TJHSST-4500MSL(config-if)#switchport mode access
 TJHSST-4500MSL(config-if)#switchport access vlan 1600 ! <--replace with server's VLAN
 TJHSST-4500MSL(config-if)#spanning-tree portfast
 %Warning: portfast should only be enabled on ports connected to a single
  host. Connecting hubs, concentrators, switches, bridges, etc... to this
  interface  when portfast is enabled, can cause temporary bridging loops.
  Use with CAUTION
 %Portfast has been configured on GigabitEthernet2/3 but will only
  have effect when the interface is in a non-trunking mode.
 TJHSST-4500MSL(config-if)#exit
 TJHSST-4500MSL(config)#exit
 TJHSST-4500MSL#wr mem

The switchport configuration should now look something like this:

 TJHSST-4500MSL#sh run inter Gi2/3
 Building configuration...
 
 Current configuration : 271 bytes
 !
 interface GigabitEthernet2/3
  description Gi2/3 Fiordland-1
  switchport access vlan 1600
  switchport trunk encapsulation dot1q
  switchport trunk allowed vlan 16,1600
  switchport mode access
  logging event link-status
  channel-protocol lacp
  spanning-tree portfast
 end

RAID Controller Configuration

If the system you are installing has a hardware RAID controller and you wish to make any changes to the array configuration, do that now. You will need to reboot the server to access the controller's BIOS (F8 on the HP Servers). From the controller's BIOS, you can change the RAID level and number of drives in each array.

Booting the Server

Boot the server to the Gentoo install media. In most cases, you will be doing this via either a USB drive or a CD-ROM. In the CSL, you also have the option of using the Netboot server off of Steeltoe. If you would like to be able to remove the media after the server has finished booting, make sure to specify the docache argument. If you are installing over a serial console (such as HP's iLO or Sun's LOM), make sure to specify the console= argument with the appropriate serial device and settings (when in doubt, try console=ttyS0,9600,8,n,1 or console=ttyS1,9600,8,n,1; if the output looks garbled, try changing the baud rate from 9600 to 115200).

Take the defaults for any questions the live media asks until you reach the red livecd prompt.

Configuring Network Access

The first step in the installation is configuring network access. Execute the following commands to bring up networking.

 ip link set eth0 up
 ip addr add 198.38.17.42/23 dev eth0
 ip route add default via 198.38.17.254 dev eth0
 echo "nameserver 198.38.16.40" > /etc/resolv.conf
 ping -c1 google.com

You should receive output similar to the following from the ping command:

 PING google.com (74.125.228.101) 56(84) bytes of data.
 64 bytes from iad23s08-in-f5.1e100.net (74.125.228.101): icmp_seq=1 ttl=55 time=17.1 ms
 
 --- google.com ping statistics ---
 1 packets transmitted, 1 received, 0% packet loss, time 0ms
 rtt min/avg/max/mdev = 17.163/17.163/17.163/0.000 ms

Setup SSH Access

While serial consoles are great for emergency access, working on them for long periods of time can become very annoying in a hurry. For that reasons, we are going to setup SSH access to our server via the LiveCD environment.

To start, unless this is a new install, you should restore a copy of the server's SSH keys to the LiveCD environment. This will prevent you from having to work around messy SSH errors.

 cd /etc/ssh/
 scp <backupserver>:/path/to/sshkeys .

Next you need to set a root password on the LiveCD. This should be something relatively secure as it is the only security you have against your install being compromised remotely.

 passwd

Finally, start sshd and reconnect to the server via SSH:

 /etc/init.d/sshd start

Date and Time

Verify that the date and time are correct:

 date

Note that the date and time are given in UTC which is 5 hours ahead of Eastern Standard Time (4 hours ahead during Daylight Savings Time). If the date and time are not correct, adjust them accordingly:

 date MMDDhhmmYYYY #MM month, DD day, hh hours, mm minutes, YYYY year

Again, make sure to use UTC when setting the date and time. Be especially careful of date wrap-around if installing a server at night.

Disk Setup

Disk Partitioning

You now need to identify the disk(s) on which you will be installing Gentoo Linux. On an older HP Server, the hardware RAID arrays generally appear as /dev/cciss/cXdY where X and Y are the controller and array number respectively. For newer HP Servers, the hardware RAID arrays will appear as /dev/sdX.

For Servers without a hardware RAID controller, the individual drives will usually appear as /dev/sd{a,b,c,etc}. In this case, you will likely want to setup mdadm software RAID for data redundancy and server availability.

You can use fdisk -l /path/to/disk to get some additional information (like capacity) about a connected drive.

For the rest of this article, we will assume that you will be installing linux to /dev/cciss/c0d0.

Run the following command to start fdisk targeted at your install drive:

 fdisk /dev/cciss/c0d0

Use p to print the current partition layout and then use d to delete any existing partitions. NOTE: if you see the following message when you start fdisk, use o to create a new DOS partition table before you proceed:

 WARNING: GPT (GUID Partition Table) detected on '/dev/cciss/c0d0'! The util fdisk doesn't support GPT. Use GNU Parted.

Use n to create the following partitions

Action	Number	First Cylinder	Last Cylinder
Primary	1	default	+100M
Primary	2	default	+5G
Primary	3	default	default

 Command (m for help): n
 Command action
    e   extended
    p   primary partition (1-4)
 p
 Partition number (1-4): 1
 First cylinder (1-26460, default 1): 
 Using default value 1
 Last cylinder, +cylinders or +size{K,M,G} (1-26460, default 26460): +100M
 
 Command (m for help): n
 Command action
    e   extended
    p   primary partition (1-4)
 p
 Partition number (1-4): 2
 First cylinder (15-26460, default 15): 
 Using default value 15
 Last cylinder, +cylinders or +size{K,M,G} (15-26460, default 26460): +5G
 
 Command (m for help): n
 Command action
    e   extended
    p   primary partition (1-4)
 p
 Partition number (1-4): 3
 First cylinder (666-26460, default 666): 
 Using default value 666
 Last cylinder, +cylinders or +size{K,M,G} (666-26460, default 26460): 
 Using default value 26460

Use t to change the type on partition 3 to 8e (Linux LVM):

 Command (m for help): t
 Partition number (1-4): 3
 Hex code (type L to list codes): 8e
 Changed system type of partition 3 to 8e (Linux LVM)

Use p to verify that the partition table looks similar to the following, then use w to write the changes to disk and close fdisk:

 Command (m for help): p
 
 Disk /dev/cciss/c0d0: 218.5 GB, 218501038080 bytes
 256 heads, 63 sectors/track, 26460 cylinders
 Units = cylinders of 16128 * 512 = 8257536 bytes
 Disk identifier: 0x00000000
 
            Device Boot      Start         End      Blocks   Id  System
 /dev/cciss/c0d0p1               1          14      112864+  83  Linux
 /dev/cciss/c0d0p2              15         665     5249664   83  Linux
 /dev/cciss/c0d0p3             666       26460   208010880   8e  Linux LVM
 
 Command (m for help): w
 The partition table has been altered!
 
 Calling ioctl() to re-read partition table.
 Syncing disks.

LVM Setup

We will be using LVM to dynamically manage the bulk of our disk space. LVM allows disk space to be reallocated among different partitions without having to repartition the physical harddisk.

Run the following commands to create our LVM volgroup:

 pvcreate /dev/cciss/c0d0p3
 vgcreate vg{hostname} /dev/cciss/c0d0p3

Create the following logical volumes to separate out important parts of the OS onto separate partitions:

 lvcreate -L 5G -n usr vg{hostname}
 lvcreate -L 4G -n var vg{hostname}
 lvcreate -L 1G -n swap vg{hostname}

Filesystem Creation

We will be using ext4 for most of our filesystems; it is a well-tested and stable filesystem with a decent feature set.

First we create an ext2 filesystem for our /boot partition. We use ext2 for the /boot partition because it is very small and infrequently modified.

 mkfs.ext2 /dev/cciss/c0d0p1

Make a swap filesystem on our swap partition and then activate it:

 mkswap /dev/vg{hostname}/swap
 swapon /dev/vg{hostname}/swap

Make ext4 filesystems on the rest of our partitions:

 mkfs.ext4 /dev/cciss/c0d0p2
 mkfs.ext4 /dev/vg{hostname}/usr
 mkfs.ext4 /dev/vg{hostname}/var

Mounting the Filesystems

We will be mounting all of our filesystems with /mnt/gentoo/ as the root of our new installation:

 mount /dev/cciss/c0d0p2 /mnt/gentoo
 cd /mnt/gentoo/
 mkdir boot usr var
 mount /dev/cciss/c0d0p1 /mnt/gentoo/boot
 mount /dev/vg{hostname}/usr /mnt/gentoo/usr
 mount /dev/vg{hostname}/var /mnt/gentoo/var

Installation Files

Download Base Files

Grab the latest versions of the appropriate stage3 tarball for the server's architecture as well as the latest portage snapshot. NOTE - the below stage3 URL is most likely out of date; check your local gentoo mirror for the latest version.

 cd /mnt/gentoo/
 wget http://mirror.tjhsst.edu/gentoo/releases/amd64/current-stage3/stage3-amd64-20121013.tar.bz2
 wget http://mirror.tjhsst.edu/gentoo/snapshots/portage-latest.tar.bz2

Unpack Base Files

Unpack the stage3 tarball and the portage snapshot:

 cd /mnt/gentoo/
 tar -xvf stage3-amd64-20121013.tar.bz2
 #Now would be a good time to get some coffee
 tar -xvf portage-latest.tar.bz2 -C /mnt/gentoo/usr/
 #Aaaand time for another coffee break...
 rm stage3-amd64-20121013.tar.bz2 portage-latest.tar.bz2

Preparing chroot environment

Copy the server template make.conf from the CSL Default Configs to /etc/portage/make.conf then edit it to set appropriate USE and MAKEOPTS values.

Make the overlay directory

 mkdir -p /mnt/gentoo/usr/local/portage/overlay

Copy the resolv.conf file and the SSH host keys from the livecd environment to the chroot environment.

 cp /etc/resolv.conf /mnt/gentoo/etc/
 cp -av /etc/ssh/ssh_host_* /mnt/gentoo/etc/ssh/

Mount additional filesystems:

 mount -t proc none /mnt/gentoo/proc
 mount -o bind /dev /mnt/gentoo/dev
 mount -o bind /sys /mnt/gentoo/sys

Chroot

chroot into the installation environment:

 chroot /mnt/gentoo /bin/bash
 env-update
 source /etc/profile
 export PS1="(chroot)$PS1"

Edit /etc/locale.gen and uncomment the two en_US locales, then run the following command to update the generated locales.

 locale-gen

Set the timezone in the installation environment

 cp /usr/share/zoneinfo/America/New_York /etc/localtime

Copy over the CSL Overlay, then update the portage tree:

 rsync -rv rsync://haimageserver.csl.tjhsst.edu/overlay/ /usr/local/portage/overlay/
 emerge --sync

Kernel Configuration and Installation

Fetching the Kernel Sources

Emerge git

 emerge -a git

Clone the kernel sources

 cd /usr/src/
 git clone git://haimageserver.csl.tjhsst.edu/linux.git linux.git
 eselect kernel set linux.git
 cd linux
 git checkout v3.7

Configuring the Kernel

You can either use a CSL stock kernel configuration or build your own kernel configuration. To use a stock kernel configuration, copy the appropriate config file to /usr/src/linux/.config

If you are building your own kernel configuration, check the wiki page for your server model and planned roles for recommendations on driver and other configuration choices.

Building and Installing the Kernel

Execute the following command to build the kernel; adjust 5 to be equal to the value you set for MAKEOPTS in /etc/portage/make.conf

 make -j5

Once again now's a good time to take a break while the kernel compiles. When it is done, run the following commands to install the kernel and modules. NOTE: the suffix for the config and the kernel files below should match the module directory name in /lib/modules/

 make modules_install
 cp .config /boot/config-3.7.0-kvm
 cp arch/x86/boot/bzImage /boot/kernel-3.7.0-kvm

Building and Installing the Initramfs

We need to build an initramfs to support our separate /usr partition. The easiest way to do this is using the latest version of Genkernel. First, we need to install it:

 mkdir -p /etc/portage/package.keywords
 echo "sys-kernel/genkernel" >> /etc/portage/package.keywords/genkernel
 emerge -a genkernel

Then we just need to tell genkernel to build an initramfs. Note, if you are using mdadm, you will need to add --mdadm to the genkernel arguments.

 genkernel --lvm initramfs

Symlinking the Kernel and initramfs

We create easy-to-remember symlinks for the kernel and the initramfs:

 cd /boot/
 ln -snf kernel-3.7.0-kvm gentoo
 ln -snf initramfs-genkernel-x86_64-3.2.0-kvm gentoo-initramfs

Essential System Software

There are a few applications that are essential to the functionality of our new system. We should install them now. They include:

• grub - Boot loader
• syslog-ng - System logger
• vixie-cron - cron daemon for scheduled tasks
• logrotate - rotates system logs
• iproute2 - needed for configuring network interfaces
• lvm2 - needed if you're using LVM for disk management
• mdadm - needed if you're using MDADM for software RAID
• ifenslave - needed for bonding network interfaces
• vconfig - needed for configuring VLAN tagging

Install an appropriate selection from the above list using a command similar to the following:

 emerge -a grub syslog-ng vixie-cron logrotate iproute2 lvm2 ifenslave

System Configuration

We need to update a number of configuration files before our system can be considered operational.

/etc/fstab

Add / edit the following lines in /etc/fstab:

 /dev/cciss/c0d0p1       /boot           ext2            noatime         1 2
 /dev/cciss/c0d0p2       /               ext4            noatime         0 1
 /dev/vg{hostname}/usr    /usr            ext4            noatime         0 2
 /dev/vg{hostname}/var    /var            ext4            noatime         0 2
 /dev/vg{hostname}/swap   none            swap            sw              0 0

/etc/conf.d/hostname

edit /etc/conf.d/hostname and set the system's hostname:

 hostname="fiordland"

/etc/conf.d/hwclock

edit /etc/conf.d/hwclock and uncomment the following two lines and set both to YES:

 clock_hctosys="YES"
 clock_systohc="YES"

/etc/timezone

Preserve the timezone across updates of sys-libs/timezone-data with the following command:

 echo "America/New_York" > /etc/timezone

/etc/conf.d/net

Setup the system's networking configuration. You may want to consult the server's livedoc page as well as any role pages (VM server, storage server) for detailed information on networking configuration for a particular server. The below configuration is for redundant access (bonding/etherchannel) to a single network or VLAN:

 slaves_bond0="eth0 eth1"
 config_bond0="198.38.17.42/23"
 routes_bond0="default via 198.38.17.254"
 dns_servers_bond0="198.38.16.40 198.38.16.41 151.188.14.2"
 dns_search_bond0="csl.tjhsst.edu tjhsst.edu sun.tjhsst.edu"

Remember to create any needed networking scripts:

 cd /etc/init.d/
 ln -snf net.lo net.bond0

/etc/inittab

Find and uncomment the following line to enable a console on the iLO Virtual Serial Port:

 s0:12345:respawn:/sbin/agetty 9600 ttyS0 vt100

/etc/securetty

Make sure the following line is in /etc/securetty to allow root logins on the iLO Virtual Serial Port:

 ttyS0

Boot Services

Setup various services to start at boot:

 rc-update add lvm boot
 
 rc-update add net.bond0 default
 rc-update add sshd default
 rc-update add syslog-ng default
 rc-update add vixie-cron default

Root Password

Set the root password (You'll be really sorry if you forget this step...)

 passwd

/boot/grub/grub.conf

Add / edit the following lines in /boot/grub/grub.conf

 default 0
 timeout 30
 
 title Gentoo Linux / KVM
 root (hd0,0)
 kernel /boot/gentoo real_root=/dev/cciss/c0d0p2 dolvm bonding.mode=4 bonding.miimon=100 console=tty0 console=ttyS0,9600,8,n,1
 initrd /boot/gentoo-initramfs

Grub MBR Installation

Run the following commands to install Grub into the MBR:

 grep -v rootfs /proc/mounts > /etc/mtab
 grub-install --no-floppy /dev/cciss/c0d0

Note in some cases grub-install doesn't function, particularly when using mdadm. In this case, you may have to manually install Grub into the MBR via the grub shell.

 grub
 grub> root (hd0,0)
 grub> setup (hd0)
 grub> quit

If using mdadm, you will want to repeat the above steps to install Grub into the MBR on each drive so that the system remains bootable in the event of a drive loss.

Exit chroot and Reboot

Exit the chroot, unmount all filesystems, and reboot to your new installation:

 exit
 cd
 umount /mnt/gentoo/boot
 umount /mnt/gentoo/dev
 umount /mnt/gentoo/proc
 umount /mnt/gentoo/sys
 umount /mnt/gentoo/usr
 umount /mnt/gentoo/var
 umount /mnt/gentoo
 sync
 sync
 reboot

Note that most failures to reboot successfully are due to kernel issues, particularly missing drivers. If your system starts to boot the kernel but then crashes, make sure you have included all the necessary drivers, particularly HDD drivers in the kernel (not as modules). Also if you are using MDADM or LVM, make sure you have the appropriate drivers selected and applications installed.

Postinstall Networking

If your networking configuration specifies bonding, you will need to re-enable bonding on the switch before you can access the server over the network. First we assign both interfaces to a channel-group:

 TJHSST-4500MSL#conf t
 Enter configuration commands, one per line.  End with CNTL/Z.
 TJHSST-4500MSL(config)#inter Gi2/3
 TJHSST-4500MSL(config-if)#channel-protocol lacp
 TJHSST-4500MSL(config-if)#channel-group 8 mode active
 TJHSST-4500MSL(config-if)#switchport mode access
 TJHSST-4500MSL(config-if)#switchport access vlan 1600
 TJHSST-4500MSL(config-if)#spanning-tree portfast
 TJHSST-4500MSL(config-if)#exit
 TJHSST-4500MSL(config)#inter Gi2/4
 TJHSST-4500MSL(config-if)#channel-protocol lacp
 TJHSST-4500MSL(config-if)#channel-group 8 mode active
 TJHSST-4500MSL(config-if)#switchport mode access
 TJHSST-4500MSL(config-if)#switchport access vlan 1600
 TJHSST-4500MSL(config-if)#spanning-tree portfast
 TJHSST-4500MSL(config-if)#exit

Next we configure the channel-group interface:

 TJHSST-4500MSL(config)#inter Port 8
 TJHSST-4500MSL(config-if)#switchport mode access
 TJHSST-4500MSL(config-if)#switchport access vlan 1600
 TJHSST-4500MSL(config-if)#spanning-tree portfast
 TJHSST-4500MSL(config-if)#exit
 TJHSST-4500MSL(config)#exit

You can verify the status of the Etherchannel with the following command:

 TJHSST-4500MSL#show etherchannel 8 sum
 Flags:  D - down        P - bundled in port-channel
         I - stand-alone s - suspended
         R - Layer3      S - Layer2
         U - in use      f - failed to allocate aggregator
 
         M - not in use, minimum links not met
         u - unsuitable for bundling
         w - waiting to be aggregated
         d - default port
 
 
 Number of channel-groups in use: 19
 Number of aggregators:           19
 
 Group  Port-channel  Protocol    Ports
 ------+-------------+-----------+-----------------------------------------------
 8      Po8(SU)         LACP      Gi2/3(P)    Gi2/4(P)

Additional Software

There are a number of additional software packages that are required to make our system fully operational. Start by copying the portage configurations from the CSL Default Configs or from a backup.

 rsync -rv hostname:/path/to/package.keywords /etc/portage/
 rsync -rv hostname:/path/to/package.use /etc/portage/

Next update portage:

 emerge -a1 portage

Make sure nano gets kept around (it is an editor that is available without /usr):

 emerge --noreplace nano

New Software

Install some new packages to provide debugging, monitoring, and useful functionality:

• eix - easily search for packages
• genlop - gentoo log parser
• gentoolkit - a selection of useful portage utilities
• nss_ldap - access user account information stored in LDAP
• heimdal - an implementation of Kerberos
• pam_krb5 - use Kerberos for password validation
• app-misc/screen - terminal multiplexer
• iftop - graph network traffic
• nrpe - nagios remote plugin executor
• nsca - nagios service check acceptor
• nagios-plugins - nagios monitoring plugins
• nmap - network mapping utility
• bind-tools - a collection of DNS utilities
• ntp - implementation of the network time protocol
• pciutils - list information on PCI cards
• usbutils - list informaton on USB devices
• lsof - manage file accesses
• htop - interactive process viewer

 emerge -a eix genlop gentoolkit nss_ldap heimdal pam_krb5 app-misc/screen iftop nrpe nsca nagios-plugins nmap bind-tools ntp pciutils usbutils lsof htop

Software Updates

Start a screen and begin a world update to apply any software updates that have been released since the stage3 was built.

 screen
 #now inside the screen
 emerge -auND @world
 emerge -a @preserved-rebuild
 emerge -a --depclean
 revdep-rebuild -- -a

Configuration Files

Copy the following files/directories from the CSL Default Configs

 /etc/krb5.conf 
 /etc/ldap.conf 
 /etc/ntp.conf 
 /etc/conf.d/ntp-client 
 /etc/portage/postsync.d/
 /etc/nsswitch.conf
 /etc/ssh/ssh_config
 /etc/ssh/sshd_config
 /etc/pam.d/system-auth
 /etc/issue
 /etc/nagios/nrpe.cfg
 /etc/nagios/send_nsca.cfg
 /root/.k5login
 /root/scripts/

/etc/issue

Edit /etc/issue to reflect the system's proper FQDN

/etc/nagios/nrpe.cfg

Edit /etc/nagios/nrpe.cfg, set the system's IP. Adjust check values if needed (eg, adjust user limits higher on a remote access server).

/etc/security/access.groups

Create /etc/security/access.groups with the appropriate access groups, one per line (should be at least the hostname group and the allaccess group).

 allaccess
 fiordland

/root/.k5login

Add any needed /root principals to /root/.k5login to grant them root access via ksu.

/etc/krb5.keytab

Restore /etc/krb5.keytab if you have a backup; otherwise use the following command to generate a new one:

 ktutil get -p ahamilto/admin host/fiordland.csl.tjhsst.edu

Verify that the permissions on /etc/krb5.keytab are root:root and 0600

 chown root:root /etc/krb5.keytab
 chmod 0600 /etc/krb5.keytab