Warning Livedoc is no longer being updated and will be deprecated shortly. Please refer to https://documentation.tjhsst.edu.

Difference between revisions of "Email"

From Livedoc - The Documentation Repository
Jump to: navigation, search
(Added quota information.)
(Added 'converted from book' template.)
Line 1: Line 1:
 +
{Converted from book}
 +
 
=Overview=
 
=Overview=
 
The current '''TJ email system''' is an idea originally designed by Aman Gupta (class of '04), and implemented by Andrew Deason (class of '06) with the help of several staff members, including Richard Washer and Susan Beasley. The actual mail is stored on a [[Debian]] GNU/Linux server, and is also accessed there, but all of the user information and authentication is done by a Windows Active Directory server.
 
The current '''TJ email system''' is an idea originally designed by Aman Gupta (class of '04), and implemented by Andrew Deason (class of '06) with the help of several staff members, including Richard Washer and Susan Beasley. The actual mail is stored on a [[Debian]] GNU/Linux server, and is also accessed there, but all of the user information and authentication is done by a Windows Active Directory server.

Revision as of 15:51, 4 February 2006

{Converted from book}

Overview

The current TJ email system is an idea originally designed by Aman Gupta (class of '04), and implemented by Andrew Deason (class of '06) with the help of several staff members, including Richard Washer and Susan Beasley. The actual mail is stored on a Debian GNU/Linux server, and is also accessed there, but all of the user information and authentication is done by a Windows Active Directory server.

Previously, the CSL only hosted mail for the CSL itself; mail that was only accesible to people with Systems Lab accounts. Today, lab-only mail does not exist anymore, and everyone gets their email off of the same system, all with @tjhsst.edu addresses.

Samba

As we now support mail for all of TJHSST, we found it necessary to authenticate against the LAN, which is slowly moving away from Novell and instead to Windows 2003 Active Directory Services. We opted to authenticate using ADS, and found that Winbind, a part of Samba, provided the authentication support we needed. Both NSS and PAM modules were provided allowing full interoperability with our Linux system.

Configuration

There are several configuration files used to facilitate integrating users from the Windows 2003 ADS into Linux, spanning different systems.

smb.conf

The first file we'll deal with is Samba's, located at /etc/samba/smb.conf. Note that the configuration for Winbind as well as the overall configuration for Samba itself is located within this file. Here are some of the more important directives.

 workgroup = LOCAL
 realm = LOCAL.TJHSST.EDU

This just declares which Windows workgroup we are a part of, and which Kereros realm. Note that the kerberos realm is not necessarily the name of any actual machine, but is just a string to represent a realm. (Even though in our case, local.tjhsst.edu is a valid DNS entry, it doesn't have to be.)

 security = ADS

This turns on security by Active Directory Service from the Windows system.

 password server = local.tjhsst.edu
 encrypt passwords = true

These lines indicate which password server to use (a round-robin address in this case), and to to enable passwords. Obviously enabling passwords is a good idea, and even more so with Microsoft servers, since many of their systems no longer support plaintext authentication (according to the smb.conf manpage).

 obey pam restrictions = yes

We will be using PAM as part of our configuration (see below for PAM config), so we want to enable this.

 invalid users = root

Obviously we don't want root authenticating via Samba! If, by chance, there is a user in the ADS called 'root', then the password for that ADS user would be checked, and someone might be able to gain the 'root' username, by a password that is not the local root password on the machine. That is clearly very bad, so here we disable hat from happening.

 unix password sync = no

This option would try to sync the Samba passwords of users with the local unix passwords. But since the users we are trying to integrate are not local unix users, we don't want this.

 idmap uid = 10000-20000
 idmap gid = 10000-20000

These lines set the range that Winbind will use for the UIDs and GIDs of the various users and groups retrieved from Samba. Make sure that these ranges are sure not to conflict with any other UID/GIDs on the local machine!

 template shell = /usr/bin/pine
 template homedir = /home/mail/users/%U

These set the environment for the user. The shell is set to pine, so if users login via SSH, they will immediately be dropped into pine to check their mail, without given shell access. (Write permission is also denied to each user's .pinerc, so they can't change the configuration to run arbitrary commands.) The home directory is merely the user's home directory, mostly just where the mail is stored.

 winbind use default domain = yes

This allows users to login without specifying their Windows domain in their username, which would make things a real pain when logging in. With this option turned on, a user can just log in as 'adeason', and they will be given the default domain. (Default option is to make usernames have their domain prepended to them before a separator, which is '+' by default.)

krb5.conf

Even though krb5.conf is a Kerberos configuration file, and not really part of Samba, it is necessary for Winbind to function, so it will be described here briefly. It is located at /etc/krb5.conf.

 default_realm = LOCAL.TJHSST.EDU

This simply sets the default realm for authenticating against.

 [realms]
     LOCAL.TJHSST.EDU = {
         kdc = LOCAL.TJHSST.EDU
         admin_server = LOCAL.TJHSST.EDU
     }

Before, it was mentioned that the Kerberos realm did not need to be an actual machine. Here is where we specify which machines each realm uses. For our purposes, the Kerberos servers happen to have the same name as the realm, but such is not always true.

PAM

Winbind itself is used as two things in our system: a PAM module called pam_winbind.so, and an NSS mechanism (in /etc/nsswitch.conf as 'winbind'). Explained briefly here is how to use the PAM module.

For example, the /etc/pam.d/ssh file:

 auth            required        pam_nologin.so
 auth            required        pam_env.so
 @include common-auth
 auth            required        pam_winbind.so use_first_pass
 
 @include common-account
 account         required        pam_winbind.so
 
 @include common-session
 session         required        pam_limits.so
 session         required        pam_winbind.so
 session         optional        pam_lastlog.so
 session         optional        pam_motd.so
 session         optional        pam_mail.so standard noenv
 
 @include common-password

The important lines are the ones referencing pam_winbind.so, but the others are included for reference. The use_first_pass option in the auth section is given so users don't get two prompts for passwords (this is similar to a kerberos authentication scheme). Since the password the user typed in was already passed to pam_unix.so (which isn't directly in this file, but is in the '@include'ed file common-auth), the module pam_winbind.so also needs the password. Normally it would prompt the user for a password, but that option uses the password already read in from the user.

Running

There are three daemons for the Samba/Winbind system: nmbd, smbd, and winbindd. The first two are controlled by /etc/init.d/samba, and winbindd is controlled by /etc/init.d/winbind. They both understand the standard init script options of start, stop, restart, and force-reload, and the samba init script also understands the command 'reload'.

Postfix

At the heart of the new mail system is the excellent mail server Postfix. It was chosen over alternatives for its robustness, ease of configuration, and ease of use.

Configuration

Postfix configuration is accomplished through several files in /etc/postfix/. Here we will outline their general structure. After changing most configuration files, Postfix must reload its configuartion files with the command postfix reload.

main.cf

This configuration file tends to be the longest, as it houses nearly all of the options one might be concerned with. For simplicity's sake, we will outline here only the most important directives.

 myhostname = mail.tjhsst.edu

Used to set the hostname of the mailserver. This is used in many later configuration parameters, and is what is sent by the mailserver when it is talking to other servers.

 mydomain = tjhsst.edu

The domain name this mailserver is hosting.

 myorigin = $mydomain

Specifies where mail that does not have a domain already is assumed to be originating from.

 inet_interfaces = all

Determines which interfaces Postfix will listen on. We want it to listen on all interfaces.

 mydestination = $myhostname, localhost.$mydomain, $mydomain, *.$mydomain, lan.$mydomain, mail-bkup.$mydomain

One of the more important parameters, this one specifies which domains to consider as local mail. The * doesn't always seem to work, so we often must add domains manually.

Caution! Do not specify virtual hosts here, or they will be consided local domains and will not be processed through the virtual maps.

 local_recipient_maps = $alias_maps $virtual_mailbox_maps

Determines how postfix determines which users are local and can be delivered to. Note that this is different from the default.

 mynetworks = 198.38.16.0/20, 127.0.0.0/8

This parameter controls who may relay mail through us. We have decided to let anybody inside the building have access.

 virtual_maps = hash:/etc/postfix/virtual_maps

This points to a file that contains mappings for virtual domains. The format of this file will be discussed later.

 virtual_mailbox_base = /home/mail/user/
 virtual_mailbox_maps = hash:/etc/postfix/virtual_mailbox_maps

These two paramters control where mail is delivered. Postfix is instructed to look at the virtual_mailbox_maps file for information on where mailboxes are stored per user.

 home_mailbox = Maildir/

This indicates that we have decided to use the Qmail-format Maildir/ form of storing one message per file in directories inside each user's home directory.

 header_checks = pcre:/etc/postfix/header_checks
 body_checks = pcre:/etc/postfix/body_checks

These two point to files that contain Perl-compatible regular expressions that are matched against incoming e-mails. Those that match are dropped.

 content_filter = smtp:localhost:20024

This parameter specifies that all email should be sent via smtp to localhost on port 20024 before being delivered. This is used to scan all email for viruses, and will be discussed in more detail in a later section.

 transport_maps = hash:/etc/postfix/transport
 mailman_destination_recipient_limit = 1

These two parameters allow operation with Mailman, which is used to run self-managed distribution lists.

 broken_sasl_auth_clients = yes
 smtpd_sasl_auth_enable = yes
 smtpd_sasl_security_options = noanonymous
 smtpd_sasl_local_domain =
 smtpd_recipient_restrictions = permit_sasl_authenticated, permit_mynetworks, reject_unauth_destination

These parameters control the use of SASL authentication, which is used for SMTP authentication so that users can relay through our mailserver outside the building. This will be discussed in more detail later.

 smtp_use_tls = yes
 smtp_tls_note_starttls_offer = yes
 smtp_tls_CAfile = /etc/postfix/ssl/tjhsst.ca.pem
 smtp_tls_key_file = /etc/postfix/ssl/mail.pem
 smtp_tls_cert_file = /etc/postfix/ssl/mail.pem
 smtp_tls_loglevel = 1
 smtpd_use_tls = yes
 smtpd_tls_key_file = /etc/postfix/ssl/mail.pem
 smtpd_tls_cert_file = /etc/postfix/ssl/mail.pem
 smtpd_tls_CAfile = /etc/postfix/ssl/tjhsst.ca.pem
 smtpd_tls_loglevel = 1
 smtpd_tls_received_header = yes
 smtpd_tls_session_cache_timeout = 3600s
 tls_random_source = dev:/dev/urandom
 smtpd_sasl_tls_security_options = $smtpd_sasl_security_options
 smtpd_tls_auth_only = yes

These parameters control TLS, or Transport Layer Security, also known as SSL. They provide encryption over SMTP channels if the remote server or client supports it. It should be noted that the last one specifies that SMTP authentication, discussed in the previous point, is only available if TLS is enabled.

 smtpd_helo_required = yes
 strict_rfc821_envelopes = yes
 smtpd_client_restrictions =
     reject_rbl_client sbl-xbl.spamhaus.org,
     reject_rbl_client relays.ordb.org,
     reject_rbl_client list.dsbl.org

These control anti-spam techniques common on the Internet.

master.cf

This configuration file describes how the Postfix daemon should run and determines which ports it should be listening on for connections and how to handle those connections. The default format of this file is perfectly fine, with only two additions necessary in our setup.

 localhost:20025   inet  n       -       -       -       -       smtpd -o content_filter
 mailman unix    -       n       n       -       -       pipe flags=FR user=list:list argv=/home/mail/mailman/bin/postfix-to-mailman.py ${nexthop} ${user}

The first line is related to our use of the content_filter = smtp:localhost:20024 in main.cf; here, we are specifying that any mail coming in via localhost on port 20025 should not go through a content filter; in other words, it should be delivered locally as it normally would. This will be discussed in greater detail in the Amavis section.

The second line creates a transport for Mailman which we will use later. It specifies that mail going to this transport will result in forking the application /home/mail/mailman/bin/postfix-to-mailman.py with arguments ${nexthop} and ${user}, which are built-in Postfix variables.

header_checks, body_checks

The format of these files is simple, with one Perl-compatible regular expression per line along with an action. The most common action you would want would be REJECT, which drops the mail and sends and SMTP error back to the originating host, indicating that the content was rejected.

An example is as follows: if you had /badtext/ REJECT on one line in body_checks, than any variation of the word badtext in the body of the email would result in the email being dropped; the expressions are applied case-insensitive by default.

transport

This file is relatively simple in our implementation, as we only have one transport mechanism we need to worry about: e-mail lists. As such, there is only one line in the file:

 lists.tjhsst.edu        mailman:

This simply specifies that any mail being sent to the lists.tjhsst.edu domain should be send via the mailman transport, which we defined in master.cf to be sent to the Postfix-Mailman handler command.

Note: After changing this file, one must run postmap on the filename to create a new hash database.

virtual_maps

This file specifies where mail for virtual domains we host should be delivered. Its syntax is relatively straightforward.

 tjpartnershipfund.org   DOMAIN
 info@tjpartnershipfund.org joan.ozdogan@fcps.edu

The first line indicates that we want to allow virtual hosting for the domain tjpartnershipfund.org. The second line lists an address in the virtual domain, followed by an address the mail should be delivered to. If the domain is omitted in the to address, it will be assumed to be delivered locally.

It should be noted that mail for local users will not be received by this domain unless they are explicitly listed in this file or the domain is added to the mydestination directive in main.cf. Note that in the latter case, this file will become irrelevant and unused.

Note: After changing this file, one must run postmap on the filename to create a new hash database.

virtual_mailbox_maps

This file specifies the maildir of each local user who is to be delivered mail. The syntax is simple.

  adeason /home/mail/users/adeason/Maildir/

This simply says that for the user 'adeason', deliver mail to the maildir in /home/mail/users/adeason/Maildir/. There must be an entry for every local user in the system in this file, under our setup. If a new local user is added to the system, then they must be added to this file.

Note: After changing this file, one must run postmap on the filename to create a new hash database.

Running

Running Postfix is relatively easy, with most utility coming from postfix <command>. The following commands are recognized:

  • check

Checks the configuration files without starting or stopping Postfix. Useful to check if you've made mistakes on a production machine.

  • start

Starts the Postfix mail system.

  • stop

Stops the Postfix mail system in a nice way. Processes stop naturally.

  • abort

Stops the Postfix mail system abruptly. This one should be avoided if possible.

  • flush

Flushes the mail queue, forcing Postfix to try delivering all mail that has accumulated in the queue.

  • reload

Causes Postfix to reload its configuration files and restart processes nicely.

Typically the daemon is started and stopped with /etc/init.d/postfix start and /etc/init.d/postfix stop.

Other commands of interest include mailq, which lists all mail currently stored in the queue. This can be flushed using postfix flush, as indicated above.

Postfix sends all its logging information to syslog, which usually appends it to the file /var/log/mail.log. Examining this file is often key to troubleshooting issues with the mail system.

Amavis

Amavisd-new is used to scan all incoming and outgoing mail for virii and spam. It serves as an interface between the MTA, Postfix, and the various scanning utilities: Clam Antivirus for virus protection, and SpamAssassin for spam detection.

Clam Antivirus

The ClamAV package is being used to scan for virii in email. It functions in two modes: first as a daemon that runs all the time and scans data fed to a socket, and also as a standalone command line utility, clamscan. In most circumstances we will be using the former due to its increased performance, falling to the latter only if the daemon fails.

The virus definitions are updated about five times a day off the Internet through the freshclam daemon. In this way, new virii in the wild will be detected early and stopped from entering the TJHSST mail system.

SpamAssassin

SpamAssassin is used to scan all mail coming into the lab for its likelihood of spam, and all messages are given points for certain key phrases that are common in spam. With our configuration, very little user-level configuration is available.

Configuration

Configuration of Amavis is relatively simple, thus its configuration file will not be repeated here. Virtually all configuration is found in the file /etc/amavis/amavisd.conf. A few important things to note involve our specific implementation of the system.

Recall in the Postfix configuration section that we defined the default transport for all mail to be localhost:20024; Amavis is configured to listen on this port. Effectively, all mail sent into the Postfix mail system will be sent to Amavis on port 20024, which will scan the mail and return it to Postfix on port 20025.

Running

Very little needs to be done to Amavis to get it running besides making sure that all necessary daemons are running, specifically clamd, freshclam, and amavisd-new. When Amavis receives a new mail on its socket, it will pass it through the virus scanner and then either discard the email or approve it and add an email header that indicates it was passed. It will then run the email through SpamAssassin and add any necessary headers indicating spam level. No spam will automatically be discarded; it is up to the user to filter as s/he wishes.

Logging information is sent to syslog, which typically appends it to /var/log/mail.log. In this will be indicated whether the emails passed or were infected, and in the case of the latter, which virus/ii were involved. Graphs of this data are also availble online at http://mail.tjhsst.edu/amavis-stats/. These graphs are produced every five minutes by the amavis-stats package.

Mailman

E-mail distribution lists are provided on the domain lists.tjhsst.edu by Mailman, a highly popular software package. Most list administration takes place through an easy online interface at https://lists.tjhsst.edu/, though many functions are also available through email, such as list subscription and some message handling.

Configuration

Configuration for this software package is minimal; the main parameters that must be specified are the urls and domains to be used. Also, some configuration of Postfix is necessary, as outlined in previous scripts. Integration with Postfix is achieved through the postfix-to-mailman.py script, which provides the necessary functionality.

Running

Postfix pipes any email with a destination of lists.tjhsst.edu to this script, which runs the necessary Mailman commands to deliver the email. Mailman, of course, ends up sending mails back to Postfix, destined for the actual recipients. Postfix will then deliver these mails locally or forward them to the appropriate mail server.

A few daemons must be running for Mailman to function properly. The first is qrunner, which processes any mail in Mailman's queue. If this is not running, e-mails to lists will be added to the queue but will never be delivered to recipients. The other necessary daemon is apache, which provides the necessary online interface for list administration.

Courier IMAP

The Courier mail system provides IMAP support for the school. Users can either directly access their email through IMAP-SSL or may access it over the web using SquirrelMail.

Cyrus SASL

Cyrus SASL is a simple authentication layer that is used by Postfix to authenicate users using SMTP.

It works like this: after an off-site user has initialed a TLS connection over SMTP, he then issues the SMTP AUTH LOGIN command, along with his credentials. Postfix takes this and passes it to the saslauthd daemon, which then tries to authenticate the user with the configured mechanism, PAM. If the supplied credentials are valid, PAM returns success, which is then passed by saslauthd back to the Postfix SMTP daemon. The user is now authorized to relay mail.

Configuration is straightforward, the only files needing attention being /etc/default/saslauthd and /etc/postfix/sasl/smtpd.conf. Saslauthd is configured to use PAM, similar to how IMAP was configured.

Caution! The SMTP daemon communicates with saslauthd through a Unix socket, but also lives in a chroot. And you've probably already guessed it: the socket, by default, is not within this chroot jail. Presently, the directory in which the socket lives is bind-mounted to the chrooted directory, though it could probably be fixed in a cleaner way.

The SMTP daemon runs as the postfix user, which by default does not have permissions to access the saslauthd socket. It must be added to the appropriate group for it to work properly.

Quotas

Currently, ext3 quotas are in place to prevent users from using too much space. The amount of quota space that someone gets is obtained from the group information in LDAP from the Windows AD server. If someone is in the group HardQuota7500, it represents that thet get 7500 KiB of quota space. Their 'Soft Limit' (the limit at which they have a grace period to delete emails before it is enforced) is always 2500 KiB below the hard limit. This is set every day by the ldap_mail script using the edquota command. The quota usage infomation is updated every day by the quotacheck program, run from cron.

Users are also notified if they are above their soft quota by a script written by Andrew Deason called csl_warnquota. It searches output from repquota to find users that are over their quota, and emails a warning to them with information if they can still receive email.

Group Emails

TJ has implemented a custom script to emulate the group email functionality provided by the previous Pegasus mail system. An example of this was the #seniors address - mail sent to this address would be delivered to all seniors. The script groupmail provides this in our new Postfix system.

The script is called through /etc/aliases. Each address pipes to the script with an argument of a Posix group name. All students are categorized into groups in ADS, and Windbind converts these into corresponding Posix groups.

When the script is called, it first checks to see if the user is authorized to be sending mail to this address. Presently, only a simple, weak check of the from address is used; if the user's address is listed on a line in /home/mail/groups/access/GROUPNAME, the email will be accepted and delivered.

If the message is accepted, it is first saved as a new file in /home/mail/groups/messages. The script then checks for membership of the groupname supplied as an argument and creates a symbolic link in each user's Maildir to this message. This has two benefits: first, disk space is conserved, as only one copy of the message is saved. Second, if the system is abused, the message in /home/mail/groups/messages can be removed and all the symlinks will become safely invalid.