Difference between revisions of "Email"

From Livedoc - The Documentation Repository
Jump to: navigation, search
m (Robot: Changing Category:Obsolete)
(Update for most recent mail system)
Line 1: Line 1:
{{Converted from book}}
+
=History=
 
+
The current '''TJ email system''' is an idea originally designed by Aman Gupta (class of '04). The original system was implemented by Andrew Deason (class of '06) with the help of several staff members, including Richard Washer and Susan Beasley. A second mail system, based on Zimbra, was configured by [[User:Lee Burton|Lee Burton]]. The latest mail system was designed by [[User:Brandon Vargo|Brandon Vargo]].
=Overview=
 
The current '''TJ email system''' is an idea originally designed by Aman Gupta (class of '04), and implemented by Andrew Deason (class of '06) with the help of several staff members, including Richard Washer and Susan Beasley. The actual mail is stored on a [[Debian]] GNU/Linux server, and is also accessed there, but all of the user information and authentication is done by a Windows Active Directory server.
 
  
 
Previously, the CSL only hosted mail for the CSL itself; mail that was only accesible to people with Systems Lab accounts. Today, lab-only mail does not exist anymore, and everyone gets their email off of the same system, all with @tjhsst.edu addresses.
 
Previously, the CSL only hosted mail for the CSL itself; mail that was only accesible to people with Systems Lab accounts. Today, lab-only mail does not exist anymore, and everyone gets their email off of the same system, all with @tjhsst.edu addresses.
  
=Samba=
+
=Overview=
 
+
The current mail system consists of two primary email servers as well as a mailman server. The two primary email servers, casey and smith are named after the founders of UPS and FedEx, Jim Casey and Frederick Smith, respectively. The servers are virtually identical; only the hostname/mailname and network information is different. Each runs postfix, dovecot, and amavisd-new. The mailman server, lists, runs a standard mailman installation.
As we now support mail for all of TJHSST, we found it necessary to authenticate against the LAN, which is slowly moving away from Novell and instead to Windows 2003 Active Directory Services. We opted to authenticate using ADS, and found that Winbind, a part of Samba, provided the authentication support we needed. Both NSS and PAM modules were provided allowing full interoperability with our Linux system.
 
 
 
==Configuration==
 
 
 
There are several configuration files used to facilitate integrating users from the Windows 2003 ADS into Linux, spanning different systems.
 
 
 
===smb.conf===
 
 
 
The first file we'll deal with is Samba's, located at /etc/samba/smb.conf. Note that the configuration for Winbind as well as the overall configuration for Samba itself is located within this file. Here are some of the more important directives.
 
 
 
  workgroup = LOCAL
 
  realm = LOCAL.TJHSST.EDU
 
  
This just declares which Windows workgroup we are a part of, and which Kereros realm. Note that the kerberos realm is not necessarily the name of any actual machine, but is just a string to represent a realm. (Even though in our case, local.tjhsst.edu is a valid DNS entry, it doesn't have to be.)
+
When mail is first received by the mail system, it is processed by postfix. If basic spam tests pass, the message is passed to amavisd-new for additional spam processing. If the message passes these tests, it is placed into another internal postfix queue. In general, mail will only be tagged instead of being discarded. At this point, postfix is configured to pass the message to dovecot for delivery. Both servers share a common NFS filestore. Dovecot is configured to handle locking.
  
  security = ADS
+
=Authentication/Authorization=
 +
Authentication is handled by the kerberos PAM module, similar to most other systems in the lab. Dovecot is configured to user PAM.
  
This turns on security by Active Directory Service from the Windows system.
+
Instead of using the Lab's LDAP tree for user information, account information is stored locally on each server in the form of standard UNIX accounts. A mail management script, described below, synchronizes the AD user information with the local UNIX accounts. The UID for the local UNIX account is a hash of the username. This allows each server to run the mail synchronization script independently and not worry about UID conflicts on the shared NFS storage.
 
 
  password server = local.tjhsst.edu
 
  encrypt passwords = true
 
 
 
These lines indicate which password server to use (a round-robin address in this case), and to to enable passwords. Obviously enabling passwords is a good idea, and even more so with Microsoft servers, since many of their systems no longer support plaintext authentication (according to the smb.conf manpage).
 
 
 
  obey pam restrictions = yes
 
 
 
We will be using PAM as part of our configuration (see below for PAM config), so we want to enable this.
 
 
 
  invalid users = root
 
 
 
Obviously we don't want root authenticating via Samba! If, by chance, there is a user in the ADS called 'root', then the password for that ADS user would be checked, and someone might be able to gain the 'root' username, by a password that is not the local root password on the machine. That is clearly very bad, so here we disable hat from happening.
 
 
 
  unix password sync = no
 
 
 
This option would try to sync the Samba passwords of users with the local unix passwords. But since the users we are trying to integrate are not local unix users, we don't want this.
 
 
 
  idmap uid = 10000-20000
 
  idmap gid = 10000-20000
 
 
 
These lines set the range that Winbind will use for the UIDs and GIDs of the various users and groups retrieved from Samba. Make sure that these ranges are sure not to conflict with any other UID/GIDs on the local machine!
 
 
 
  template shell = /usr/bin/pine
 
  template homedir = /home/mail/users/%U
 
 
 
These set the environment for the user. The shell is set to pine, so if users login via SSH, they will immediately be dropped into pine to check their mail, without given shell access. (Write permission is also denied to each user's .pinerc, so they can't change the configuration to run arbitrary commands.) The home directory is merely the user's home directory, mostly just where the mail is stored.
 
 
 
  winbind use default domain = yes
 
 
 
This allows users to login without specifying their Windows domain in their username, which would make things a real pain when logging in. With this option turned on, a user can just log in as 'adeason', and they will be given the default domain. (Default option is to make usernames have their domain prepended to them before a separator, which is '+' by default.)
 
 
 
===krb5.conf===
 
 
 
Even though krb5.conf is a Kerberos configuration file, and not really part of Samba, it is necessary for Winbind to function, so it will be described here briefly. It is located at /etc/krb5.conf.
 
 
 
  default_realm = LOCAL.TJHSST.EDU
 
 
 
This simply sets the default realm for authenticating against.
 
 
 
  [realms]
 
      LOCAL.TJHSST.EDU = {
 
          kdc = LOCAL.TJHSST.EDU
 
          admin_server = LOCAL.TJHSST.EDU
 
      }
 
 
 
Before, it was mentioned that the Kerberos realm did not need to be an actual machine. Here is where we specify which machines each realm uses. For our purposes, the Kerberos servers happen to have the same name as the realm, but such is not always true.
 
 
 
===PAM===
 
 
 
Winbind itself is used as two things in our system: a PAM module called pam_winbind.so, and an NSS mechanism (in /etc/nsswitch.conf as 'winbind'). Explained briefly here is how to use the PAM module.
 
 
 
For example, the /etc/pam.d/ssh file:
 
 
 
  auth            required        pam_nologin.so
 
  auth            required        pam_env.so
 
  @include common-auth
 
  auth            required        pam_winbind.so use_first_pass
 
 
 
  @include common-account
 
  account        required        pam_winbind.so
 
 
 
  @include common-session
 
  session        required        pam_limits.so
 
  session        required        pam_winbind.so
 
  session        optional        pam_lastlog.so
 
  session        optional        pam_motd.so
 
  session        optional        pam_mail.so standard noenv
 
 
 
  @include common-password
 
 
 
The important lines are the ones referencing pam_winbind.so, but the others are included for reference. The use_first_pass option in the auth section is given so users don't get two prompts for passwords (this is similar to a kerberos authentication scheme). Since the password the user typed in was already passed to pam_unix.so (which isn't directly in this file, but is in the '@include'ed file common-auth), the module pam_winbind.so also needs the password. Normally it would prompt the user for a password, but that option uses the password already read in from the user.
 
 
 
==Running==
 
 
 
There are three daemons for the Samba/Winbind system: nmbd, smbd, and winbindd. The first two are controlled by /etc/init.d/samba, and winbindd is controlled by /etc/init.d/winbind. They both understand the standard init script options of start, stop, restart, and force-reload, and the samba init script also understands the command 'reload'.
 
  
 
=Postfix=
 
=Postfix=
At the heart of the new mail system is the excellent mail server Postfix. It was chosen over alternatives for its robustness, ease of configuration, and ease of use.
+
At the heart of the mail system is the excellent mail server Postfix. It was chosen over alternatives for its robustness, ease of configuration, and ease of use.
  
 
==Configuration==
 
==Configuration==
Line 110: Line 21:
  
 
===main.cf===
 
===main.cf===
This configuration file tends to be the longest, as it houses nearly all of the options one might be concerned with. For simplicity's sake, we will outline here only the most important directives.
+
This configuration file tends to be the longest, as it houses nearly all of the options one might be concerned with. As this file is commented, there is no need to duplicate its function here.
 
 
  myhostname = mail.tjhsst.edu
 
 
 
Used to set the hostname of the mailserver. This is used in many later configuration parameters, and is what is sent by the mailserver when it is talking to other servers.
 
 
 
  mydomain = tjhsst.edu
 
 
 
The domain name this mailserver is hosting.
 
 
 
  myorigin = $mydomain
 
 
 
Specifies where mail that does not have a domain already is assumed to be originating from.
 
 
 
  inet_interfaces = all
 
 
 
Determines which interfaces Postfix will listen on. We want it to listen on all interfaces.
 
 
 
  mydestination = $myhostname, localhost.$mydomain, $mydomain, *.$mydomain, lan.$mydomain, mail-bkup.$mydomain
 
 
 
One of the more important parameters, this one specifies which domains to consider as local mail. The * doesn't always seem to work, so we often must add domains manually.
 
 
 
{{Warning|Do not specify virtual hosts here, or they will be consided local domains and will not be processed through the virtual maps.}}
 
 
 
  local_recipient_maps = $alias_maps $virtual_mailbox_maps
 
 
 
Determines how postfix determines which users are local and can be delivered to. Note that this is different from the default.
 
 
 
  mynetworks = 198.38.16.0/20, 127.0.0.0/8
 
 
 
This parameter controls who may relay mail through us. We have decided to let anybody inside the building have access.
 
 
 
  virtual_maps = hash:/etc/postfix/virtual_maps
 
 
 
This points to a file that contains mappings for virtual domains. The format of this file will be discussed later.
 
 
 
  virtual_mailbox_base = /home/mail/user/
 
  virtual_mailbox_maps = hash:/etc/postfix/virtual_mailbox_maps
 
 
 
These two paramters control where mail is delivered. Postfix is instructed to look at the virtual_mailbox_maps file for information on where mailboxes are stored per user.
 
 
 
  home_mailbox = Maildir/
 
 
 
This indicates that we have decided to use the Qmail-format Maildir/ form of storing one message per file in directories inside each user's home directory.
 
 
 
  header_checks = pcre:/etc/postfix/header_checks
 
  body_checks = pcre:/etc/postfix/body_checks
 
 
 
These two point to files that contain Perl-compatible regular expressions that are matched against incoming e-mails. Those that match are dropped.
 
 
 
  content_filter = smtp:localhost:20024
 
 
 
This parameter specifies that all email should be sent via smtp to localhost on port 20024 before being delivered. This is used to scan all email for viruses, and will be discussed in more detail in a later section.
 
 
 
  transport_maps = hash:/etc/postfix/transport
 
  mailman_destination_recipient_limit = 1
 
 
 
These two parameters allow operation with Mailman, which is used to run self-managed distribution lists.
 
 
 
  broken_sasl_auth_clients = yes
 
  smtpd_sasl_auth_enable = yes
 
  smtpd_sasl_security_options = noanonymous
 
  smtpd_sasl_local_domain =
 
  smtpd_recipient_restrictions = permit_sasl_authenticated, permit_mynetworks, reject_unauth_destination
 
 
 
These parameters control the use of SASL authentication, which is used for SMTP authentication so that users can relay through our mailserver outside the building. This will be discussed in more detail later.
 
 
 
  smtp_use_tls = yes
 
  smtp_tls_note_starttls_offer = yes
 
  smtp_tls_CAfile = /etc/postfix/ssl/tjhsst.ca.pem
 
  smtp_tls_key_file = /etc/postfix/ssl/mail.pem
 
  smtp_tls_cert_file = /etc/postfix/ssl/mail.pem
 
  smtp_tls_loglevel = 1
 
  smtpd_use_tls = yes
 
  smtpd_tls_key_file = /etc/postfix/ssl/mail.pem
 
  smtpd_tls_cert_file = /etc/postfix/ssl/mail.pem
 
  smtpd_tls_CAfile = /etc/postfix/ssl/tjhsst.ca.pem
 
  smtpd_tls_loglevel = 1
 
  smtpd_tls_received_header = yes
 
  smtpd_tls_session_cache_timeout = 3600s
 
  tls_random_source = dev:/dev/urandom
 
  smtpd_sasl_tls_security_options = $smtpd_sasl_security_options
 
  smtpd_tls_auth_only = yes
 
 
 
These parameters control TLS, or Transport Layer Security, also known as SSL. They provide encryption over SMTP channels if the remote server or client supports it. It should be noted that the last one specifies that SMTP authentication, discussed in the previous point, is only available if TLS is enabled.
 
 
 
  smtpd_helo_required = yes
 
  strict_rfc821_envelopes = yes
 
  smtpd_client_restrictions =
 
      reject_rbl_client sbl-xbl.spamhaus.org,
 
      reject_rbl_client relays.ordb.org,
 
      reject_rbl_client list.dsbl.org
 
 
 
These control anti-spam techniques common on the Internet.  
 
  
 
===master.cf===
 
===master.cf===
This configuration file describes how the Postfix daemon should run and determines which ports it should be listening on for connections and how to handle those connections. The default format of this file is perfectly fine, with only two additions necessary in our setup.
+
This configuration file describes how the Postfix daemon should run and determines which ports it should be listening on for connections and how to handle those connections. The default format of this file is perfectly fine, with only two additions necessary in our setup. The first is the amavis line, which is configured for spam filtering. The second is the line beginning with 127.0.0.1. This second line accepts mail on an alternate port and delivers the mail directly, bypassing spam filtering. This is the port to which amavisd-new delivers mail. If amavisd-new delivered mail to the standard postfix port, all messages would be put into an endless loop of spam filtering.
 
 
  localhost:20025  inet  n      -      -      -      -      smtpd -o content_filter
 
  mailman unix    -      n      n      -      -      pipe flags=FR user=list:list argv=/home/mail/mailman/bin/postfix-to-mailman.py ${nexthop} ${user}
 
 
 
The first line is related to our use of the content_filter = smtp:localhost:20024 in main.cf; here, we are specifying that any mail coming in via localhost on port 20025 should not go through a content filter; in other words, it should be delivered locally as it normally would. This will be discussed in greater detail in the Amavis section.
 
 
 
The second line creates a transport for Mailman which we will use later. It specifies that mail going to this transport will result in forking the application /home/mail/mailman/bin/postfix-to-mailman.py with arguments ${nexthop} and ${user}, which are built-in Postfix variables.
 
 
 
===header_checks, body_checks===
 
 
 
The format of these files is simple, with one Perl-compatible regular expression per line along with an action. The most common action you would want would be REJECT, which drops the mail and sends and SMTP error back to the originating host, indicating that the content was rejected.
 
 
 
An example is as follows: if you had /badtext/ REJECT on one line in body_checks, than any variation of the word badtext in the body of the email would result in the email being dropped; the expressions are applied case-insensitive by default.
 
 
 
===transport===
 
 
 
This file is relatively simple in our implementation, as we only have one transport mechanism we need to worry about: e-mail lists. As such, there is only one line in the file:
 
 
 
  lists.tjhsst.edu        mailman:
 
 
 
This simply specifies that any mail being sent to the lists.tjhsst.edu domain should be send via the mailman transport, which we defined in master.cf to be sent to the Postfix-Mailman handler command.
 
 
 
'''Note:'''
 
After changing this file, one must run '''postmap''' on the filename to create a new hash database.
 
 
 
===virtual_maps===
 
This file specifies where mail for virtual domains we host should be delivered. Its syntax is relatively straightforward.
 
 
 
  tjpartnershipfund.org  DOMAIN
 
  info@tjpartnershipfund.org joan.ozdogan@fcps.edu
 
 
 
The first line indicates that we want to allow virtual hosting for the domain tjpartnershipfund.org. The second line lists an address in the virtual domain, followed by an address the mail should be delivered to. If the domain is omitted in the to address, it will be assumed to be delivered locally.
 
 
 
It should be noted that mail for local users will not be received by this domain unless they are explicitly listed in this file or the domain is added to the mydestination directive in main.cf. Note that in the latter case, this file will become irrelevant and unused.
 
 
 
'''Note:'''
 
After changing this file, one must run '''postmap''' on the filename to create a new hash database.
 
 
 
===virtual_mailbox_maps===
 
This file specifies the maildir of each local user who is to be delivered mail. The syntax is simple.
 
 
 
  adeason /home/mail/users/adeason/Maildir/
 
 
 
This simply says that for the user 'adeason', deliver mail to the maildir in /home/mail/users/adeason/Maildir/. There must be an entry for every local user in the system in this file, under our setup. If a new local user is added to the system, then they must be added to this file.
 
 
 
'''Note:'''
 
After changing this file, one must run '''postmap''' on the filename to create a new hash database.
 
  
 
==Running==
 
==Running==
Line 287: Line 58:
 
==SpamAssassin==
 
==SpamAssassin==
  
SpamAssassin is used to scan all mail coming into the lab for its likelihood of spam, and all messages are given points for certain key phrases that are common in spam. With our configuration, very little user-level configuration is available.
+
SpamAssassin is used to scan all mail coming into the lab for its likelihood of spam, and all messages are given points for certain key phrases that are common in spam. With our configuration, very little user-level configuration is available. Additional rule files have been added under /etc/spamassassin/. SpamAssassin can be run as a daemon, but the mail system is configured to use the spamassassin tool directly.
  
===Configuration===
+
==Configuration==
Configuration of Amavis is relatively simple, thus its configuration file will not be repeated here. Virtually all configuration is found in the file /etc/amavis/amavisd.conf. A few important things to note involve our specific implementation of the system.
+
Configuration of Amavis is relatively simple and commented well, thus its configuration file will not be repeated here. Virtually all configuration is found in the files in /etc/amavis/conf.d/. Files are processed in order.
  
Recall in the Postfix configuration section that we defined the default transport for all mail to be localhost:20024; Amavis is configured to listen on this port. Effectively, all mail sent into the Postfix mail system will be sent to Amavis on port 20024, which will scan the mail and return it to Postfix on port 20025.
+
==Running==
 
+
Very little needs to be done to Amavis to get it running besides making sure that all necessary daemons are running, specifically clamd, freshclam, and amavisd-new. When Amavis receives a new mail on its socket, it will pass it through the virus scanner and then either discard the email or approve it and add an email header that indicates it was passed. It will then run the email through SpamAssassin and add any necessary headers indicating spam level. No spam will automatically be discarded; it is up to the user to filter as s/he wishes. Dovecot will automatically put spam in a spam folder.
===Running===
 
Very little needs to be done to Amavis to get it running besides making sure that all necessary daemons are running, specifically clamd, freshclam, and amavisd-new. When Amavis receives a new mail on its socket, it will pass it through the virus scanner and then either discard the email or approve it and add an email header that indicates it was passed. It will then run the email through SpamAssassin and add any necessary headers indicating spam level. No spam will automatically be discarded; it is up to the user to filter as s/he wishes.
 
  
Logging information is sent to syslog, which typically appends it to /var/log/mail.log. In this will be indicated whether the emails passed or were infected, and in the case of the latter, which virus/ii were involved. Graphs of this data are also availble online at [http://mail.tjhsst.edu/amavis-stats/ http://mail.tjhsst.edu/amavis-stats/]. These graphs are produced every five minutes by the amavis-stats package.
+
Logging information is sent to syslog, which typically appends it to /var/log/mail.log. In this will be indicated whether the emails passed or were infected, and in the case of the latter, which virus/ii were involved.
  
 
=Mailman=
 
=Mailman=
Line 303: Line 72:
  
 
==Configuration==
 
==Configuration==
Configuration for this software package is minimal; the main parameters that must be specified are the urls and domains to be used. Also, some configuration of Postfix is necessary, as outlined in previous scripts. Integration with Postfix is achieved through the postfix-to-mailman.py script, which provides the necessary functionality.
+
Configuration for this software package is minimal; the main parameters that must be specified are the urls and domains to be used. A modified version of the main postfix+amavisd-new configuration files are used. Integration with Postfix is achieved through the postfix-to-mailman.py script, which provides the necessary functionality.
  
 
==Running==
 
==Running==
Postfix pipes any email with a destination of lists.tjhsst.edu to this script, which runs the necessary Mailman commands to deliver the email. Mailman, of course, ends up sending mails back to Postfix, destined for the actual recipients. Postfix will then deliver these mails locally or forward them to the appropriate mail server.
+
Postfix pipes any email with a destination of lists.tjhsst.edu to this script, which runs the necessary Mailman commands to deliver the email. Mailman, of course, ends up sending mails back to Postfix, destined for the actual recipients. Postfix will then deliver these mails to the mail mail systems or forward them to the appropriate mail server.
 
 
A few daemons must be running for Mailman to function properly. The first is '''qrunner''', which processes any mail in Mailman's queue. If this is not running, e-mails to lists will be added to the queue but will never be delivered to recipients. The other necessary daemon is '''apache''', which provides the necessary online interface for list administration.
 
 
 
=Courier IMAP=
 
The Courier mail system provides IMAP support for the school. Users can either directly access their email through IMAP-SSL or may access it over the web using SquirrelMail.
 
 
 
=Cyrus SASL=
 
Cyrus SASL is a simple authentication layer that is used by Postfix to authenicate users using SMTP.
 
 
 
It works like this: after an off-site user has initialed a TLS connection over SMTP, he then issues the SMTP AUTH LOGIN command, along with his credentials. Postfix takes this and passes it to the saslauthd daemon, which then tries to authenticate the user with the configured mechanism, PAM. If the supplied credentials are valid, PAM returns success, which is then passed by saslauthd back to the Postfix SMTP daemon. The user is now authorized to relay mail.
 
 
 
Configuration is straightforward, the only files needing attention being /etc/default/saslauthd and /etc/postfix/sasl/smtpd.conf. Saslauthd is configured to use PAM, similar to how IMAP was configured.
 
 
 
{{Warning|Caution! The SMTP daemon communicates with saslauthd through a Unix socket, but also lives in a chroot. And you've probably already guessed it: the socket, by default, is not within this chroot jail. Presently, the directory in which the socket lives is bind-mounted to the chrooted directory, though it could probably be fixed in a cleaner way.}}
 
 
 
The SMTP daemon runs as the postfix user, which by default does not have permissions to access the saslauthd socket. It must be added to the appropriate group for it to work properly.
 
 
 
=Quotas=
 
Currently, ext3 quotas are in place to prevent users from using too much space. The amount of quota space that someone gets is obtained from the group information in LDAP from the Windows AD server. If someone is in the group HardQuota7500, it represents that thet get 7500 KiB of quota space. Their 'Soft Limit' (the limit at which they have a grace period to delete emails before it is enforced) is always 2500 KiB below the hard limit. This is set every day by the [[ldap_mail]] script using the '''edquota''' command. The quota usage infomation is updated every day by the '''quotacheck''' program, run from cron.
 
 
 
Users are also notified if they are above their soft quota by a script written by Andrew Deason called '''csl_warnquota'''. It searches output from '''repquota''' to find users that are over their quota, and emails a warning to them with information if they can still receive email.
 
 
 
=whitelister=
 
In order to utilize [http://en.wikipedia.org/wiki/Sender_Policy_Framework SPF], we make use of a postfix SMTP proxy daemon called '''whitelister''', which we altered from version 0.6 (the source is in <code>/usr/local/src/</code> on [[chinstrap]]). The diff between the two versions looks like this:
 
 
 
  diff -u whitelister-0.6/rules.ml whitelister-0.6.new/rules.ml
 
  --- whitelister-0.6/rules.ml 2005-08-27 07:00:38.000000000 -0400
 
  +++ whitelister-0.6.new/rules.ml 2005-11-12 17:58:42.000000000 -0500
 
  @@ -87,11 +87,11 @@
 
        if dorej then raise (Reject (s ^ Policy.spf_explain pcy)) else raise (Dirty s)
 
      in try
 
        match spf_query (sender pcy) (client_address pcy) (helo_name pcy) with
 
  -        | SPF_pass    -> ()
 
  -        | SPF_softerr -> fail "SPF soft error"
 
  -        | SPF_harderr -> fail "SPF hard error"
 
  -        | SPF_none    -> if mode != Spf_normal then raise (Dirty "no SPF record found")
 
  -        | SPF_neutral -> if mode = Spf_paranoid then raise (Dirty "SPF neutral")
 
  +        | SPF_pass    -> raise (Dirty "pass")
 
  +        | SPF_softerr -> fail "softfail"
 
  +        | SPF_harderr -> fail "fail"
 
  +        | SPF_none    -> raise (Dirty "none")
 
  +        | SPF_neutral -> raise (Dirty "neutral")
 
      with
 
  -      | Spf.Error -> raise (Dirty "SPF Internal error")
 
  +      | Spf.Error -> raise (Dirty "error")
 
 
 
  Common subdirectories: whitelister-0.6/tpl and whitelister-0.6.new/tpl
 
  diff -u whitelister-0.6/whitelister.ml whitelister-0.6.new/whitelister.ml
 
  --- whitelister-0.6/whitelister.ml 2005-08-27 17:31:25.000000000 -0400
 
  +++ whitelister-0.6.new/whitelister.ml 2005-11-12 18:01:34.000000000 -0500
 
  @@ -61,18 +61,18 @@
 
        let pcy = Policy.read (Unix.in_channel_of_descr s) in
 
        let ans = (
 
          try
 
  -          Rules.check_rbl cfg.rbl pcy;
 
  +(*          Rules.check_rbl cfg.rbl pcy;
 
            Rules.check_rhbl Rules.Helo  cfg.rhbl_helo  pcy;
 
            Rules.check_rhbl Rules.Sender cfg.rhbl_sender pcy;
 
            Rules.check_rhbl Rules.Rcpt  cfg.rhbl_rcpt  pcy;
 
  -          Rules.check_rhbl Rules.Client cfg.rhbl_client pcy;
 
  +          Rules.check_rhbl Rules.Client cfg.rhbl_client pcy;*)
 
            Rules.check_spf cfg.spf cfg.spfrej pcy;
 
            if cfg.verb > 0 then log_event "Clean" "OK" pcy;
 
            "OK"
 
          with
 
            | Rules.Dirty  s ->
 
                log_event "Dirty" (Printf.sprintf "DUNNO (%s)" s) pcy;
 
  -              "DUNNO"
 
  +              "PREPEND Received-SPF: " ^ s
 
            | Rules.Reject s ->
 
                let s' = "REJECT "^s in
 
                log_event "Reject" s' pcy;
 
  
All it does is insert a "Received-SPF:" header according to what SPF results whitelister found, and lets the message through. Spamassassin then uses the Received-SPF header to add or subtract from the message's spam score, depending on what is in the header.
+
A few daemons must be running for Mailman to function properly. The first is '''qrunner''', which processes any mail in Mailman's queue. If this is not running, e-mails to lists will be added to the queue but will never be delivered to recipients. This is started by the mailman init script. The other necessary daemon is '''apache''', which provides the necessary online interface for list administration.
  
===Configuration===
+
=Dovecot=
{{Warning|Do not use the RBLs/RHBLs configuration directives! This patched version of whitelister does not use them, so fiddling with the directives may break it. Make sure all RBL/RHBL configuration directives for whitelister are commented out.}}
+
Dovecot proves IMAPs, POPs, and sieve support, in addition to being the local mail delivery agent. Dovecot receives mail from postfix and is responsible for delivering it to the user's mail directory. If dovecot is not running, postfix will queue mail.
  
Whitelister uses a single configuration file: <code>/etc/whitelister.conf</code>. The directives used are:
+
Dovecot also supports quotas. If a user is over their soft quota, dovecot will give a warning but still store the message. If a user exceeds their hard quota, the mail message will be rejected.
  sock: 127.0.0.1:10000
 
Listen on the loopback address on port 10000.
 
  spf: 1
 
This tells whitelister to indicate that the SPF check "failed" only where an IP mismatch occurs. A value of 0 here turns off SPF, 2 fails on an empty record, 3 fails on a neutral record.
 
  spfrej: 0
 
Determines whether or not to issue 'PREPEND' or 'REJECT' actions on the message to postfix when an SPF failure occurs. Since we are only using whitelister for inserting Received-SPF headers, we don't want it to reject anything, so this is set to 0. Setting it to 1 would cause it to reject non-SPF matched mail.
 
  verb: 0
 
This disables verbose logging, and only logs 'failed' requests. It probably logs all requests anyway, since all messages are marked as 'failed' in order to prepend the Received-SPF: header.
 
  
=Group Emails=
+
=Account Management=
TJ has implemented a custom script to emulate the group email functionality provided by the previous Pegasus mail system. An example of this was the #seniors address - mail sent to this address would be delivered to all seniors. The script groupmail provides this in our new Postfix system.
+
See [[Account provisioning]] for information on running the synchronization script.
  
The script is called through /etc/aliases. Each address pipes to the script with an argument of a Posix group name. All students are categorized into groups in ADS, and Windbind converts these into corresponding Posix groups.
+
The '''mail.py''' script is a python script responsible for syncing the local user information with the Active Directory user database. This script should be run on all mail servers, even though there is a common file storage.
  
When the script is called, it first checks to see if the user is authorized to be sending mail to this address. Presently, only a simple, weak check of the from address is used; if the user's address is listed on a line in /home/mail/groups/access/GROUPNAME, the email will be accepted and delivered.
+
Users are stored in groups in AD according to whether a mail account is required and the desired quota. This information is compared to a local user database, '''/etc/mailmanage/users''', a CSV file that lists all mail users past and present. Users in AD that are not in the local database will have accounts created. Users in the local database but not in AD will have their account disabled unless an override is specified. Currently it is necessary to edit this file directly to override a quota or keep an account enabled.
  
If the message is accepted, it is first saved as a new file in /home/mail/groups/messages. The script then checks for membership of the groupname supplied as an argument and creates a symbolic link in each user's Maildir to this message. This has two benefits: first, disk space is conserved, as only one copy of the message is saved. Second, if the system is abused, the message in /home/mail/groups/messages can be removed and all the symlinks will become safely invalid.
+
CSV fields:
 +
#The username
 +
#The quota, before the multiplication value is taken into effect
 +
#Source of the user account
 +
#*There are two values: ldap and override. A ldap source means the user comes from ldap and user attributes should be overridden on sync. A type of override means the local values should not be replace by those in AD. This is useful for overriding quotas or keeping accounts enabled that would otherwise be disabled.
 +
#Enabled/disabled state
  
[[Category:Obsolete Page]]
+
The script is commented, so its internal workings will not be copied here. However, of particular note is the '''perform_actions''' variable at the top of the parameters section of the script. This variable should be set to 0 for an initial test run, in order to perform a sanity check of the output. The script will print all actions it will perform, including exact commands that it will run. Setting this variable to 1 will result in the script actually running these commands.

Revision as of 20:01, 5 September 2009

History

The current TJ email system is an idea originally designed by Aman Gupta (class of '04). The original system was implemented by Andrew Deason (class of '06) with the help of several staff members, including Richard Washer and Susan Beasley. A second mail system, based on Zimbra, was configured by Lee Burton. The latest mail system was designed by Brandon Vargo.

Previously, the CSL only hosted mail for the CSL itself; mail that was only accesible to people with Systems Lab accounts. Today, lab-only mail does not exist anymore, and everyone gets their email off of the same system, all with @tjhsst.edu addresses.

Overview

The current mail system consists of two primary email servers as well as a mailman server. The two primary email servers, casey and smith are named after the founders of UPS and FedEx, Jim Casey and Frederick Smith, respectively. The servers are virtually identical; only the hostname/mailname and network information is different. Each runs postfix, dovecot, and amavisd-new. The mailman server, lists, runs a standard mailman installation.

When mail is first received by the mail system, it is processed by postfix. If basic spam tests pass, the message is passed to amavisd-new for additional spam processing. If the message passes these tests, it is placed into another internal postfix queue. In general, mail will only be tagged instead of being discarded. At this point, postfix is configured to pass the message to dovecot for delivery. Both servers share a common NFS filestore. Dovecot is configured to handle locking.

Authentication/Authorization

Authentication is handled by the kerberos PAM module, similar to most other systems in the lab. Dovecot is configured to user PAM.

Instead of using the Lab's LDAP tree for user information, account information is stored locally on each server in the form of standard UNIX accounts. A mail management script, described below, synchronizes the AD user information with the local UNIX accounts. The UID for the local UNIX account is a hash of the username. This allows each server to run the mail synchronization script independently and not worry about UID conflicts on the shared NFS storage.

Postfix

At the heart of the mail system is the excellent mail server Postfix. It was chosen over alternatives for its robustness, ease of configuration, and ease of use.

Configuration

Postfix configuration is accomplished through several files in /etc/postfix/. Here we will outline their general structure. After changing most configuration files, Postfix must reload its configuartion files with the command postfix reload.

main.cf

This configuration file tends to be the longest, as it houses nearly all of the options one might be concerned with. As this file is commented, there is no need to duplicate its function here.

master.cf

This configuration file describes how the Postfix daemon should run and determines which ports it should be listening on for connections and how to handle those connections. The default format of this file is perfectly fine, with only two additions necessary in our setup. The first is the amavis line, which is configured for spam filtering. The second is the line beginning with 127.0.0.1. This second line accepts mail on an alternate port and delivers the mail directly, bypassing spam filtering. This is the port to which amavisd-new delivers mail. If amavisd-new delivered mail to the standard postfix port, all messages would be put into an endless loop of spam filtering.

Running

Running Postfix is relatively easy, with most utility coming from postfix <command>. The following commands are recognized:

  • check

Checks the configuration files without starting or stopping Postfix. Useful to check if you've made mistakes on a production machine.

  • start

Starts the Postfix mail system.

  • stop

Stops the Postfix mail system in a nice way. Processes stop naturally.

  • abort

Stops the Postfix mail system abruptly. This one should be avoided if possible.

  • flush

Flushes the mail queue, forcing Postfix to try delivering all mail that has accumulated in the queue.

  • reload

Causes Postfix to reload its configuration files and restart processes nicely.

Typically the daemon is started and stopped with /etc/init.d/postfix start and /etc/init.d/postfix stop.

Other commands of interest include mailq, which lists all mail currently stored in the queue. This can be flushed using postfix flush, as indicated above.

Postfix sends all its logging information to syslog, which usually appends it to the file /var/log/mail.log. Examining this file is often key to troubleshooting issues with the mail system.

Amavis

Amavisd-new is used to scan all incoming and outgoing mail for virii and spam. It serves as an interface between the MTA, Postfix, and the various scanning utilities: Clam Antivirus for virus protection, and SpamAssassin for spam detection.

Clam Antivirus

The ClamAV package is being used to scan for virii in email. It functions in two modes: first as a daemon that runs all the time and scans data fed to a socket, and also as a standalone command line utility, clamscan. In most circumstances we will be using the former due to its increased performance, falling to the latter only if the daemon fails.

The virus definitions are updated about five times a day off the Internet through the freshclam daemon. In this way, new virii in the wild will be detected early and stopped from entering the TJHSST mail system.

SpamAssassin

SpamAssassin is used to scan all mail coming into the lab for its likelihood of spam, and all messages are given points for certain key phrases that are common in spam. With our configuration, very little user-level configuration is available. Additional rule files have been added under /etc/spamassassin/. SpamAssassin can be run as a daemon, but the mail system is configured to use the spamassassin tool directly.

Configuration

Configuration of Amavis is relatively simple and commented well, thus its configuration file will not be repeated here. Virtually all configuration is found in the files in /etc/amavis/conf.d/. Files are processed in order.

Running

Very little needs to be done to Amavis to get it running besides making sure that all necessary daemons are running, specifically clamd, freshclam, and amavisd-new. When Amavis receives a new mail on its socket, it will pass it through the virus scanner and then either discard the email or approve it and add an email header that indicates it was passed. It will then run the email through SpamAssassin and add any necessary headers indicating spam level. No spam will automatically be discarded; it is up to the user to filter as s/he wishes. Dovecot will automatically put spam in a spam folder.

Logging information is sent to syslog, which typically appends it to /var/log/mail.log. In this will be indicated whether the emails passed or were infected, and in the case of the latter, which virus/ii were involved.

Mailman

E-mail distribution lists are provided on the domain lists.tjhsst.edu by Mailman, a highly popular software package. Most list administration takes place through an easy online interface at https://lists.tjhsst.edu/, though many functions are also available through email, such as list subscription and some message handling.

Configuration

Configuration for this software package is minimal; the main parameters that must be specified are the urls and domains to be used. A modified version of the main postfix+amavisd-new configuration files are used. Integration with Postfix is achieved through the postfix-to-mailman.py script, which provides the necessary functionality.

Running

Postfix pipes any email with a destination of lists.tjhsst.edu to this script, which runs the necessary Mailman commands to deliver the email. Mailman, of course, ends up sending mails back to Postfix, destined for the actual recipients. Postfix will then deliver these mails to the mail mail systems or forward them to the appropriate mail server.

A few daemons must be running for Mailman to function properly. The first is qrunner, which processes any mail in Mailman's queue. If this is not running, e-mails to lists will be added to the queue but will never be delivered to recipients. This is started by the mailman init script. The other necessary daemon is apache, which provides the necessary online interface for list administration.

Dovecot

Dovecot proves IMAPs, POPs, and sieve support, in addition to being the local mail delivery agent. Dovecot receives mail from postfix and is responsible for delivering it to the user's mail directory. If dovecot is not running, postfix will queue mail.

Dovecot also supports quotas. If a user is over their soft quota, dovecot will give a warning but still store the message. If a user exceeds their hard quota, the mail message will be rejected.

Account Management

See Account provisioning for information on running the synchronization script.

The mail.py script is a python script responsible for syncing the local user information with the Active Directory user database. This script should be run on all mail servers, even though there is a common file storage.

Users are stored in groups in AD according to whether a mail account is required and the desired quota. This information is compared to a local user database, /etc/mailmanage/users, a CSV file that lists all mail users past and present. Users in AD that are not in the local database will have accounts created. Users in the local database but not in AD will have their account disabled unless an override is specified. Currently it is necessary to edit this file directly to override a quota or keep an account enabled.

CSV fields:

  1. The username
  2. The quota, before the multiplication value is taken into effect
  3. Source of the user account
    • There are two values: ldap and override. A ldap source means the user comes from ldap and user attributes should be overridden on sync. A type of override means the local values should not be replace by those in AD. This is useful for overriding quotas or keeping accounts enabled that would otherwise be disabled.
  4. Enabled/disabled state

The script is commented, so its internal workings will not be copied here. However, of particular note is the perform_actions variable at the top of the parameters section of the script. This variable should be set to 0 for an initial test run, in order to perform a sanity check of the output. The script will print all actions it will perform, including exact commands that it will run. Setting this variable to 1 will result in the script actually running these commands.