Warning Livedoc is no longer being updated and will be deprecated shortly. Please refer to https://documentation.tjhsst.edu.

Difference between revisions of "IPSec VPN"

From Livedoc - The Documentation Repository
Jump to: navigation, search
(Addendum for TJ users: removed statement that is no longer true)
(/etc/racoon/racoon.conf: md5 hash_alg not supported in newer firmwares by third party report)
Line 78: Line 78:
proposal {
proposal {
encryption_algorithm aes;
encryption_algorithm aes;
hash_algorithm md5;
hash_algorithm sha1;
authentication_method xauth_psk_server;
authentication_method xauth_psk_server;
dh_group 2;
dh_group 2;

Revision as of 15:21, 18 June 2013

The primary Systems Lab VPN uses OpenVPN. An IPSec VPN server was set up to allow the use of the Sun Ray 2 series built-in VPN client so users can easily connect from home (in a roadwarrior configuration where we do not know the IP of the remote host). In addition, the VPN server is neither the subnet router nor the border router/internet gateway. It should be noted that connecting to the IPSec VPN from within the primary subnet and trying to access an address on the same primary subnet from a Sun Ray does not seem to work with pre-4.1 firmware. If you are on the primary subnet, you should be using a direct connection to the Sun Ray servers anyways.

This has also been tested and found to work with the Cisco Systems VPN Client on Windows XP SP3 and Windows 7 in Vista Compatibility Mode ( and Solaris 10 10/08 SPARC (

For a free and open source alternative to the Cisco VPN client, try the Shrew Soft VPN client for Windows, Linux, and BSD. This is the only TJ-test client known to work with 64-bit Windows. It has been tested to work on Windows XP SP3 and Windows 7, but should work for all versions of Windows. We have not tested the Linux client, but have no reason to believe that it doesn't connect correctly.


The Sun Ray VPN client was designed to be used with a Cisco EasyVPN (or sometimes called EzVPN) server. While there is no documented way to use any other VPN server, we have been able to set up a configuration which is compatible as EasyVPN is standards-compliant. The VPN server requires the following features:

  • Pre-shared key authentication
  • Aggressive mode connection
  • 3DES or AES encryption
  • MD5 or SHA1 hash algorithm
  • Mode config

Solaris IPSec currently does not support all of these features. The two Linux VPN servers identified as supporting these features are OpenSWAN and Racoon. However, the OpenSWAN keying daemon attempts to negotiate XAUTH after initiating IKE phase 2 negotiations, whereas the specification (which the Sun Ray follows) calls for XAUTH after completing IKE phase 1 but before beginning IKE phase 2. As a result, racoon is the only known compatible server at this time. We filed bug 969 against OpenSWAN, so there is a chance that OpenSWAN will become compatible in the future.

The VPN server was implemented on Debian lenny (etch does not contain a new enough racoon and/or ipsec-tools to support the above listed features) using ipsec-tools and racoon versions 0.7-2.1.

Note: you may want to use a kernel no newer than 2.6.24. A "bug" was introduced in 2.6.25 and newer which does not appear to be fixed in the newest kernel yet. This bug causes messages like:

Jan 10 22:59:36 ipsecvpn kernel: [1539189.352603] SKB BUG: Invalid truesize (360) len=174, sizeof(sk_buff)=232

to appear in the kernel log under certain operational conditions. It is unclear if these messages actually mean anything, or that the bug does not exist in 2.6.24 and earlier, but at least the messages do not appear in earlier kernel versions. See also http://bugzilla.kernel.org/show_bug.cgi?id=10996.

The SPARC version of Debian also appears to contain a bug where if the client attempts to use a group ID that is longer than 16 characters, racoon will silently crash.

Network details

  • is the primary internal subnet.
  • is the public IP address of the remote Sun Ray or the NAT router in front of it.
  • is the IPSec VPN server service IP.
  • is the pool of addresses we've chosen to be assigned to connecting remote devices to present to the internal network.
  • (hostname centauri.sun.tjhsst.edu) is a Sun Ray server.
  • is a DNS server.

Configuring the VPN server

  • Make sure your kernel has the necessary IPSec modules compiled in. Note that, while most kernels do, the Xen kernel is a notable one that does not by default.
  1. Install ipsec-tools and racoon.
  2. Enable IP forwarding.
    1. echo 1 >> /proc/sys/net/ipv4/ip_forward
    2. edit /etc/sysctl.conf so that forwarding is enabled at boot.
  3. Set up your pre-shared key file (see "/etc/racoon/psk.txt" below). We use e-mail address for the PSK identifier, but other forms of identifier can also be used. While it is possible to share a single ID and key among multiple users, we have chosen to assign each user a unique PSK.
  4. Configure PAM for the racoon service, or just allow racoon to fall back on the "other" service.
  5. Set up the racoon configuration file (see "/etc/racoon/racoon.conf" below).
  6. Start or restart racoon (/etc/init.d/racoon start).

Sample configuration files:


#groupid            groupkey
wyang@tjhsst.edu    secretpsk


#Below line is not required for Sun Ray VPN to work.
path certificate "/etc/ssl/certs";
path pre_shared_key "/etc/racoon/psk.txt";

#This section restricts listening to the service IP; not strictly necessary.
listen {
	isakmp_natt [4500];

remote anonymous {
	exchange_mode aggressive;
	passive on;
	generate_policy on;
	#The below line makes it work with other devices that propose
	#  a lifetime longer than the racoon default of 28800.
	#Alternatively, set a longer lifetime.  "proposal_check claim" may
	#  work suboptimally with frequent renegotiation of phase 2.
	proposal_check obey;
	#we need nat force for some connections to work
	#  if this breaks things for you, try "on" instead
	nat_traversal force;
	ike_frag on;
	proposal {
		encryption_algorithm aes;
		hash_algorithm sha1;
		authentication_method xauth_psk_server;
		dh_group 2;

sainfo anonymous {
	encryption_algorithm aes, 3des;
	#sha1 is considered a little more secure, md5 is considered
	#   a little faster, enforce one if desired
	#see http://www.ciscopress.com/articles/article.asp?p=25473
	authentication_algorithm hmac_md5, hmac_sha1;
	compression_algorithm deflate;

mode_cfg {
	pool_size 254;
	#Below line is not strictly necessary because it is the default,
	#  but may be needed for other networks.
	#Below 2 lines only needed if you wish to use hostnames
	#  or allow the Sun Ray to query for sunray-config-servers
	#  or sunray-servers
	default_domain "sun.tjhsst.edu";
	#Below line is not required for Sun Ray VPN to work.  See man page for details.
	split_network include;
	auth_source pam;

Logging for racoon occurs in /var/log/syslog. This is what a successful connection looks like:

Jul 17 20:09:10 vpnserver racoon: INFO: respond new phase 1 negotiation:[500]<=>[500]
Jul 17 20:09:10 vpnserver racoon: INFO: begin Aggressive mode.
Jul 17 20:09:10 vpnserver racoon: INFO: received Vendor ID: RFC 3947
Jul 17 20:09:10 vpnserver racoon: INFO: received Vendor ID: draft-ietf-ipsec-nat-t-ike-03
Jul 17 20:09:10 vpnserver racoon: INFO: received Vendor ID: draft-ietf-ipsec-nat-t-ike-02
Jul 17 20:09:10 vpnserver racoon: INFO: received Vendor ID: draft-ietf-ipsec-nat-t-ike-02
Jul 17 20:09:10 vpnserver racoon: INFO: received Vendor ID: draft-ietf-ipsra-isakmp-xauth-06.txt
Jul 17 20:09:10 vpnserver racoon: INFO: received Vendor ID: CISCO-UNITY
Jul 17 20:09:10 vpnserver racoon: INFO: Selected NAT-T version: RFC 3947
Jul 17 20:09:10 vpnserver racoon: INFO: Adding remote and local NAT-D payloads.
Jul 17 20:09:10 vpnserver racoon: INFO: Hashing[500] with algo #1
Jul 17 20:09:10 vpnserver racoon: INFO: Hashing[500] with algo #1
Jul 17 20:09:10 vpnserver racoon: INFO: Adding xauth VID payload.
Jul 17 20:09:10 vpnserver racoon: INFO: NAT-T: ports changed to:[50341]<->[4500]
Jul 17 20:09:10 vpnserver racoon: INFO: Hashing[4500] with algo #1
Jul 17 20:09:10 vpnserver racoon: INFO: NAT-D payload #0 verified
Jul 17 20:09:10 vpnserver racoon: INFO: Hashing[50341] with algo #1
Jul 17 20:09:10 vpnserver racoon: INFO: NAT-D payload #1 doesn't match
Jul 17 20:09:10 vpnserver racoon: WARNING: ignore INITIAL-CONTACT notification, because it is only accepted after phase1.
Jul 17 20:09:10 vpnserver racoon: INFO: NAT detected: PEER
Jul 17 20:09:10 vpnserver racoon: INFO: Sending Xauth request
Jul 17 20:09:10 vpnserver racoon: INFO: ISAKMP-SA established[4500]-[50341] spi:dd210acdb6499143:ba2c78cce1c8f439
Jul 17 20:09:11 vpnserver racoon: INFO: Using port 0
Jul 17 20:09:11 vpnserver racoon: INFO: login succeeded for user "wyang"

At this point, the VPN status on the Sun Ray should read "PH1 agg I est" indication completion of IKE phase 1. After a brief pause, the Sun Ray will begin IKE phase 2 negotiation "PH2 quick I msg":

Jul 17 20:09:20 vpnserver racoon: INFO: respond new phase 2 negotiation:[4500]<=>[50341]
Jul 17 20:09:20 vpnserver racoon: INFO: no policy found, try to generate the policy :[0][0] proto=any dir=in
Jul 17 20:09:20 vpnserver racoon: INFO: Adjusting my encmode UDP-Tunnel->Tunnel
Jul 17 20:09:20 vpnserver racoon: INFO: Adjusting peer's encmode UDP-Tunnel(3)->Tunnel(1)
Jul 17 20:09:21 vpnserver racoon: INFO: IPsec-SA established: ESP/Tunnel[50341]->[4500] spi=25604673(0x186b241)
Jul 17 20:09:21 vpnserver racoon: INFO: IPsec-SA established: ESP/Tunnel[4500]->[50341] spi=12811916(0xc37e8c)
Jul 17 20:09:21 vpnserver racoon: ERROR: such policy does not already exist: "[0][0] proto=any dir=in"
Jul 17 20:09:21 vpnserver racoon: ERROR: such policy does not already exist: "[0][0] proto=any dir=fwd"
Jul 17 20:09:21 vpnserver racoon: ERROR: such policy does not already exist: "[0][0] proto=any dir=out"

The last three messages are not actually errors. They simply indicate that there was no predefined policy. After IKE phase 2 the Sun Ray should briefly read "PH2 quick I est" before attempting to connect to the Sun Ray server.

Configuring routing

Packets coming from VPN clients will originate from While the Sun Ray will be able to send packets to other systems on the network, it will not be able to receive replies unless the appropriate routes are set up as only the VPN server knows where actually is. Since our primary subnet has a Cisco IOS switch/router, we simply access our switch and add the following route:

ip route

If you don't have a router on the subnet you want the Sun Rays to access, you will need to add a route to each Sun Ray server. For example, if we did not have a central router on our subnet (assuming that somehow the Sun Ray is still able to connect to the VPN server), we would run this command on each Sun Ray server:

route -p add

The "-p" argument can be used on Solaris 10 and newer to keep the route persistent across reboots. For older Solaris versions and for Linux, ensure that the route is set at boot time using the appropriate method for the distribution.

For Debian-based distributions (including Ubuntu), you may find the following helpful: http://www.itsyourip.com/networking/add-persistent-static-routes-in-debian-linux/

If you configure the DNS options for the VPN, you will need to make sure that the DNS server can talk to the VPN subnet as well.

Configuring the client

Configuring the Sun Ray client

You will first need to load the client with a GUI firmware version. The steps to configure the server to do that will not be covered here (TJ users see below).

  1. Access the menu by pressing Stop-M or Stop-S. On a non-Sun keyboard, Ctrl-Break can be used in place of Stop (so Ctrl-Break-M).
  2. Broadcasting for a Sun Ray server will not work over the VPN tunnel.
    1. If VPN DNS was configured (TJ users use the first bullet under this heading):
      1. No configuration is required if sunray-config-servers is registered in DNS and either /tftpboot/*.parms files are configured or sunray-servers is registered in DNS.
      2. Optionally, you can enter the hostname (centauri.sun.tjhsst.edu) or IP ( of one of your Sun Ray servers.
    2. If VPN DNS was NOT configured:
      1. Under "Servers", enter the IP of one of your Sun Ray servers (
    3. Firmware and Log Host may be left blank. Save change and exit to the main menu.
  3. Under "VPN/IPsec":
    1. Set enable to on.
    2. Set peer to the VPN server (
    3. Set group to the groupid that you set in the psk.txt file on the VPN server (wyang@tjhsst.edu).
    4. Set group key as indicated in the psk.txt file (secretpsk).
    5. Set username and password to a user that can succesfully authenticate via PAM.
    6. Save changes and return to the main menu.
  4. Exit the main menu. The Sun Ray should now power cycle and begin attempting to connect to your VPN server.

Addendum for TJ users

  • If your Sun Ray was checked out without ever having been plugged in to the network within the building, chances are you do not have the GUI firmware. To get the GUI firmware you will need to do one of two things:
    • Plug in the Sun Ray to the network where there is currently already a Sun Ray (so if it is at home you will need to bring it in, but you don't need any cables or other parts other than the unit itself). The firmware should update itself within a few minutes. Once the firmware has been updated, you can take the Sun Ray home and set it up per the instructions above.
    • Bridge the Sun Ray into the TJ network. You can do this using OpenVPN and a computer with 2 NICs (although there are probably other ways to do it). This is more complicated than the first method, but you do not have to bring anything back in the building.
  • Send an e-mail to either Trey Repetski or another Sun admin, or access /etc/racoon/psk.txt on the VPN server to retrieve your group key. Your groupid will be your TJ e-mail address.
  • Currently, you will need to use your CSL Kerberos username and password. Eventually you will also be able to use your LOCAL password. If your Kerberos password is failing to log you on, please contact one of the above administrators and ask that you be added to access control for the server.

Configuring Shrew Soft VPN

  • "Host Name or IP Address" is any of the hostnames for the IPSec VPN server - ipsecvpn[1-4].tjhsst.edu; the port can be left at 500
  • Authentication
    • Mutial PSK + XAuth
    • Under Local Identity, use a Key Identifier and the Key ID is your TJ email
    • The Pre Shared Key under the Credentials tab should be the passoword specified on the VPN server

Multiple Connections from the same NAT

If you have multiple IPSec clients (either Sun Rays or otherwise) coming from behind the same NAT IP address, you will need to give your VPN server multiple public IP addresses. Each client should connect to a different IP address on the VPN server. It is unclear whether the IPSec NAT tunnelling specification or a bug in the racoon/ipsec-tools software causes this issue. If you use just a single IP, only one client at a time will be able to maintain a connection to the VPN.

Byline and Acknowledgements

This document was authored by 2006-08 Sun Technology Coordinator William Yang. Special thanks to Lee Burton and Brandon Vargo for their technical advice, and also to Stephen (Trey) Repetski and Matt Green for their help with testing. Also, a very special thanks to Sun for their Academic Excellence Grant, without which we would have had neither the equipment nor the motivation to pursue making this work.

Additional thanks go to Jonathan Mills at Sun Microsystems for his suggestions.