...making Linux just a little more fun!

<-- prev | next -->

Digging Secure Tunnels with IPsec

By René Pfeiffer


The Internet was born using plain text and no encryption. For a long time the TCP/IP protocol suite had no mechanism for cryptographically protecting transported data. Encryption was added at the application layer — Netscape's Secure Socket Layer (SSL) being a famous example. The design process of IPv6 incorporated encryption into the protocol itself, and the IPsec (IP security) framework came into existence. IPsec provides encryption and authentication at the packet level. While IPsec is obligatory for IPv6, you can optionally use it with IPv4. The Linux 2.6.x kernel series added full IPsec functionality to the main source tree. In this article we will explore how we can use IPsec to build encrypted data paths between networked machines.

The Parts of IPsec

IPsec consists of a number of protocols. Encryption was not the only design criteria. Protection against replay attacks, detection of unauthorised packet modification and correctly authenticating the communication partners are also included in the design requirements. IPSec also offers methods to manage keys used for encrypted communication. The protocols by name have the following task.

AH and ESP packets transport data. Both are new protocols whose protocol numbers can be found in /etc/protocols. AH and ESP are managed by the kernel, while IKE is managed by a userspace program.

Preparing Kernel and System

In order to use IPsec your kernel needs to have some code options enabled. Most modern GNU/Linux distributions have IPsec-capable kernels. If you want to compile your own, make sure that you have the following options set:

These options can be found in the section Networking --> Networking options. You also have to enable some or all cryptographic ciphers in the Cryptographic options submenu. You will need at least MD5, SHA1, HMAC, DES, Triple DES EDE and AES. Remember, the encryption is handled by the kernel. If the kernel doesn't know about a cipher, it cannot use it. This is important in case you want to speak to other IPsec devices or hosts (such as MS Windows or Cisco machines). If you want to use IPsec with IPv6, then you have to make sure the IP transformations are set for IPv6 as well. I have two screenshots for you where you can see the make menuconfig menu: screenshot 1 and screenshot 2. If your machine acts as a router, you might wish to consider activating the IP: advanced router option, too. The kernel treats packet queues differently if you use the advanced routing option.

Now our kernel can handle IPsec. We now need some tools to make it work. We will need at least the ipsec-tools package. This is the name of the project and the Debian package. If we want to deal with key management and IKE, we need another program. The Linux IPsec stack can work together with pluto from the Openswan project, OpenBSD's isakmpd or racoon from the KAME project. The use of IKE is optional though. We will use racoon in our examples.

Manual Keying, Policies, Tunnel and Transport Mode

IPsec can be used to link networks via tunnels by using the so-called tunnel mode. In its simplest form it can also be used to encrypt network traffic between two or more hosts by using the transport mode. The only things you need for that are the keys and a way to tell the kernel which packets need to be sent via IPsec. The 2.6.x kernels have no special device for handling IPsec packets. Everything is sent over the already existing network interfaces. The Security Policy Database (SPD) decides which packets are to be handled by IPsec. In order to manipulate this database you need the setkey command from the ipsec-tools package. Usually you prepare a file with all the settings and activate it by using setkey -f /etc/setkey.conf. As an example, let's say we want to enable IPsec between the machines and The policy for telling the kernel looks like this:

#!/usr/sbin/setkey -f
# SPD for
spdadd any -P out ipsec

spdadd any -P in ipsec
The first policy states that any packet coming from and leaving for has to be encapsulated in IPsec packets. Transport mode has to be used. The policy is valid for ESP and AH packets alike (that's why we have to use spdadd twice). IPsec is mandatory as indicated by the keyword "require". If one of the hosts lacks the right key or hasn't initialised its SPD, there won't be any traffic because it can't be encrypted. The second policy is the first one reversed. The IP addresses are swapped and the direction is changed from out to in.

So the kernel knows when to use IPsec. There are still no keys. Apart from that I told you about authentication checks that the IPsec protocols can do for us. Our setkey.conf needs to be extended to include this information as well. setkey also defines the Security Association Database (SAD). The SAD tells the kernel who our neigbours are and how we can make sure that we are not talking to an impostor. Extending our setkey.conf by the following lines enables authentication and encryption. In addition to that we supply the keys for every host.

# AH SAD entries with 160 bit keys
add ah 0x200 -A hmac-sha1 0x46915c30ed7e2465b42861b6ab19f2772813020c;
add ah 0x300 -A hmac-sha1 0xc4dac594f8228e0b94a54758f7fbf2fdf4e37f3e;

# ESP SAD entries with 192 bit keys
add esp 0x201 -E rijndael-cbc 0xa3993b3dfc41ef0a1aa8d168a8bf2c27e48249ac17b61e09;
add esp 0x301 -E rijndael-cbc 0x8f6498928ba354bd45cfad147f54c67b3b742896b3bafc02;
Again we have a line for every direction. The IP addresses are reversed, but we use a different key for every IP and protocol. The bit length of the key corresponds to the authentication or encryption algorithm used. The switches -A and -E indicate the algorithm to use for AH and ESP respectively. hmac-sha1 requires a key length of 160 bits or 20 bytes. rijndael-cbc can be used with 128, 192 or 256 bits. The example uses 192 bits or 24 bytes. The man page of setkey has a table with all possible values for every supported algorithm. Bear in mind that the kernel must also have a module for the algorithm in its cryptographic options or else you cannot use this particular algorithm. The hexadecimal value behind the protocol name is called Security Parameters Index (SPI). The SPI identifies a set of parameters used for the IPsec connection in combination with the IP addresses involved. When doing manual keying, make sure that the SPIs are unique. Speaking of unique, make sure that your keys are unique and random. Never use any keys that have been published! I used Ralf's method from the IPsec HOWTO to extract the sample keys from the Linux random device.
# dd if=/dev/random count=24 bs=1 | xxd -ps
24+0 records in
24+0 records out
24 bytes transferred in 0.000180 seconds (133298 bytes/sec)
Set count to the desired byte amount. The command xxd used to convert the binary output from the device is part of the vim package.

Testing Transport Mode

We are now ready to test our configuration. In order to do this you will need the full setkey.conf. I only added two lines that clear the SAD and SPD before loading new rules, just to be sure.

Copy it to your hosts. Be careful, we've only created the setkey.conf for If you use this file on you have to swap the policy for the direction of the packet flow (the in and out keywords for the SPD). Now use a root shell on and enter the command:
setkey -f /path/to/setkey.conf
Check if you can ping This should not be possible, because we told to communicate with over IPsec only. If you run the setkey command on as well, you should be able to ping from Take a look with your favourite sniffer in order to make sure the kernel isn't playing unencrypted tricks on you. If you use an ICMP ping, the sniffer should only show you encrypted AH or ESP packets. The same goes for TCP and UDP transmissions.

Next time we will configure an IPsec tunnel to connect two different networks and we will take a look at automatic keying with X.509 certificates.

Further reading

Talkback: Discuss this article with The Answer Gang


René was born in the year of Atari's founding and the release of the game Pong. Since his early youth he started taking things apart to see how they work. He couldn't even pass construction sites without looking for electrical wires that might seem interesting. The interest in computing began when his grandfather bought him a 4-bit microcontroller with 256 byte RAM and a 4096 byte operating system, forcing him to learn assembler before any other language.

After finishing school he went to university in order to study physics. He then collected experiences with a C64, a C128, two Amigas, DEC's Ultrix, OpenVMS and finally GNU/Linux on a PC in 1997. He is using Linux since this day and still likes to take things apart und put them together again. Freedom of tinkering brought him close to the Free Software movement, where he puts some effort into the right to understand how things work. He is also involved with civil liberty groups focusing on digital rights.

Since 1999 he is offering his skills as a freelancer. His main activities include system/network administration, scripting and consulting. In 2001 he started to give lectures on computer security at the Technikum Wien. Apart from staring into computer monitors, inspecting hardware and talking to network equipment he is fond of scuba diving, writing, or photographing with his digital camera. He would like to have a go at storytelling and roleplaying again as soon as he finds some more spare time on his backup devices.

Copyright © 2006, René Pfeiffer. Released under the Open Publication license unless otherwise noted in the body of the article. Linux Gazette is not produced, sponsored, or endorsed by its prior host, SSC, Inc.

Published in Issue 125 of Linux Gazette, April 2006

<-- prev | next -->