IP Security

IPsec Protocols

1
IPsec – SANS ©2000 - 2003

This course will provide students with a basics understanding of the terms,
concepts, and operation of the IP Security (IPsec) protocol. The Internet
Protocol (IP), of course, is inherently non-secure and IPsec provides
mechanisms for message integrity, confidentiality, and authentication. IPsec
is also becoming a critically important protocol for the implementation of
virtual private networks (VPNs), which will be discussed more in this
chapter.
This course is based upon one originally developed by Jean Triquet, a senior
communications specialist with Public Works and Government Services of
Canada. This particular version has been written by Gary Kessler, program
coordinator of the Computer Networking major at Champlain College in
Burlington, Vermont.

9-1
 Outline

• IPsec overview, applications, terms, and concepts

• Security Associations (SA)
– Databases
– Modes
• Internet Key Exchange (IKE)
• The Authentication Header (AH) protocol
• The Encapsulating Security Payload (ESP) protocol
• Summary

2
IPsec – SANS ©2000 - 2003

The objective of this presentation is to teach the basic of IPsec as an introduction to

the topic. As such, we will focus on an overview of what IPsec does, and define
important terms and concepts. A secondary function of this course is to prepare
individuals to learn more about IPsec in other courses, either for purposes of intrusion
detection or for building VPNs. Where possible, pure theory is left for those later
courses; this course will focus on the basic concepts and show some IP/IPsec traffic
flows.
The slide above lists the main topics that we will cover. Security Associations are the
fundamental concept of IPsec. SAs define how two IPsec parties communicate. There
are some databases associated with SAs that we need to discuss as well as the
operational modes that can be employed.
Another fundamental concept is key exchange for IPsec. This is perhaps the most
complex part of IPsec (and a place where product incompatibilities often lie). In this
course, we will only briefly examine how keys are managed in IPsec but there is a lot
more to learn!
There are two protocols defined for IPsec, namely the Authentication Header (AH)
and the Encapsulating Security Payload (ESP). Both will be discussed with several
examples of packet formats and protocol captures.

9-2
 IPsec - Security Services
IPsec is a set of protocols which add security
services to the Internet Protocol:

Host Host
- Confidentiality
- Authentication
- Integrity
- Access control
- Some protection from traffic flow analysis
Public IP
Network

Security Security IPsec procedures can be used:

Gateway Gateway
- Host-to-Host
- Security gateway-to-Security gateway
Private IP - Host-to-Security gateway
Private IP
Network Network

3
IPsec – SANS ©2000 - 2003

The diagram on the slide shows the different type of relationships that can be
established with IPsec. First, let's define the devices shown here.
An IPsec host is any end-communicating device in which IPsec has been implemented
in the TCP/IP protocol stack. An IPsec host, then, can be any IP host (such as my PC),
a router, firewall, or other server.
A security gateway is some sort of intermediary device that implements IPsec for the
protection of a network, such as a firewall or router. If a device uses IPsec to secure
communications coming from and/or going to a network segment, than it is a security
gateway for IPsec definitional purposes.
As the diagram suggests, IPsec can be implemented between two hosts, between two
security gateways, or between a host and a security gateway.
IPsec is a constantly evolving set of protocols. RFC 2401 describes the overall
architecture of IPsec and RFC 2411 is the "IP Security Document Roadmap," a
necessary document considering that IPsec currently comprises more than a dozen
RFCs. Those dozen RFCs can broadly be categorized as describing:
• Internet Key Exchange (IKE)
• Authentication Header (AH)
• Encapsulating Security Payload (ESP)
All of these topics will be discussed further in this chapter.

9-3
 IPsec Applications

• Designed to add security features to IP

– Viewed as protocol layer between IP and
TCP/UDP
• Application for virtual private networks
– Intranet VPN
– Extranet VPN
– Remote access VPN

4
IPsec – SANS ©2000 - 2003

IPsec's primary role is to provide security for IP communications. As we will see later in this
chapter, the IPsec protocols "sit" between IP and higher layer protocols such as TCP, UDP, and
ICMP.
As an aside, it is worth noting that situations like these cause the Open Systems Interconnection
(OSI) model to start to unravel. If IP is at the OSI Network Layer and TCP/UDP is at the OSI
Transport Layer, then what is the layer in between? We just have to rely on our understanding
of encapsulation and peer-to-peer communications to get us out of this quandary. TCP/UDP is
still used on an end-to-end, host-to-host basis. IP is still used router-to-router. This new IPsec
security layer, as we will see, provides communication between IPsec devices. The IPsec
devices are interconnected with routers, which is why we need IP below IPsec, and the IPsec
devices are not TCP/UDP hosts, which is why we need that layer above IPsec. So we just have
one Network Layer (IPsec) carried in another (IP); this, by the way, is the definition of
tunneling.
The main application for IPsec is to build private, secure communications channels between IP
devices. As such, we are using IPsec to create VPNs. There are basically three kinds of VPNs
where IPsec can play an important role:
• An Intranet VPN is a connection between two corporate sites where all users at one site can
access the network resources found at the other site. An intranet VPN basically extends a site's
LAN to another site.
• An Extranet VPN provides a connection between two sites where full access to the other's
resources is not required (or desired). An example might be communication between two
business partners, where we want incoming access to some of our systems but not all.
• A remote access VPN provides access to a corporate network from a "roaming" user, such as a
user in a hotel room with a laptop.

9-4
 Why Study IPsec?

• Use of IPsec affects operation of the

firewall
• IPsec can provide a secure channel
from an IDS sensor to analysis station
• Security analysts need to be able to
identify when IPsec is in use
• Incident handler may use IPsec within
team communications

IPsec – SANS ©2000 - 2003 5

As the slide suggests, there are many reasons why IPsec is a necessary topic for
security professionals in both the intrusion detection and firewall spaces.
First and foremost, IPsec represents a new set of protocols that will default to
being blocked by most firewalls. To use IPsec, then, "holes" must be punched
through the firewall and care must be taken when doing that.
Intrusion detection personnel may want to employ IPsec between the sensor
and analysis stations. If the IDS sensor lies outside of the firewall, it can
possibly be compromised and information sent to the protected sensor can be
monitored. IPsec can be used to provide a secure sensor-analyzer connection.
Even if IPsec is not employed for the IDS, it may be used on the VPN. In this
case, an analyst needs to know how to recognize and interpret IPsec traffic.
Finally, when large security events occur, some significant problems in site-to-
site communication invariably result. One is that many sites are knocked offline
(or made inaccessible); another is that you may not be able to trust messages
that come in from other sites ostensibly telling you how to protect yourself.
IPsec provides a way to establish VPNs between trusted sites for precisely this
reason.

9-5
 IPsec Terms and Concepts

• Security Associations
– Security Policy Database
– Security Associations Database
– Tunnel mode and Transport mode
• Internet Key Exchange
• IPsec protocols
– Authentication Header (AH)
– Encapsulating Security Payload (ESP)

IPsec – SANS ©2000 - 2003 6

On the next several pages, we will step through the IPsec concepts listed on this
slide. The discussion of the IPsec protocols will also show packet traces.

9-6
 IPsec Reference Network

10.10.2.0
10.10.8.0

172.16.12.65 10.20.20.17 FTP

192.168.1.1 10.10.8.1 server
192.168.1.10 Internet
(client) 10.10.8.21
Security Security 10.10.1.0 10.10.3.0
Gateway A Gateway B

192.168.0.0

10.10.1.10
FTP
server

10.10.0.0

7
IPsec – SANS ©2000 - 2003

This slide presents a fictitious, totally-contrived network that will be the reference
network for all discussions and examples in this chapter. We will use this network as
the basis to explain the IPsec Security Associations databases, as well as the IPsec
modes of operation and protocols.
Note that this diagram uses RFC 1918 private addresses for all network segments.
These address assignments are purely for this example reference network only. It is
not necessary to use private addresses with IPsec; in fact, as we will see, IPsec is not
terribly NAT-friendly so private addresses are actually a potential problem with IPsec.

9-7
 Security Associations

• The central IPsec concept

• Defines kind of security measures to be
applied to a packet based on sender,
receiver, and payload type
• Negotiated dynamically between a pair
of IPsec devices

8
IPsec – SANS ©2000 - 2003

Security Associations (SA) are the central concept to understanding IPsec. An

SA defines the kind of security measures that should be applied to packets
based upon who sent the packets, who is to receive the packets, and the type of
payload that the packets carry.
SAs are negotiated dynamically by two peer IPsec devices when they wish to
use any IPsec security procedures. The available procedures are defined by the
security administrator as a set of available security policies.

9-8
 Security Associations (2)

• Each SA is uniquely identified by

– Destination IP address of outer header
– Security Parameter Index (SPI)
– IPsec protocol identifier
• An SA only uses the destination IP
address because they are simplex
– Two SAs are required if IPsec services are
required bidirectionally

9
IPsec – SANS ©2000 - 2003

SAs are maintained in databases and are uniquely identified by three pieces of
information:
• Destination IP address: The IP address of the destination IPsec endpoint for
the SA.
• Security Parameter Index (SPI): A 32-bit integer chosen at random by the
destination endpoint for the SA and having significance only to that endpoint,
used to distinguish between SAs terminating at that IPsec device.
• IPsec protocol identifier: The protocol identifier for the AH (51) or ESP (50).
It is important to note that the SA uses only the destination address and even
the SPI has significance only at the destination side of the SA. The reason for
this is that SAs are simplex (unidirectional), so that the level of security agreed
upon by two IPsec devices is for data sent to the destination. If IPsec security
procedures are required in both directions, two SAs must be established. In
addition, note that the characteristics of the two SAs do not have to be the same
in both directions.

9-9
 SA Databases

• Each IPsec node maintains two

databases
• Security Policy Database (SPD)
contains set of security policies defined
for this node
• Security Association Database (SAD)
contains parameters for active SAs

10
IPsec – SANS ©2000 - 2003

Each IPsec node maintains two databases, called the Security Policy Database
(SPD) and the Security Association Database (SAD).
The Security Policy Database contains the set of security policies that meet the
needs of all of the types of IP traffic that will come in and out of this IPsec
node. The SPD contains, then, the possible rules to use when another IPsec
node requests connectivity and guides the negotiation of a new SA; two IPsec
nodes needs to have at least one compatible rule in order to establish an SA.
When an SA is successfully established between two IPsec nodes, the SA and
its parameters is registered in the Security Association Database (SAD).

9 - 10
 SA Rules (for Humans!)

SA Rules (Security Gateway A) SA Rules (Security Gateway B)

• Packets coming into this network
must be authenticated and the
information must be encrypted
• Packets coming into network segments 10.10.2.0 or
10.10.8.0 have to be authenticated and the information
must be encrypted (10.10.2.0 is particularly sensitive)
• Packets coming into network segment 10.10.1.0 are
not subject to IPsec policies
• Packets coming into any other segment will be
discarded

10.10.2.0

Internet 10.10.3.0

Security
Security
192.168.0.0 Gateway A Gateway B
10.10.8.0
10.10.1.0

10.10.0.0

11
IPsec – SANS ©2000 - 2003

We will continue to explore the IPsec databases but we will first take a minor detour. In this slide, we
return to our reference network and we describe the security rules that we'd like to put in place. Note
that these rules are described in simple prose for the sake of us people!
We'll start with the 192.168.0.0 network and Security Gateway A on the left because it is simpler to
deal with. The policy at this network is, simply, that any packet coming into this network must be
authenticated and the data encrypted.
The policy for the 10.10.0.0 network and Security Gateway B on the right is a little more complicated.
The 10.10.0.0 network is composed of four subnets:
• Any traffic coming to the 10.10.2.0 or 10.10.8.0 subnets must be authenticated and the information
encrypted. Furthermore, we note that the 10.10.2.0 subnet is particularly sensitive (and, therefore, we
might want a slightly higher level of security).
• Any traffic coming to the 10.10.1.0 subnet is not subject to any IPsec procedures. Thus, any traffic
coming into that subnet are allowed in freely.
• Any traffic coming to the 10.10.3.0 subnet is to be discarded. We have set this policy because, for
example, this subnet is for intranet use only and any packets from the outside are strictly prohibited.
Note that we have stated the rule above in the slide as traffic to "any other segment" (meaning neither
10.10.1.0, 10.10.2.0, nor 10.10.8.0). Explicitly stating 10.10.3.0 and stating "none of the above" may
appear to be the same but, in fact, the latter is safer; if another network segment is defined later on,
traffic will be blocked unless explicitly allowed. This may be a style thing, but it is one factor to be
aware of.

9 - 11
 Security Policy Database
Direction = Inbound
Direction = Inbound
Selector Security
Selector Security (Dest. Addr.) Action Level
(Dest. Addr.) Action Level
10.10.1.0/24 Bypass IPsec n/a
192.168.0.0/16 Apply IPsec Secret
10.10.2.0/24 Apply IPsec Top Secret
10.10.8.0/24 Apply IPsec Secret
10.10.0.0/16 Discard packet n/a

10.10.2.0

Internet 10.10.3.0

Security
Security
192.168.0.0 Gateway A Gateway B
10.10.8.0
10.10.1.0

10.10.0.0

12
IPsec – SANS ©2000 - 2003

This slide examines the SPD in a slightly more formal fashion. The IPsec node's SPD contains an
ordered list of security policies that applies to both inbound and outbound traffic. The information in
the list is processed in the order in which the rules are entered (much like packet filtering rules in a
firewall). SPD entries have at least two fields, namely a selector and an action.
The selector is one or more parameters that describe the packets subject to the security policy. Selector
parameters include information that is carried within the IP packet (such as source and destination IP
addresses, and TCP/UDP port numbers) and information that is derived from the authentication
credentials of the peer communicating entity (such as an e-mail address or digital certificate
distinguished name).
The action describes what IPsec processing should be performed, and there are three choices. "Apply
IPsec" means to apply some IPsec processing to the packet while "Bypass IPsec" means that IPsec rules
are not applied to the packet. "Discard" means what it says and the packet is discarded.
Our example SPDs in the slide merely implement the policies states in English on the previous slide.
First, note that both SPDs are for inbound packets. Second, note that the selector we use is destination
address. Note also that the SPD for the 10.0.0.0 network defines individual rules for the 10.10.1.0,
10.10.2.0, and 10.10.8.0 subnets, followed by a "default" policy of discard for the entire 10.10.0.0
network. Since the rules are processed in order, there is no ambiguity here.
The only new element we have added here is a third item called the Security Level. In this case, the
security level merely describes the choices of authentication and encryption algorithms that might be
employed on an SA; this is described more on the next page.

9 - 12
 Security Level Definitions

Secret
IPsec "ESP DES HMAC-MD5 MINUTES 300
or ESP 3DES HMAC-SHA MINUTES 300
or ESP 3DES HMAC-MD5 MINUTES 300"
ISAKMP "DES MD5 MINUTES 1440"

Top Secret
IPsec "ESP 3DES HMAC-SHA MINUTES 300"
ISAKMP "DES MD5 MINUTES 1440"

13
IPsec – SANS ©2000 - 2003

In the SPD on the previous page, we defined a Security Level. This is an example of a
local policy parameter that defines what level of security is acceptable for an SA based
upon the destination address. In the SPD we refer to the security levels as "secret" and
"top secret," both names that were chosen by the policy administrator.
Each definition above contains a section for IPsec data exchange and ISAKMP key
management. Each IPsec line contains the IPsec protocol to employ, the encryption
scheme, the authentication scheme, and the lifetime of the SA. Each ISAKMP line
contains the encryption scheme, hash algorithm, and key lifetime.
The secret definition, then, means that the SA must use the ESP protocol and that a
new encryption key must be created and exchanged every 5 hours (300 minutes). The
SA can support either the Data Encryption Standard (DES) for encryption in
combination with the Hashed Message Authentication Code (HMAC) and Message
Digest 5 (MD5) for authentication, or Triple-DES (3DES) encryption in combination
with either HMAC-MD5 or HMAC-Secure Hash Algorithm (SHA) authentication.
ISAKMP key exchange will use DES encryption and MD5 hashing. The lifetime of an
SA is allowed to be 1 day (1,440 minutes).
The top secret definition is similar to secret except that it only accepts the strongest
encryption and authentication algorithms, namely 3DES and HMAC-SHA.
(Note: DES uses a 56-bit key; 3DES uses a 56-, 112-, or 168-bit key. HMAC-MD5
yields a 96- or 128-bit hash value and SHA yields a 160-bit hash.)

9 - 13
 Security Association Database
Source Destination Tunnel IPsec IPsec Encrypt. Auth.
SPI Address Address Endpoint Protocol Mode Protocol Protocol
12345 10.10.2.0/24 192.168.0.0/16 172.16.12.65 ESP Tunnel 3DES HMAC-MD5
13579 10.10.8.0/24 192.168.0.0/16 172.16.12.65 ESP Tunnel DES HMAC-MD5

10.10.2.0

172.16.12.65 10.20.20.17
Internet 10.10.3.0

Security
IPsec Security
192.168.0.0 Gateway A Gateway B
SA
10.10.8.0
10.10.1.0

10.10.0.0
Source Destination Tunnel IPsec IPsec Encrypt. Auth.
SPI Address Address Endpoint Protocol Mode Protocol Protocol
67890 192.168.0.0/24 10.10.8.0/24 10.20.20.17 ESP Tunnel DES HMAC-MD5

14
IPsec – SANS ©2000 - 2003

The Security Association Database is the set of parameters associated with all
established SAs. Each SAD entry specifies all of the things necessary for IPsec
processing of packets belonging to the SA including IPsec protocol, mode of operation
(discussed on the next few pages), IP addresses, IPsec tunnel endpoint, sequence
number counter, and crypto algorithm information.
SADs are shown in this example for Security Gateways A and B. For inbound
processing, the appropriate SA is found by matching the SPI, destination IP address,
and IPsec protocol type.

9 - 14
 SA Modes

• Transport Mode
– Protects only higher layer information
– Can only be used by IPsec hosts
• Tunnel Mode
– Protects entire packet
– Can be used by hosts or security gateways

IPsec – SANS ©2000 - 2003 15

As alluded to earlier, IPsec supports two modes of operation for SAs, namely
transport mode and tunnel mode. A transport mode SA protects only the higher
layer data in the packets associated with the SA. Transport mode can only be
used between IPsec hosts; security gateways cannot use this mode of operation.
Tunnel mode creates an IPsec tunnel over between two IPsec devices, allowing
IPsec procedures to be applied to the entire IP packet. Either security gateways
or hosts can use tunnel mode.

9 - 15
 IPsec Transport Mode

10.10.2.0
10.10.8.0

172.16.12.65 10.20.20.17 FTP

server
192.168.1.10 Internet
(client) 10.10.8.21
Security Security 10.10.1.0 10.10.3.0
Gateway A Gateway B

192.168.0.0
IP Header 10.10.0.0
Source Destination Encrypted and/or
Address Address Authenticated

10.10.8.21 192.168.1.10 Payload

IPsec – SANS ©2000 - 2003 16

This slide shows an example of transport mode. In this mode, only the payload portion
of IP packets associated with the SA are subject to IPsec procedures.
The example in the slide shows the IPsec host at address 10.10.8.21 sending an IP
packet to the IPsec host at IP address 192.168.1.10 employing a transport mode SA.
The resulting IP packet (shown on the slide) containing the end-host IP source and
destination addresses in the header, while IPsec encryption and/or authentication is
applied to the higher layer data.
Transport mode can only be used between IPsec hosts.

9 - 16
 IPsec Tunnel Mode

10.10.2.0
10.10.8.0

172.16.12.65 10.20.20.17 FTP

server
192.168.1.10 Internet
(client) 10.10.8.21
Security Security 10.10.1.0 10.10.3.0
Gateway A Gateway B

192.168.0.0 10.10.0.0
IPsec
Encrypted and/or Authenticated
IP Header
Original
IP Header
Source Destination Source Destination
Address Address Address Address

10.20.20.17 172.16.12.65 10.10.8.21 192.168.1.10 Payload

IPsec – SANS ©2000 - 2003 17

This slide shows communication between the two IPsec hosts as in the previous slide,
but now using tunnel mode between the two security gateways.
In this example, the sender (the FTP server at 10.10.8.21) prepares an IP packet and
places the end-host IP addresses in the header of the packet. When this packet arrives
at Security Gateway B, the gateway applies IPsec encryption and/or authentication to
the entire packet. The new data entity becomes payload to a new IP packet being sent
from Security Gateway B (at IP address 10.20.20.17) to Security Gateway A (at IP
address 172.16.12.65).
In this case, IPsec employs the tunnel mode SA between the two security gateways.
As the packet travels across the Internet, anyone who happens to look at it obtains no
information about the true source and destination host. When the packet arrives at
Security Gateway A, the IPsec procedures are "undone," the payload unwrapped, and
the original packet retrieved so that it can be delivered to its ultimate destination.
Without getting bogged down too much in detail, it is worth noting that the modes can
be mixed in one host-to-host connection. As an example, the FTP server and client
might use a transport mode SA between them in addition to the tunnel mode SA
between the security gateways. This would protect the entire packet on the Internet, as
described here, and also protect the original payload while the packet is in transit on
the local IP networks.
Tunnel mode can be used by IPsec hosts or gateways; gateways can only use tunnel
mode.

9 - 17
 Internet Key Exchange
• IKE defines rules for SA negotiation
– Dynamic SAs needed for functionality and
security
• IPsec authentication includes pre-
shared keys and PKI certificates
• IKE provides management and safe
exchange of keys
– Employs ISAKMP and Oakley protocols

18
IPsec – SANS ©2000 - 2003

The last piece to the IPsec puzzle has to do with how the IPsec SAs are negotiated. So far we've told
you why they need to be negotiated and what parameters get negotiated, but we haven't yet explored the
how.
IPsec SAs are negotiated using the Internet Key Exchange protocol, described in RFC 2409. IKE
defines the principles of management and key exchange between the gateways and, in fact, other key
exchange mechanisms could well be employed.
While it is true that IPsec SAs could be pre-defined and static, this would not be the right way to build
VPNs. First, it isn't very practical and would destroy any chance of extending IPsec into a dynamic
environment such as e-commerce and the transient trust relationships that they employ. Second, static
SAs would re-use the same data encryption keys and that is just bad cryptography.
Indeed, gateways can employ pre-shared keys or a number of variants of public keys, such as those
employing digital certificates and the public key infrastructure (PKI). Pre-shared keys are used by two
IPsec peers to mutually authenticate so that they can then negotiate the SA parameters, such as choice
of protocol, mode, crypto algorithm, etc.
IKE is based on the functional framework provided by the Internet Security Association and Key
Management Protocol (ISAKMP) defined in RFC 2408. IKE implements a number of key exchange
algorithms, including Oakley (RFC 2412). ISAKMP lays out guidelines for how peer devices can create
a secure communications channel, and provides some rules for peer authentication, key exchange, and
security service negotiation . ISAKMP does not specify how authentication is accomplished nor what
keys are generated; this is left for other protocols.

9 - 18
 IKE Phases and Modes

• Phase 1: ISAKMP peers

establish authenticated,
secure channel using Phase 1:
IKE
preshared keys or
certificates
Phase 2:
– ISAKMP SA IPsec
• Phase 2: Non-ISAKMP
peers negotiate policy
– IPsec SA

IPsec – SANS ©2000 - 2003 19

IKE procedures have two phases. In Phase 1, the two IPsec peers establish a secure
communications channel for IPsec SA negotiation. This communications channel is
called an ISAKMP Security Association and should not be confused with an IPsec SA.
The ISAKMP SA establishment process requires that some initial authentication take
place. One possible method is the exchange of a pre-shared secret between two nodes,
such as a password or passphrase. A second method is the use of public keys, in which
case one node would access the other node's public certificate prior to IKE procedures.
The former method is easier to implement but doesn't scale well.
Once the ISAKMP SA is established, the IPsec peers commence with Phase 2 where
the characteristics of the IPsec SA are negotiated. After the IPsec SA is created, IPsec
protected data exchange can occur. Since the crypto keys have a finite lifetime,
ISAKMP procedures continue to be used to manage the keys.
This is as far as we are going to delve into IPsec's key management process. It has to
be noted, however, that this process is actually quite complex and, many have argued,
more complex than it needs to be in the name of being general, extendible, and
backward compatible. The Phase 1 negotiation can occur in one of two modes, namely
main mode or aggressive mode; the former mode is safe but slow. The Phase 2 process
works is what is called quick mode. By the time you put this all together, it gets rather
involved. At this point, you just need to know how we build up to negotiate the IPsec
SAs.
ISAKMP messages are carried in UDP datagrams using port 500.

9 - 19
 IPsec Scenarios

10.10.2.0
10.10.8.0

172.16.12.65 10.20.20.17 FTP

192.168.1.1 10.10.8.1 server
192.168.1.10 Internet
(client) 10.10.8.21
Security Security 10.10.1.0 10.10.3.0
Gateway A
Gateway B

192.168.0.0 • Client connection with server at 10.10.1.10

– Bypass IPsec
10.10.1.10
• Client connection with server at 10.10.8.21 FTP
server
– Authentication Header
– Encapsulating Security Payload 10.10.0.0

20
IPsec – SANS ©2000 - 2003

After all of this preamble (and believe me, it was necessary!), we are now going to
examine in detail three communication scenarios.
We will first look at some packet exchange between the client system at IP address
192.168.1.10 and the FTP server at 10.10.1.10. Since the SPD indicates that packets to
the 10.10.1.0 subnet will bypass IPsec, we will just see a typical FTP exchange. We do
this to use something familiar to then compare with IPsec!
The next two scenarios are between the client and the FTP server at 10.10.8.21, which
requires IPsec processing. We will first describe, and then show a detailed example of,
the Authentication Header protocol followed by a description and detailed example of
the Encapsulating Security Payload protocol.

9 - 20
 Bypass IPsec – FTP Exchange
192.168.1.10.1035 > 10.10.1.10.ftp: P 25:41(16) ack 31 win 8602 (DF)
[tos 0x1f] (ttl 128, id 14849)

10.10.1.10.ftp > 192.168.1.10.1035: P 31:96(65) ack 41 win 8684

(ttl 126, id 54794)

10.10.1.10.ftp-data > 192.168.1.10.1036: S 522326774:522326774(0) win 8192

<mss 1460> (ttl 126, id 55050)

192.168.1.10.1036 > 10.10.1.10.ftp-data: S 126568:126568(0) ack 522326775

win 8760 <mss 1460> (DF) (ttl 128, id 15105)

10.10.1.10.ftp-data > 192.168.1.10.1036: . ack 1 win 8760 (ttl 126, id 55306)

10.10.1.10.ftp-data > 192.168.1.10.1036: P 1:12(11) ack 1 win 8760

(ttl 126, id 55562)

10.10.1.10.ftp-data > 192.168.1.10.1036: F 12:12(0) ack 1 win 8760

(ttl 126, id 55818)

21
IPsec – SANS ©2000 - 2003

This slide shows a simple FTP exchange between the FTP server at 10.10.1.10 and an
FTP client at 192.168.1.10. Since the security policy for packets going to the 10.10.1.0
subnet is to "bypass IPsec," there is no IPsec processing on these packets. What we see
here, then, is pretty normal traffic. The top of the tcpdump trace shows us in the
middle of an FTP connection and then we see the server initiate a connection back to
the client for FTP data transfer.

(The reader is asked for a moment to forget that we showed a policy earlier for the
192.168.1.0 network that said that all inbound packets to that network had to be
authenticated and encrypted. That policy, obviously, is not in effect right now!)

9 - 21
 Bypass IPsec – FTP Exchange
The Whole Packet
Internet Protocol
Version: 4
Header length: 20 bytes (5 32-bit words)
Service type: 0x00 (Precd=Routine,Delay=Normal,Thrput=Normal,Reli=Normal)
Total length: 51 bytes
Fragment ID: 41476
Flags: 0x0000 (May be fragmented,Last fragment,Offset=0)
Time to live: 126 seconds/hops
Protocol ID: TCP
Checksum: 0x7032
Addresses: 10.10.1.10 --> 192.168.1.10
Transmission Control Protocol
Port File Transfer (Default Data) ---> 1034
Sequence Number: 3904510
Acknowledgement Number: 266981
Header Length: 5 (32-bit word)
Flags: ACK,PSH,
Window: 8760 bytes
Checksum: 0x5f39
File Transfer Protocol
hello world

IN HEXADECIMAL
0000: 45 00 00 33 bc 01 00 00 7e 06 70 32 0a 0a 01 0a | E..3....~.p2....
0010: c0 a8 01 0a 00 14 04 0a 00 3b 93 fe 00 04 12 e5 | .........;......
0020: 50 18 22 38 5f e9 00 00 68 65 6c 6c 6f 20 77 6f | P."8_...hello wo
0030: 72 6c 64 | rld

22
IPsec – SANS ©2000 - 2003

This slide shows the decode of a single IP packet containing the string "hello world".
Although somewhat abbreviated, you will note that all of the IP, TCP, and FTP
information that is displayed is obtainable from the hex dump below, which shows the
entire IP packet.
Do note that this packet is 51 bytes in length; 20 bytes for the IP header, 20 bytes for
the TCP header, and 11 bytes for the data characters.

9 - 22
 Building an SA
Authentication (LDAP)
10.20.20.17.1038 > 172.16.12.65.389: S 395784:395784(0) win...
172.16.12.65.389 > 10.20.20.17.1038: S 1757781809:1757781809(0)
ack 395785 win...

Key and security parameters exchanges (ISAKMP)

10.20.20.17.500 > 172.16.12.65.500: udp 990 (ttl 128, id 37896)
10.20.20.17.500 > 172.16.12.65.500: udp 92 (ttl 128, id 38152)

IPsec traffic (ESP)

10.20.20.17 > 172.16.12.65: ip-proto-50 132 (ttl 128, id 32522)
10.20.20.17 > 172.16.12.65: ip-proto-50 132 (ttl 128, id 32778)

IPsec – SANS ©2000 - 2003 23

Before we get into the details of IPsec-protected packets, we need to at least mention
some of the details that we will not be discussing in this section. Before exchanging
IPsec AH or ESP packets, the peer IPsec devices need to establish an SA. This process
is introduced in the slide by the set of packets shown above, which is divided into
three functional parts. In this example, we are setting up an SA from Security Gateway
B (10.20.20.17) to Security Gateway A (172.16.12.65).
The first action is that of authentication, where we see the beginning of a typical TCP
3-way handshake. In this case, Gateway B is connecting to TCP port 389 (LDAP) on
Gateway A. LDAP may be used as part of the authentication activity occurring
between two IPsec nodes and an X.500 directory service part of a Public Key
Infrastructure. This corresponds to the primary authentication services which must
take place before any IPsec process can begin. Authentication could also be something
as simple as pre-shared keys, manual key exchange via e-mail, a face-to-face paper
exchange, etc.
The next couple of packets employ UDP port number 500, indicating use of the
ISAKMP protocol for key exchange. A detailed examination of this protocol is well
beyond the scope of what we will cover here, but ISAKMP packets would be
exchanged if IPsec automatic key exchange protocols were invoked. These packets
correspond to the IPsec peer negotiation regarding security policies, encryption keys,
and other necessary SA parameters.
Finally, we see the exchange of IPsec packets using the ESP (IP protocol 50) once the
SA has been established.

9 - 23
 SA Processing

• Outbound
– Check SPD to see if IPsec required
– Check SAD to determine IPsec requirements
– Assemble packet using SPI, appropriate mode,
and crypto algorithms
• Inbound
– Packet Header indicates use of IPsec
– Check SAD for match with destination, protocol,
and SPI
– SAD entry tells node how to interpret the packet

24
IPsec – SANS ©2000 - 2003

Once an SA has been established, IPsec nodes have to determine when and how
to use the SA on a per-packet basis.
When an IPsec node wants to send a packet to some destination IP address, it
looks into the SPD to determine if any IPsec protection is required. Assuming
that it is, the node checks the SAD to see if there is an existing SA and what the
requirements are. Assuming that the SA exists, the node will find the SPI , the
IPsec protocol to apply, the mode, and the algorithms to use. That will
determine how the outbound packet is assembled.
When a packet arrives at an IPsec node, the use of AH or ESP is noted in the IP
Header. The node can then extract the SPI from the packet (it is, obviously,
never encrypted). Using the destination IP address, IPsec protocol type, and
SPI, the node can check the SAD to see whether this packet is associated with
an active connection. If so, the node can now determine the mode and
algorithms employed so that it can interpret the packet; if there is no SA, the
packet is discarded.
One final point. AH and ESP processing should be applied to entire packets
rather than fragments. If an IPsec-processed packet is fragmented in transit, the
security gateway will have to reassemble the packet prior to processing with
either AH or ESP procedures.

9 - 24
 Authentication Header

• Provides per-packet authentication for the IP

payload
– Data integrity
– Data origin authentication
– Anti-replay protection
• AH also covers immutable fields in the IP
header
– This includes the addresses which is why AH and
NAT don't get along!

IPsec – SANS ©2000 - 2003 25

The IPsec Authentication Header, described in RFC 2402, is used to provide authentication for
the IP packet payload on a per-packet basis. As such, AH offers data integrity and data origin
authentication for the higher layer header and data being carried in an IP packet. AH, then, can
be used to assure that the data really comes from the IP host that ostensibly sent it and that the
data has not been altered in transit.
AH can also provides anti-replay protection, guarding against a packet being resent at a later
time. Note that AH provide no privacy mechanisms.
AH also covers as much of the IP packet header as possible, meaning that it will provide these
same authentication and integrity protections to all of the immutable fields of the header.
Remember that several fields within the IP packet header will change at each hop during transit;
namely, TOS, Fragment Offset, flags, TTL, and Checksum. These fields are mutable, or
changeable. The remaining fields do not change from end-to-end and, therefore, can be
protected by the AH mechanism.
(Think of the difficulty in using NAT with AH. NAT, by its nature, changes one or both IP
addresses at some point along the way. The AH integrity value, however, is calculated using the
original addresses and assumes that the addresses do not change. And if addresses were allowed
to change, consider the vulnerability to attack.)
The use of AH is determined when the SA is negotiated between two IPsec peer nodes. In the
security descriptors, there will be a section describing which parameters to use with AH, such
as the authentication algorithms (e.g., HMAC-MD5 or HMAC-SHA1). The use of tunnel or
transport mode will also be negotiated at that time.
For the duration of the SA, every IP packet will be modified to include an authentication header
in the packet. The header format and placement within the packet are the subject of the next
several pages.

9 - 25
 AH Header

1111111111222222222233
01234567890123456789012345678901
Next Header Payload Len. Reserved
Security Parameters Index
Sequence Number

Authentication Data (variable)

IPsec – SANS ©2000 - 2003 26

This slide shows the format of the IPsec authentication header. As shown, the header
has six fields:
• Next Header: This 8-bit field identifies the protocol header that follows AH, or the
higher layer data carried in this packet. This is the functional equivalent to the IP
Protocol Identifier (PID) field in IPv4, but Next Header is the IPv6 nomenclature. This
field uses the same values as IPv4's PID field, such as 0x04 (IP), 0x06 (TCP), or 0x11
(UDP).
• Payload Length: An 8-bit field that specifies the length of the Authentication Header
(not the payload, as the name implies). The value is expressed as the number of 32-bit
words minus 2 (the rationale for this is for compatibility with IPv6).
• Reserved: This field is 16 bits and filled with zeros.
• Security Parameter Index (SPI): This random 32-bit value is used with the destination
IP address and IPsec protocol (AH in this case) to uniquely identify the SA to which
this packet is associated. The SPI is generally selected by the destination IPsec node.
• Sequence Number: A 32-bit sequence number for the IPsec packets. This value is
initialized to zero and is incremented by one with each new packet sent using this SA.
This monotonically increasing sequence number is the AH anti-replay mechanism.
• Authentication Data: A variable-length field that contains the Integrity Check Value
(ICV) for this packet. The length of the ICV must be an integral multiple of 32 bits; if it
is of a different length, the ICV value will either be padded or truncated to meet this
requirement.

9 - 26
 AH in Tunnel & Transport Modes

AH Transport Mode

Original IP Packet Original Authentication Upper Layer Upper Layer

IP Header Header Header Data
Payload
Authenticated
Original Upper Layer Upper Layer
IP Header Header Data
AH Tunnel Mode

Encapsulation
New Authentication Original Upper Layer Upper Layer
IP Header Header IP Header Header Data

Authenticated

IPsec – SANS ©2000 - 2003 27

This slide shows the use of AH in transport and tunnel modes. Remember that the
choice of mode is negotiated when the SA is established.
When AH is used in transport mode, it adds a few additional bytes of information to
the IP packet. As shown, the original IP header remains intact, with the notable change
that the PID field contains a value of 51 (0x33). The AH is inserted between the IP
Header and the original upper layer header, and the ICV in the AH covers the entire
packet (except for the mutable fields discussed earlier). The AH's Next Header field
would specify the Upper Layer Header (e.g., TCP or UDP).
Recall that tunnel mode views the original IP packet as payload to a new IP packet.
The "inner" IP Header contains the addresses of peer hosts while the "outer" IP Header
contains the addresses of the peer IPsec nodes. In this case, the AH is inserted between
the new IP Header and the original IP packet, with the ICV again covering the entire
packet, and the AH Next Header field would indicate IP (0x04).
Assembling and disassembling these packets follows the steps outlined earlier. A SAD
entry tells the node what SPI to use as well as what mode and crypto protocols to
employ. Upon receipt, the node sees use of AH, can extract an SPI, and do a SAD
lookup. The SAD entry will then tell the receiver what mode and crypto algorithms are
being employed so that the packet can be disassembled.

9 - 27
 Authentication Header
Tcpdump Trace

10.20.20.17 > 172.16.12.65: ip-proto-51 95 (DF) [tos 0x4] (ttl 128, id 65281)
172.16.12.65 > 10.20.20.17: ip-proto-51 94 (DF) (ttl 127, id 16135)
10.20.20.17 > 172.16.12.65: ip-proto-51 80 (DF) [tos 0x4] (ttl 128, id 2)
172.16.12.65 > 10.20.20.17: ip-proto-51 129 (DF) (ttl 127, id 16391)
172.16.12.65 > 10.20.20.17: ip-proto-51 68 (DF) (ttl 127, id 16647)
10.20.20.17 > 172.16.12.65: ip-proto-51 68 (DF) (ttl 128, id 258)
172.16.12.65 > 10.20.20.17: ip-proto-51 64 (DF) (ttl 127, id 16903)
172.16.12.65 > 10.20.20.17: ip-proto-51 75 (DF) (ttl 127, id 17159)
172.16.12.65 > 10.20.20.17: ip-proto-51 64 (DF) (ttl 127, id 17415)
10.20.20.17 > 172.16.12.65: ip-proto-51 64 (DF) (ttl 128, id 514)
10.20.20.17 > 172.16.12.65: ip-proto-51 64 (DF) (ttl 128, id 770)
172.16.12.65 > 10.20.20.17: ip-proto-51 64 (DF) (ttl 127, id 17671)
10.20.20.17 > 172.16.12.65: ip-proto-51 64 (DF) [tos 0x4] (ttl 128, id 1026)
172.16.12.65 > 10.20.20.17: ip-proto-51 88 (DF) (ttl 127, id 17927)
10.20.20.17 > 172.16.12.65: ip-proto-51 64 (DF) [tos 0x4] (ttl 128, id 1282)

IPsec – SANS ©2000 - 2003 28

So now let's put this all together. This slide shows tcpdump output that might be picked
up somewhere on the Internet in our reference network. Since the source and
destination addresses are those of the two security gateways, this packet trace suggests
that this is a tunneled SA. Use of IP protocol 51 tells us that this is AH traffic.
If we take a close look at the first line, we see:
ip-proto-51 95 (DF) [tos 0x4] (ttl 128, id 65281)

Again, protocol number 51 is AH and the IP packet is 95 bytes in length. We also see a
few of the usual IP header fields, such as the TOS, TTL, and packet identifier, but there
is clearly not as much information as we are used to seeing in a tcpdump trace.

9 - 28
 Authentication Header
The Whole Packet 1 (Outer IP)
Internet protocol
Version: 4
Header length: 20 bytes (5 32-bit words)
Service type: 0x04 (Precd=Routine,Delay=Normal,Thrput=Normal,Reli=High)
Total length: 95 bytes
Fragment ID: 65281
Flags: 0x4000 (May not be fragmented,Last fragment,Offset=0)
Time to live: 128 seconds/hops
Protocol ID: unknown (0x33)
Checksum: 0x9021
Addresses: 10.20.20.17 --> 172.16.12.65
No option

IN HEXADECIMAL
0000: 45 04 00 5f ff 01 40 00 80 33 90 21 0a 14 14 11 | E.._..@..3.!....
0010: ac 10 0c 41 04 04 00 00 00 01 09 32 00 00 00 1e | ...A.......2....
0020: b1 0e ea ff 9f c3 57 b3 bd 8d a5 fb 45 00 00 33 | +....AW./..UE..3
0030: a2 04 40 00 7f 06 02 45 0a 0a 08 15 c0 a8 01 0a | ..@....E........
0040: 00 14 04 13 00 70 43 a6 00 07 b5 33 50 18 20 70 | .....Pc....3P. P
0050: c8 8f 00 00 68 65 6c 6c 6f 20 77 6f 72 6c 64 | ._..hello world

IPsec – SANS ©2000 - 2003 29

This slide starts a detailed examination of an FTP message used to transfer the string
"hello world" that has been protected using IPsec AH in tunnel mode. In the slide
above, the hex output shows the entire IP packet captured from somewhere on the
Internet.
The detailed listing above is an interpretation of the IP Header found in the packet.
There's nothing really unexpected here; this is a standard IPv4 packet with a pretty
standard 20-byte Header (this is the "new IP Header"). The only notable thing here is
that the Protocol Identifier refers to protocol number 51 (0x33), or AH.
Do note the packet length of 95 bytes. Remember that our original IP packet showing
an FTP transfer of this string was 51 bytes in length. We have added here a 24-byte
AH header plus our 20-byte outer IP Header.

9 - 29
 Authentication Header
The Whole Packet 2 (AH)
Authentication Header
Next Header: 0x04 (IP)
Payload Length: 0x04 (24 bytes)
Reserved: 0x0000
Security Parameters Index: 67890
Sequence Number: 30
Authentication Data: b1 0e ea ff 9f c3 57 b3 bd 8d a5 fb

IN HEXADECIMAL
0000: 45 00 00 5f a2 04 40 00 7f 33 90 21 0a 14 14 11 | E.._..@..3.!....
0010: ac 10 0c 41 04 04 00 00 00 01 09 32 00 00 00 1e | ...A.......2....
0020: b1 0e ea ff 9f c3 57 b3 bd 8d a5 fb 45 00 00 33 | +....AW./..UE..3
0030: a2 04 40 00 7f 06 02 45 0a 0a 08 15 c0 a8 01 0a | ..@....E........
0040: 00 14 04 13 00 70 43 a6 00 07 b5 33 50 18 20 70 | .....Pc....3P. P
0050: c8 8f 00 00 68 65 6c 6c 6f 20 77 6f 72 6c 64 | ._..hello world

IPsec – SANS ©2000 - 2003 30

This slide now peeks inside of the AH header itself. At this point, the receiving node
knows that this is AH (we will assume that we've already verified that this is a valid
SA).
The first byte of the header (0x04) tells us that the higher layer payload is IP; this is
expected because we already know that this SA uses tunnel mode.
The second byte of the header (0x04) tells us that the AH header is 24 bytes in length.
We arrive at this by adding 2 to the length value (to get 32-bit words) and then
multiplying by 4 (to get the number of bytes).
The next two bytes (0x0000) are reserved and set to 0.
The next four bytes (0x0010932) are the SPI, 67890. It is this value (along with the
destination IP address and protocol number) that tells the receiving IPsec node to use
tunnel mode, etc.
The next four bytes (0x0000001e) is the sequence number (30), which suggests that
the SA has been up for some time since the sequence number always starts at 0.
The remaining 12 bytes 96 bits of the AH are the ICV authentication data.
We have now looked in detail at the first 44 bytes of this packet. The remaining 51
bytes are nearly identical to the original IP packet that we saw previously when IPsec
procedures were bypassed.

9 - 30
 Encapsulating Security Payload

• Provides authentication & confidentiality

methods for IP packets
– Confidentiality uses encryption
– ESP normally employs both encryption and
authentication although both are optional
• ESP can also provide replay protection

IPsec – SANS ©2000 - 2003 31

The second IPsec security protocol is the Encapsulating Security Payload,

described in RFC 2406. ESP's services are, in some ways, broader than those
offered by AH; ESP provides authentication, confidentiality, and (optional)
replay attack prevention. Encryption is used for data confidentiality.
It is interesting to note that both confidentiality and authentication are optional
in ESP; in fact, ESP could be used with neither although that would seem to
obviate the reason to use ESP in the first place! In addition, most applications
requiring confidentiality should also employ authentication to prevent certain
types of attacks.

9 - 31
 ESP Packet

1111111111222222222233
01234567890123456789012345678901
Security Parameters Index ESP
Sequence Number Header

ESP
Payload Data (variable)
Payload

Padding (0-255 bytes) ESP

Pad length Next Header Trailer

Authentication Data (variable) ESP

Authentication

32
IPsec – SANS ©2000 - 2003

This slide shows the ESP packet format. Note that AH supplies just a new data header while ESP
employs a new packet format, complete with a header and trailer. Use of ESP is indicated by placing a
value of 50 (0x32) in the IPv4 PID or IPv6 Next Header field. The fields of the ESP packet are:
• ESP Header
• Security Parameters Index: As described for the AH, a 32-bit value to identify the SA to which
this packet is associated.
• Sequence Number: As described for the AH, a 32-bit value, initially set to 0, used by the
receiver to protect against replay attacks.
• ESP Payload
•Payload Data: A variable-length field containing the data to be protected by the ESP protocol;
i.e., the original IP packet or the upper layer information. The format of the data contained in
this field is indicated by the value in the Next Header field.
• ESP Trailer
• Padding: A 0-255 byte filed used for a variety of purposes. It is primarily used to ensure that
the Payload, Pad Length, and Next Header align on a 32-bit boundary. It can also be used if the
ESP encryption algorithm requires a certain minimum number of bytes. Finally, it may be used
to hide the real size of the payload, providing partial protection against traffic flow analysis.
• Pad Length: 8-bit value indicating the number of Pad bytes that were inserted.
• Next Header: As described for the AH, an 8-bit field identifying the type of Payload
transmitted.
• ESP Authentication
• Authentication Data: A variable-length field containing an Integrity Check Value computed
over the entire ESP packet (except for this field). The length of this field is dependent upon the
authentication function that is used. This field is present only if an authentication service is
being employed in this SA.

9 - 32
 ESP in Tunnel & Transport Modes

ESP Transport Mode

Original IP Packet
Original ESP Upper Layer Upper Layer ESP ESP
Payload IP Header Header Header Data Trailer Auth.

Original Upper Layer Upper Layer Encrypted

IP Header Header Data
Authenticated

ESP Tunnel Mode

Encapsulation

New ESP Original Upper Layer Upper Layer ESP ESP

IP Header Header IP Header Header Data Trailer Auth.

Encrypted
Authenticated

IPsec – SANS ©2000 - 2003 33

This slide shows the use of ESP in transport and tunnel modes. Again, remember that
the choice of mode is negotiated when the SA is established.
When ESP is used in transport mode, we see that the original IP header remains intact,
with the notable change that the PID field contains a value of 50 (0x32). The original
upper layer header and data is payload to the ESP packet; we see the ESP header
inserted after the IP packet header plus the ESP trailer and authentication fields. The
ESP Header's Next Header field would indicate the Upper Layer Header (e.g., TCP or
UDP).
Tunnel mode, of course, views the entire original IP packet as payload to this new
ESP packet. In this case, the original IP packet is placed in the ESP packet's Payload
field and the ESP Next Header field would indicate IP (0x04). A new IP packet header
is generated to move the packet between security gateways.
The ESP packet payload and trailer are encrypted using the negotiated encryption
scheme (or NULL encryption if confidentiality is not needed). The ESP authentication
information, covering the ESP header, payload, and trailer, is present only if
authentication has been requested for this SA.
Assembling and disassembling these packets follows the steps outlined earlier. A SAD
entry tells the node what SPI to use as well as what mode and crypto protocols to
employ. Upon receipt, the node sees use of AH, can extract an SPI, and do a SAD
lookup. The SAD entry will then tell the receiver what mode and crypto algorithms are
being employed so that the packet can be disassembled.

9 - 33
 Encapsulating Security Payload
Tcpdump Trace
10.20.20.17 > 172.16.12.65: ip-proto-50
172.16.12.65 > 10.20.20.17: ip-proto-50
10.20.20.17 > 172.16.12.65: ip-proto-50
172.16.12.65 > 10.20.20.17: ip-proto-50
172.16.12.65 > 10.20.20.17: ip-proto-50
10.20.20.17 > 172.16.12.65: ip-proto-50
172.16.12.65 > 10.20.20.17: ip-proto-50
172.16.12.65 > 10.20.20.17: ip-proto-50
10.20.20.17 > 172.16.12.65: ip-proto-50
10.20.20.17 > 172.16.12.65: ip-proto-50
172.16.12.65 > 10.20.20.17: ip-proto-50
172.16.12.65 > 10.20.20.17: ip-proto-50
10.20.20.17 > 172.16.12.65: ip-proto-50
10.20.20.17 > 172.16.12.65: ip-proto-50
172.16.12.65 > 10.20.20.17: ip-proto-50
10.20.20.17 > 172.16.12.65: ip-proto-50
100 (DF) [tos 0x4] (ttl 128, id 52740)
88 (DF) (ttl 127, id 37383)
80 (DF) [tos 0x4] (ttl 128, id 52996)
128 (DF) (ttl 127, id 37639)
64 (DF) (ttl 127, id 37895)
64 (DF) (ttl 128, id 53252)
64 (DF) (ttl 127, id 38151)
72 (DF) (ttl 127, id 38407)
64 (DF) (ttl 128, id 53508)
64 (DF) [tos 0x4] (ttl 128, id 53764)
64 (DF) (ttl 127, id 38663)
88 (DF) (ttl 127, id 38919)
64 (DF) (ttl 128, id 54020)
64 (DF) (ttl 128, id 54276)
64 (DF) (ttl 127, id 39175)
64 (DF) [tos 0x4] (ttl 128, id 54532)

IPsec – SANS ©2000 - 2003 34

So now, once again, let's put this all together. This slide shows tcpdump output that
might be picked up somewhere on the Internet in our reference network. This trace
looks pretty much the same as the earlier AH traffic with the obvious difference being
the IP protocol type, which is now 50. Use of tunnel mode is again visible by the use
of the security gateways' IP addresses instead of the addresses of the FTP client and
server.

9 - 34
 Encapsulating Security Payload
The Whole Packet 1 (Outer IP)
Internet Protocol
Version: 4
Header length: 20 bytes (5 32-bit words)
Service type: 0x04 (Precd=Routine,Delay=Normal,Thrput=Normal,Reli=High)
Total length: 100 bytes
Fragment ID: 52740
Flags summary: 0x4000 (May not be fragmented,Last fragment,Offset=0)
Time to live: 128 seconds/hops
IP protocol type: Unknown (0x32)
Checksum: 0x591C
IP address: 10.20.20.17 --> 172.16.12.65
No option

IN HEXADECIMAL
0000: 45 04 00 64 ce 04 40 00 80 32 59 1c 0a 14 14 11 | E..d..@..2Y.....
0010: ac 10 0c 41 00 01 09 32 00 00 00 27 0e 0e 4b d8 | ...E...2...'..K.
0020: 0e 42 94 60 fa 06 8d 5c 33 a3 02 84 f0 2b 6b f3 | .B.`...\3.._.+k.
0030: 5a f1 72 08 f8 7f b5 4e 73 ce b4 9f e7 f6 60 fa | Z.r....Ns.....`.
0040: 19 a7 e6 96 38 9f 93 d6 91 bc 18 50 45 5d ed 28 | ....8......PE].(
0050: 66 ff 90 83 a6 b4 b6 71 98 ab c6 33 26 ac 1b 98 | f._....q...3&...
0060: 60 e3 b1 ab | `.+.

IPsec – SANS ©2000 - 2003 35

This slide starts a detailed examination of the FTP message used to transfer the string
"hello world" that has been protected using IPsec ESP in tunnel mode. In the slide
above, the hex output shows the entire IP packet captured from somewhere on the
Internet.
As before, the detailed listing above is an interpretation of the IP Header found in the
packet. There's nothing unexpected here; this is a standard IPv4 packet with a standard
20-byte Header. The Protocol Identifier refers to protocol number 50 (0x32), or ESP.
Note the packet length of 100 bytes. Our original IP packet showing an FTP transfer
of this string was 51 bytes in length. Given that we have a 20-byte IP Header, the ESP
packet must be 80 bytes in length.

9 - 35
 Encapsulating Security Payload
The Whole Packet 2 (ESP Header)
ESP Header
Security Parameters Index: 67890
Sequence Number: 39

IN HEXADECIMAL
0000: 45 04 00 64 d8 01 40 00 7f 32 59 1c 0a 14 14 11 | E..d..@..2Y.....
0010: ac 10 0c 41 00 01 09 32 00 00 00 27 0e 0e 4b d8 | ...E...2...'..K.
0020: 0e 42 94 60 fa 06 8d 5c 33 a3 02 84 f0 2b 6b f3 | .B.`...\3.._.+k.
0030: 5a f1 72 08 f8 7f b5 4e 73 ce b4 9f e7 f6 60 fa | Z.r....Ns.....`.
0040: 19 a7 e6 96 38 9f 93 d6 91 bc 18 50 45 5d ed 28 | ....8......PE].(
0050: 66 ff 90 83 a6 b4 b6 71 98 ab c6 33 26 ac 1b 98 | f._....q...3&...
0060: 60 e3 b1 ab | `.+.

IPsec – SANS ©2000 - 2003 36

This slide peeks inside of the ESP packet itself.

The first four bytes (0x0010932) are the SPI, 67890. It is this value (along with the
destination IP address and protocol number) that tells the receiving IPsec node to use
tunnel mode, etc.
The next four bytes (0x00000027) represent the sequence number (39), which suggests
that the SA has been up for some time since the sequence number always starts at 0.
And we can't quickly read the remaining 72 bytes; recall that the ESP Payload and
Trailer are encrypted!
Assuming the use of authentication, and further assuming that the ICV is 96 bits, we
can determine the composition of this 100-byte IP packet:
• IP Header (20)
• ESP Packet (80)
• ESP Header (8)
• ESP Payload (51; the original IP packet!)
• ESP Trailer (9)
• Padding (7)
• Pad Length (1)
• Next Header (1)
•ESP Authentication (12)
Yes, we can determine the composition of the packet... but we still can't read it!!

9 - 36
 Wrap-Up
10.10.2.0

172.16.12.65 10.20.20.17
Internet 10.10.3.0

Security Security
Gateway A IPsec
192.168.0.0 Gateway B
Tunnel
10.10.8.0
10.10.1.0

10.10.0.0

Security Gateway A Security Gateway B

Secret : abcdefg Secret : abcdefg

Policies: Policies:
Local 192.168.0.0, ESP, DES, HMAC-MD5 Local 10.0.0.0, ESP, DES, HMAC-MD5
Remote 10.10.0.0, tunnel security gateway B Remote 192.168.0.0, tunnel security gateway A,

37
IPsec – SANS ©2000 - 2003

To conclude this class, turn to slide “IPsec Step-by-Step”. Let’s summarize the information presented
throughout the slides by going step-by-step over how IPsec is established between the two security
gateways to secure traffic between network 10.0.0.0 and 192.168.0.0.

First, there is the preparation. Prior to everything, Security Gateways A and B need to prepare some
authentication. It can be done by the exchange of secrets (abcdefg on the slide) or the use of PKI
certificates. Then, the Security Associations’ security policies must be created for the IPsec
implementations. Second, the creation of Security Associations between the two gateways are initiated
by the Internet Key Exchange protocol. Initially, a secure negotiation channel is established using the
Oakley Main or Aggressive mode. During this phase, the secrets of the gateways are compared to
verify each other’s identities.

After that, using this secure channel and the Oakley quick mode, the Security Associations are created.
The two security gateways decide which authentication and/or encryption algorithm to use, which
key(s) to use and for how long before they exchange a new key. As shown on the slide, the gateways
will use ESP with 3DES for encryption and HMAC-MD5 for authentication . When this step is
completed, there are active Security Associations on both gateways resulting in an IPsec connection
between the gateways.

For the duration of the connection, IP packets will be secured by ESP for encryption and authentication.
IPsec protects the network layer, not just an application or a service. It secures each IP packet. It is
based on four elements, three protocols: IKE, ESP and AH, and one concept, the Security Associations.
The combination of all these elements create flexible and secure VPNs.

9 - 37
 IPsec Configuration
crypto isakmp policy 1
hash md5
authentication pre-share
lifetime 86400
crypto isakmp key abcdefg address 10.20.20.17

crypto ipsec transform-set secret ah-md5-hmac esp-des

crypto map sans-ipsec 10 ipsec-isakmp

set peer 10.20.20.17
set security-association lifetime seconds 86400
set transform-set secret
match address 101

interface serial 0
ip address 172.16.12.65 255.255.255.252
crypto map sans-ipsec

access-list 101 permit icmp host 172.16.12.65 host 10.20.20.17

access-list 101 permit ip host 172.16.12.65 host 10.20.20.17

38
IPsec – SANS ©2000 - 2003

Just to demonstrate some reality, the slide shows the configuration of Security Gateway A to create an
IPsec tunnel mode VPN using pre-shared keys and ESP applied to all IP and ICMP. The example
assumes a Cisco router using IOS.
There are three things that we need to do to set up this IPsec VPN. First, we need to determine how we
will exchange keys; in this case, they will be pre-shared (which makes sense for a 2-node VPN but
rapidly loses its appeal as the number of nodes grows). Second, we need to configure IKE for SA
negotiation. Finally, we need to configure IPsec.
In the first block of commands, we set the policy around key exchange. Here we define that the pre-
shared key will be abcdefg and shared with Security Gateway B (10.20.20.17). The lifetime of SAs is
set to 86,400 seconds (1 day). (We use pre-shared keys in this example because it is simple and still the
prevalent ways of building IPsec tunnels today.)
The crypto ipsec transform-set command specifies the type of authentication and encryption
protocols; recall that secret was the name we gave the policy that specified HMAC-MD5 and DES,
respectively. We also set the SA lifetime here although it isn't really necessary; the setting here will
override the setting in the isakmp policy block above. This command block also specifies what
traffic is subject to this IPsec tunnel by referencing access list 101. Access list 101 indicates that IP
packets and ICMP messages between Security Gateway A (172.16.12.65) and Security Gateway B
(10.20.20.17) are subject to IPsec processing. Access lists for IPsec purposes should be interpreted
differently than access lists for the purpose of packet filtering; in the IPsec case, allow means to apply
IPsec protection and deny means to not apply IPsec protection.
The final block of commands above assigns an IP address to the router's serial (WAN) port. It also
assigns the IPsec rules to that port.

9 - 38
 IPsec Terms and Concepts

• Security Association
– Security Policy Database
– Security Association Database
– Tunnel mode and Transport mode
• Internet Key Exchange
• IPsec protocols
– Authentication Header (AH)
– Encapsulating Security Payload (ESP)

IPsec – SANS ©2000 - 2003 39

By way of summary, we have taken a tour of IPsec and its procedures. For our
discussion, we have used IPsec to provide secure communication -- a VPN! --
between the two security gateways connecting networks 10.10.0.0 and
192.168.0.0.
Before any action can take place, the two gateways need to mutually
authenticate. We elected to use a pre-shared key although other methods, such
as digital certificates, could have also been used.
Using IKE, the gateways then establish the SA in accordance with the policies
implemented at each site (and maintained at each gateway's SPD). This is when
the gateways negotiate the IPsec protocol (AH or ESP), mode (tunnel or
transport), and authentication and encryption protocols (e.g., HMAC with MD5
or SHA; DES or 3DES). When this step is completed, the new active SA is
added to the SAD on both gateways.

9 - 39
 Glossary and Definitions (1)
Access Control A security service that prevents unauthorized use of a resource, including the prevention of use of a
resource in an unauthorized manner. In the IPsec context, the resource to which access is being controlled is often: o for a
host, computing cycles or data o for a security gateway, a network behind the gateway or bandwidth on that network.
Anti-replay [See "Integrity" below]
Asymmetric cryptography A form of cryptography where encryption is done with one key and decryption is done with a
second key. The two keys are mathematically related but knowledge of one key does not yield knowledge of the other key so
that one key can be widely distributed (the public key) and the other kept a secret (the private key).
Authentication This term is used informally to refer to the combination of two nominally distinct security services, data origin
authentication and connectionless integrity. See the definitions below for each of these services. Availability Availability, when
viewed as a security service, addresses the security concerns engendered by attacks against networks that deny or degrade
service. For example, in the IPsec context, the use of anti-replay mechanisms in AH and ESP support availability.
Confidentiality Confidentiality is the security service that protects data from unauthorized disclosure. The primary
confidentiality concern in most instances is unauthorized disclosure of application level data, but disclosure of the external
characteristics of communication also can be a concern in some circumstances. Traffic flow confidentiality is the service that
addresses this latter concern by concealing source and destination addresses, message length, or frequency of
communication. In the IPsec context, using ESP in tunnel mode, especially at a security gateway, can provide some level of
traffic flow confidentiality. (See also traffic analysis, below.)
Data Origin Authentication Data origin authentication is a security service that verifies the identity of the claimed source of
data. This service is usually bundled with connectionless integrity service.
Encryption Encryption is a security mechanism used to transform data from an intelligible form (plaintext) into an
unintelligible form (ciphertext), to provide confidentiality. The inverse transformation process is designated "decryption".
Oftimes the term "encryption" is used to generically refer to both processes.

IPsec – SANS ©2000 - 2003 40

This page intentionally left blank.

9 - 40
 Glossary and Definitions (2)
Hash function A form of one-way cryptography where the original data is mathematically transformed in such a
way so that the original data, as well as the length of the original data, is unrecoverable from the hash value..
Integrity Integrity is a security service that ensures that modifications to data are detectable. Integrity comes in
various flavors to match application requirements. IPsec supports two forms of integrity: connectionless and a
form of partial sequence integrity. Connectionless integrity is a service that detects modification of an individual
IP datagram across a connectionless network, without regard to the ordering of the datagram in a stream of
traffic. The form of partial sequence integrity offered in IPsec is referred to as anti-replay integrity, and it detects
arrival of duplicate IP datagrams (within a constrained window). This is in contrast to connection-oriented
integrity, which imposes more stringent sequencing requirements on traffic, e.g., to be able to detect lost or re-
ordered messages. Although authentication and integrity services often are cited separately, in practice they are
intimately connected and almost always offered in tandem.
Integrity Check Value (ICV) A checksum used to detect modification of field values
Lightweight Directory Access Protocol (LDAP) A set of protocols for accessing information directories; LDAP
can be used to distribute PKI certificates to the public.
Public Key Infrastructure (PKI) The use of public services that issue/validate public keys for purposes of
secure communication.
Replay attacks A network attack where an attacker intercepts packets and transmits them again later.
Security Association (SA) A simplex (uni-directional) logical connection, created for security purposes. All
traffic traversing an SA is provided the same security processing. In IPsec, an SA is an Internet layer abstraction
implemented through the use of AH or ESP.

IPsec – SANS ©2000 - 2003 41

This page intentionally left blank.

9 - 41
 Glossary and Definitions (3)
Security Gateway A security gateway is an intermediate system that acts as the communications interface
between two networks. The set of hosts (and networks) on the external side of the security gateway is viewed as
untrusted (or less trusted), while the networks and hosts and on the internal side are viewed as trusted (or more
trusted). The internal subnets and hosts served by a security gateway are presumed to be trusted by virtue of
sharing a common, local, security administration. (See "Trusted Subnetwork" below.) In the IPsec context, a
security gateway is a point at which AH and/or ESP is implemented in order to serve a set of internal hosts,
providing security services for these hosts when they communicate with external hosts also employing IPsec
(either directly or via another security gateway).
Security Parameters Index (SPI) The combination of a destination address, a security protocol, and an SPI
uniquely identifies a security association (SA, see above). The SPI is carried in AH and ESP protocols to enable
the receiving system to select the SA under which a received packet will be processed. An SPI has only local
significance, as defined by the creator of the SA (usually the receiver of the packet carrying the SPI); thus an
SPI is generally viewed as an opaque bit string. However, the creator of an SA may choose to interpret the bits
in an SPI to facilitate local processing.
Symmetric key cryptography A form of cryptography where the sender and receiver share a key (usually
called the secret key).
Traffic Analysis The analysis of network traffic flow for the purpose of deducing information that is useful to an
adversary. Examples of such information are frequency of transmission, the identities of the conversing parties,
sizes of packets, flow identifiers, etc. [Sch94]
Trusted Subnetwork A subnetwork containing hosts and routers that trust each other not to engage in active or
passive attacks. There also is an assumption that the underlying communications channel (e.g., a LAN or CAN)
isn't being attacked by other means.
X.500 An ITU-T standard defining the structure of global directories.

IPsec – SANS ©2000 - 2003 42

This page intentionally left blank.

9 - 42
 IPsec quiz
1) IPsec can be implemented between:
a) two hosts
b) two security gateways
c) a host and security gateway
d) all of the above

2) If you were to examine a dump of a packet that had ESP encryption

using tunnel mode, you could read/decode the following fields:
a) The new IP header, the ESP header
b) The new IP header, the ESP header, and the original IP header
c) The new IP header, the ESP header, and the original payload
d) The ESP header, the original IP header

IPsec – SANS ©2000 - 2003 43

This page intentionally left blank.

9 - 43
 IPsec quiz (2)
3) ip-proto-50 in tcpdump output designates:
a) Encapsulation Security Payload
b) Security Association database transfer
c) Security Association policy transfer
d) Aggressive mode key exchange

4) Two databases used for IPsec are:

a) ESP and AH databases
b) Key and authentication databases
c) Tunneling and non-tunneling databases
d) A security policy database (SPD) and a security association
database (SAD)

IPsec – SANS ©2000 - 2003 44

This page intentionally left blank.

9 - 44
 IPsec quiz (3)
5) Each security association must be uniquely identified by:
a) source and destination IP addresses
b) a Security Parameter Index (SPI), destination IP address, and
security protocol (AH or ESP)
c) a master security gateway and slave security gateway
d) security gateway and encryption algorithm

6) ip-proto-51 in tcpdump output designates:

a) Authentication Header
b) Security Association database transfer
c) Security Association policy transfer
d) Aggressive mode key exchange

45
IPsec – SANS ©2000 - 2003

This page intentionally left blank.

9 - 45
 IPsec quiz (4)
7) AH or ESP in Transport Mode between two IPsec devices will:
a) Alter the source IP address but not the destination IP address
b) Alter the destination IP address but not the source IP address
c) Alter both source and destination IP addresses
d) Not alter the source or destination IP addresses

8) AH or ESP in Tunnel mode between two IPsec devices will:

a) Alter the source IP address but not the destination IP address
b) Alter the destination IP address but not the source IP address
c) Alter both source and destination IP addresses
d) Not alter the source and destination IP addresses

IPsec – SANS ©2000 - 2003 46

This page intentionally left blank.

9 - 46
 IPsec quiz (5)
9) Which best describes Internet Key Exchange:
a) It manages AH traffic only
b) It manages ESP traffic only
c) It defines how two IPsec nodes decide which algorithms they will use
for authentication and encryption, and how long the session will last
d) It defines the public key length

10) Which of the following is true:

a) AH provides authentication and encryption for the entire packet
b) AH provides authentication for the entire packet except for mutable
fields
c) AH provides encryption for the entire packet
d) AH provides neither authentication nor encryption for the entire
packet

47
IPsec – SANS ©2000 - 2003

This page intentionally left blank.

9 - 47
 IPsec quiz (6)
11) Which of the following is true:
a) ESP authenticates everything but the IP header and encrypts the
packet payload
b) ESP authenticates the entire packet and provides no encryption
c) ESP provides neither authentication nor encryption for the payload
d) ESP encrypts the IP header only and provides no authentication

12) AH traffic in Transport Mode will have:

a) AH header, ESP header, data
b) Original IP header, AH header, upper layer protocol and data
c) AH header, ESP data
d) IKE header, AH header, ESP header

48
IPsec – SANS ©2000 - 2003

This page intentionally left blank.

9 - 48
 IPsec quiz (7)
13) AH in Tunnel Mode will have:
a) New IP header, AH header, original IP header, upper layer protocol
and data
b) Original AH header, AH data, new AH header
c) AH header, AH data
d) New AH header, AH data, original AH header

14) If we capture an ESP packet operating in Tunnel Mode after encryption

and examine it; we cannot:
a) See the ip-proto field
b) See the security gateway source IP address
c) See the security gateway destination IP address
d) Decrypt the payload

49
IPsec – SANS ©2000 - 2003

This page intentionally left blank.

9 - 49
 IPsec quiz (8)
15) A common use for IPsec is:
a) Virus eradication
b) To communicate between two remote networks over the Internet
c) To improve router convergence time
d) To cache DNS lookups

50
IPsec – SANS ©2000 - 2003

This page intentionally left blank.

9 - 50
 IPsec quiz (9)
16) IPsec is a set of protocols that provide security at the IP layer. (T/F)
17) Two security protocols offered by IPsec are Authentication Header and
Authentication Trailer. (T/F)
18) An established Security Association lasts forever. (T/F)
19) The Security Policy Database maintains the engagement rules when
another IPsec node requests connectivity. (T/F)
20) Each Security Association created for an IPsec session must be uniquely
identified. (T/F)
21) Three possible security policies for inbound traffic are: apply IPsec, or
apply intermittent IPsec, or apply random IPsec. (T/F)
22) When using ESP, either authentication or encryption must be employed.
(T/F)

51
IPsec – SANS ©2000 - 2003

This page intentionally left blank.

9 - 51
 IPsec quiz (10)
23) Tunnel Mode never uses security gateways. (T/F)
24) ISAKMP, which is used for key and security parameter exchanges uses
UDP port 500 (T/F)
25) Three IKE modes are main, aggressive and quick. (T/F)
26) IP protocol 51 AH authenticates the entire packet. (T/F)
27) AH encrypts the payload. (T/F)
28) ESP can provide confidentiality and integrity. (T/F)
29) ESP encrypts the original packet payload. (T/F).
30) If you see ip-proto-50 or ip-proto-51 in tcpdump output, it means IPsec is
in use. (T/F)

52
IPsec – SANS ©2000 - 2003

Answers:
1) d 16) T
2) a 17) F
3) a 18) F
4) d 19) T
5) b 20) T
6) a 21) F
7) d 22) F
8) c 23) F
9) c 24) T
10) b 25) T
11) a 26) T
12) b 27) F
13) a 28) T
14) d 29) T
15) b 30) T

9 - 52
 Course Revision History

53
IPsec – SANS ©2000 - 2003

v1.0 – Jean Triquet

v1.1 – edited by S. Northcutt – 23 Oct 2000
v1.2 – edited by J. Novak – 25 Dec 2000
v1.3 – J. Kolde, formatting changes – 22 Jan 2001
v1.4 - G. Kessler, major changes to graphics, chapter flow, and depth of coverage -- 13 June 2001
v.1.5 – J. Novak corrections (96bytes to bits ) slide 30, slide 18 deleted word respectively, slide 27 word
immutable corrected to mutable. – 10 Aug 2002, quiz updated
v.1.6 – J. Novak slide 1 changed basic to basics in first sentence of notes, slide 9 – changed bullet to
“Destination IP address of outer header”, slide 14 changed IPsec Mode in table of first entry to tunnel,
slide 25 – notes page grammar corrections, and omission of phrase “and encryption algorithms (e.g.,
DES or 3DES)” 12 Nov 2002

9 - 53