How IPSec Works

IPSec involves many component technologies and encryption methods.
Yet IPSec’s operation can be broken down into five main steps:

  1. “Interesting traffic” initiates the IPSec process. Traffic is deemed interesting when the IPSec
    security policy configured in the IPSec peers starts the IKE process.
  2. IKE phase 1. IKE authenticates IPSec peers and negotiates IKE SAs during this phase,
    setting up a secure channel for negotiating IPSec SAs in phase 2.
  3. IKE phase 2. IKE negotiates IPSec SA parameters and sets up matching IPSec SAs in the peers.
  4. Data transfer. Data is transferred between IPSec peers based on the IPSec parameters and
    keys stored in the SA database.
  5. IPSec tunnel termination. IPSec SAs terminate through deletion or by timing out.

This five-step process :

The five steps of IPSec for dummies like Mufty:

Step 1—Defining Interesting Traffic

What type of traffic is deemed interesting is determined as part of formulating a security
policy for use of a VPN. The policy is then implemented in the configuration interface for
each particular IPSec peer.

For example, in Cisco routers and PIX Firewalls, access lists are used to determine the
traffic to encrypt. The access lists are assigned to a cryptography policy;
the policy’s permit statements indicate that the selected traffic must be encrypted, and deny statements
indicate that the selected traffic must be sent unencrypted. With the Cisco Secure VPN Client, you
use menu windows to select connections to be secured by IPSec. When interesting traffic is generated
or transits the IPSec client, the client initiates the next step in the process,
negotiating an IKE phase 1 exchange.

Defining “interesting traffic.”

Step 2—IKE Phase 1

The basic purpose of IKE phase 1 is to authenticate the IPSec peers and to set up a secure channel
between the peers to enable IKE exchanges. IKE phase 1 performs the following functions:

  • Authenticates and protects the identities of the IPSec peers
  • Negotiates a matching IKE SA policy between peers to protect the IKE exchange
  • Performs an authenticated Diffie-Hellman exchange with the end result of having matching
    shared secret keys
  • Sets up a secure tunnel to negotiate IKE phase 2 parameters

IKE phase 1 occurs in two modes: main mode and aggressive mode.
These modes are described in the following sections.

Main Mode

Main mode has three two-way exchanges between the initiator and the receiver.

  • First exchange: The algorithms and hashes used to secure the IKE communications are
    agreed upon in matching IKE SAs in each peer.
  • Second exchange: Uses a Diffie-Hellman exchange to generate shared secret keying material
    used to generate shared secret keys and to pass nonces—random numbers sent to the other
    party and then signed and returned to prove their identity.
  • Third exchange: Verifies the other side’s identity. The identity value is the IPSec peer’s IP address
    in encrypted form. The main outcome of main mode is matching IKE SAs between peers to provide
    a protected pipe for subsequent protected ISAKMP exchanges between the IKE peers.
    The IKE SA specifies values for the IKE exchange: the authentication method used, the encryption
    and hash algorithms, the Diffie-Hellman group used, the lifetime of the IKE SA in seconds or kilobytes,
    and the shared secret key values for the encryption algorithms.
    The IKE SA in each peer is bi-directional    .

Aggressive Mode

In aggressive mode, fewer exchanges are made, and with fewer packets. On the first exchange,
almost everything is squeezed into the proposed IKE SA values: the Diffie-Hellman public key;
a nonce that the other party signs; and an identity packet, which can be used to verify identity
via a third party.

The receiver sends everything back that is needed to complete the exchange. The only thing left is
for the initiator to confirm the exchange. The weakness of using the aggressive mode is that both sides
have exchanged information before there’s a secure channel. Therefore, it’s possible to “sniff” the wire and
discover who formed the new SA. However, it is faster than main mode.

Step 3—IKE Phase 2

The purpose of IKE phase 2 is to negotiate IPSec SAs to set up the IPSec tunnel.
IKE phase 2 performs the following functions:

  • Negotiates IPSec SA parameters protected by an existing IKE SA
  • Establishes IPSec security associations
  • Periodically renegotiates IPSec SAs to ensure security
  • Optionally performs an additional Diffie-Hellman exchange

IKE phase 2 has one mode, called quick mode. Quick mode occurs after IKE has established the
secure tunnel in phase 1. It negotiates a shared IPSec policy, derives shared secret keying material
used for the IPSec security algorithms, and establishes IPSec SAs. Quick mode exchanges nonces
that provide replay protection. The nonces are used to generate new shared secret key material and
prevent replay attacks from generating bogus SAs.

Quick mode is also used to renegotiate a new IPSec SA when the IPSec SA lifetime expires.
Base quick mode is used to refresh the keying material used to create the shared secret key based on the
keying material derived from the Diffie-Hellman exchange in phase 1.

Perfect Forward Secrecy

If perfect forward secrecy (PFS) is specified in the IPSec policy, a new Diffie-Hellman
exchange is performed with each quick mode, providing keying material that has
greater entropy (key material life) and thereby greater resistance to cryptographic attacks.
Each Diffie-Hellman exchange requires large exponentiations, thereby increasing CPU use
and exacting a performance cost.

Step 4—IPSec Encrypted Tunnel

After IKE phase 2 is complete and quick mode has established IPSec SAs, information is exchanged via
an IPSec tunnel. Packets are encrypted and decrypted using the encryption specified in the IPSec SA.

IPSec encrypted tunnel.

Step 5—Tunnel Termination

IPSec SAs terminate through deletion or by timing out. An SA can time out when a
specified number of seconds have elapsed or when a specified number of bytes have passed through
the tunnel. When the SAs terminate, the keys are also discarded. When subsequent IPSec SAs are
needed for a flow, IKE performs a new phase 2 and, if necessary, a new phase 1 negotiation.
A successful negotiation results in new SAs and new keys. New SAs can be established before the
existing SAs expire, so that a given flow can continue uninterrupted.