IPsec is a group of protocols that together encapsulate, authentication and encrypt traffic.  IPsec is a common solution for corporate VPN gateways.  The 2 endpoints of an encrypted tunnel are usually referred to as "peers", but in some cases can also be called a client and a gateway (but even then they are still peers). 

This is not meant to be a complete discussion of VPNs or IPsec - there are entire books dedicated to just this topic (some of them are on my bookshelf I'm sad to say).  If you spot any errors, find something that should be added, or have a personal experience you'd like to share on this topic, please add your comments !

The most common protocols used in today's VPNs are:

IKE (Internet Key Exchange) (formerly known as ISAKMP - Internet Security Association and Key Management Protocol) is the most common protocol used to authenticate the VPN session.  IKE is transported on 500/udp. 
Setting up an IKE Security Association is generally split into 2 phases:
Phase 1 sets up an initial secure tunnel between the peers.  This phase always uses a Diffie-Hellman (DH) key agreement.  IKE Phase 1 can use either "Main Mode" or "Aggressive Mode" to get this job done.  Main Mode uses three 2-way exchanges in its process, Aggressive Mode squeezes this down to a "one and a half exchange" (see below)

Main Mode negotiation has 6 messages.
Client -> Server
Server -> Client
Client -> Server
Server -> Client
Client -> Server
Server -> Client

Aggressive Mode has 3 messages.
Client -> Server
Server -> Client
Client -> Server

The weakness of Aggressive mode is that some data is exchanged before a secure tunnel is in place, so sniffing an aggressive mode connection can result in compromise of the final association.  However, Aggressive mode is a quicker process than Main Mode.

Phase 2 negotiates the IPSEC Security Association, which among other things includes what encryption algorithms to use (AES or ESP are common protocols that might be negotiated).  The IPsec security association is what actually transports and encrypts the data.

IKE can use Pre-shared keys, RSA Signatures or RSA encrypted nonces to authenticate peers.

ESP (Encapsulating Security Payload) is the most common protocol for encapsulation of the actual data in the VPN session.  ESP is IP Protocol 50, so is not based TCP or UDP protocols.  Because of this, NAT devices often have a problem with ESP (read on for more on this).   The actual encryption algorithm within the tunnel is negotiated when the ESP session starts up.  The vpn gateway generally has a list of supported protocols (3DES and AES are the most common these days), which it will match up to a similar list that the client will offer up as it attempts to connect. 

A common configuration decision is to decide if a VPN tunnel will use "tunnel mode" or "transport mode".  Transport mode encrypts the data payload, but maintains the original IP header fields.  Tunnel mode encapsulates the whole packet, so encrypts both the header and payload, and adds its own header fields, treating the entire original packet as payload.  In other words, Transport Mode has less overhead (both in encryption cycles and added bytes), but Tunnel Mode encrypts the entire packet.

NAT-T-IKE (4500/udp) allows the ESP session to be encapsulated within a more NAT-friendly UDP packet.  Most VPN clients and gateways natively support NAT-T. NAT-T is sometimes also seen on 500/udp, especially in older implementations, or on any configured tcp or udp port (10000 is the default tcp port on cisco gear).  Some vendors also allow "NAT-T style" encapsulation within TCP packets

Common IPSEC NAT problems solved by NAT-T: 
1/ If an intermediate device passes only TCP and UDP traffic, the ESP encapsulation will fail, since ESP is IP Protocol 50 (ie - it's not TCP or UDP).  The symptom that the end-user sees in this case is that their VPN session connects, but then no traffic is passed.  Helpdesk calls for this are typically "I can connect, but can't get my email / files / etc"

2/ When a person in a remote location (conference hotel, branch office, wherever) VPN's into head office, their session works.  When a second person at that same location VPN's into the same head office, both sessions immediately fail.  This is a problem with NAT.  Since ESP is not based on TCP or UDP, it does not use ports, so the second ESP session from A to B looks exactly like the first ESP session from A to B. The fix for this is NAT-T, though if the two ESP sessions NAT out on different public IP's, that will work as well.

Both of these issues are more completely described in the RFC for NAT-T (see below).

PPTP is actually a "meld" of a PPP session (tcp/1723) and a GRE (Generic Route Encapsulation) tunnel (IP protocol 47).  PPTP is generally authenticated using MSCHAP-v2 or EAP-TLS, and in later versions is encrypted using Microsoft's MPPE (Microsoft Point-to-Point Encryption, RFC 3078).

AH (Authentication Header) is always covered in discussions of IPSEC and VPN's, but I've never personally used it for a production VPN solution - since AH does not encrypt the payload, it's not particularly appealing in situations that require confidentiality (as most internet based VPNs do).  AH is all about verifying the integrity of the data, rather than actually encrypting it for confidentiality.  AH uses IP Protocol 51.

For security assessments, ike-scan is a handy command line tool that uses IKE to discover and fingerprint IPSEC VPN gateways.  This tool also comes with a brute-forcer for IPSEC preshared keys (psk-crack).   Linux, OSX and Windows versions are available here ==>  http://www.nta-monitor.com/tools/ike-scan

SSL based VPNs seem to be an industry trend recently.  While these capitalize on the encryption and certificate support built into today's operating systems and browsers, they also migrate us into a world of a more well-understood protocol, with a much larger stable of exploit tools.  If implemented correctly, SSL VPNs can be a great thing.  However, if implemented without a solid understanding of the potential pitfalls, SSL VPNs are susceptible to attack using many of the same tools you might use against an SSL session to a protected website.  For more information, see our Cyber Security Awareness Month day 25 Discussion of ports 80 and 443 ( http://isc.sans.org/diary.html?storyid=7450 ) and this recent entry on Sniffing SSL ( http://isc.sans.org/diary.html?storyid=7477 )


===============================================================

Typical IKE-SCAN sessions are shown below (ip addresses represented as "a.b.c.d").  Note that some hosts return a full handshake, some simply return a notify

Scanning a single host:

c:> ike-scan.exe vpn.somecompany.com
Starting ike-scan 1.9 with 1 hosts (http://www.nta-monitor.com/tools/ike-scan/)
a.b.c.d    Main Mode Handshake returned HDR=(CKY-R=4feb587b34903be0) SA=(En
c=3DES Hash=SHA1 Group=2:modp1024 Auth=PSK LifeType=Seconds LifeDuration=28800)
VID=4048b7d56ebce88525e7de7f00d6c2d3c0000000 (IKE Fragmentation)

Ending ike-scan 1.9: 1 hosts scanned in 0.152 seconds (6.58 hosts/sec).  1 returned handshake; 0 returned notify


Scanning a subnet with several VPN gateways:

c:> ike-scan a.b.c.0/24
Starting ike-scan 1.9 with 64 hosts (http://www.nta-monitor.com/tools/ike-scan/)

a.b.c.10  Notify message 14 (NO-PROPOSAL-CHOSEN) HDR=(CKY-R=353da7769f710767)
a.b.c.33  Notify message 7 (INVALID-EXCHANGE-TYPE) HDR=(CKY-R=c03397649b1d76fe)
a.b.c.58  Notify message 14 (NO-PROPOSAL-CHOSEN) HDR=(CKY-R=41665b363ae98c7c, msgid=71d55f0e)
a.b.c.75  Main Mode Handshake returned HDR=(CKY-R=68f5d1b1b716f49e) SA=(En
c=3DES Hash=SHA1 Group=2:modp1024 Auth=PSK LifeType=Seconds LifeDuration=28800)
VID=4048b7d56ebce88525e7de7f00d6c2d3c0000000 (IKE Fragmentation)
a.b.c.77  Notify message 14 (NO-PROPOSAL-CHOSEN) HDR=(CKY-R=36431b961d8d5237)
a.b.c.105 Main Mode Handshake returned HDR=(CKY-R=1ecbf70c195d4aaf) SA=(En
c=3DES Hash=MD5 Group=2:modp1024 Auth=PSK LifeType=Seconds LifeDuration=28800)
a.b.c.130 Main Mode Handshake returned HDR=(CKY-R=59e03fd934356da6) SA=(Au
th=PSK Hash=SHA1 Enc=3DES Group=2:modp1024 LifeType=Seconds LifeDuration(4)=0x00
007080) VID=8404adf9cda05760b2ca292e4bff537b (Maybe Sidewinder G2) VID=baf4ee8d4
373a44b44e7efc091adb5c2


Ending ike-scan 1.9: 256 hosts scanned in 40.409 seconds (6.34 hosts/sec).  3 returned handshake; 4 returned notify


Now, let's try forcing these same hosts to an aggressive mode IKE exchange. Note the new information for hosts .10 and .105 (including the VPN vendor), and that host .130 returns less information this time.  Hosts .33 .58 and .75 didn't answer at all this time.  Differences like this not only give you new information directly, but the information that is left out can be valuable as well during recon, allowing you to sometimes identify targets right down to OS versions, which can in turn identify potential bugs that might be used during a penetration test.


c:> ike-scan --aggressive a.b.c.0/24
Starting ike-scan 1.9 with 256 hosts (http://www.nta-monitor.com/tools/ike-scan/)
a.b.c.10  Aggressive Mode Handshake returned HDR=(CKY-R=353da776bbb4b996)
SA=(Enc=3DES Hash=MD5 Group=2:modp1024 Auth=PSK LifeType=Seconds LifeDuration=28
800) VID=12f5f28c457168a9702d9fe274cc0100 (Cisco Unity) VID=afcad71368a1f1c96b86
96fc77570100 (Dead Peer Detection v1.0) VID=c0fa006bbbb5b99678f8a0fa93809964 VID
=09002689dfd6b712 (XAUTH) KeyExchange(128 bytes) ID(Type=ID_IPV4_ADDR, Value=a
.b.c.10) Nonce(20 bytes) Hash(16 bytes)
a.b.c.77  Notify message 14 (NO-PROPOSAL-CHOSEN) HDR=(CKY-R=36431b96ca6a9cf4)
a.b.c.105 Aggressive Mode Handshake returned HDR=(CKY-R=1ecbf70ceb9f629b)
SA=(Enc=3DES Hash=MD5 Group=2:modp1024 Auth=PSK LifeType=Seconds LifeDuration=28
800) VID=09002689dfd6b712 (XAUTH) VID=afcad71368a1f1c96b8696fc77570100 (Dead Pee
r Detection v1.0) VID=12f5f28c457168a9702d9fe274cc0100 (Cisco Unity) VID=eb0c501
1eb9e629b323f3948e404dff2 KeyExchange(128 bytes) ID(Type=ID_FQDN, Value=pix01.
somecompany.com) Nonce(20 bytes) Hash(16 bytes)
a.b.c.130 Unexpected IKE payload returned: Delete Notification=(Type=INVALID-EXCHANGE-TYPE, SPI=, Data=)

Ending ike-scan 1.9: 256 hosts scanned in 42.905 seconds (5.97 hosts/sec).  2 re
turned handshake; 1 returned notify


=======================================

References:

As you can see, reading RFC's is a great way to get way to get the real "nuts and bolts" source material for all of these protocols.  They aren't always the easiest to get through, but any understanding of internet basics should include the RFC's upon which they're based.  RFC's build on each other, you will see that many of these RFC references are updated versions of now obsolete original documents.

IKE:
RFC2409 - RFC4306 Internet Key Exchange (IKEv2) Protocol - http://tools.ietf.org/html/rfc4306
(Obsoletes RFC2407, RFC2408, RFC2409, Updated by RFC5282)

ESP:
IP Encapsulating Security Payload (ESP) - http://tools.ietf.org/html/rfc4303
(Obsoletes RFC2406)

NAT-T:
Negotiation of NAT-Traversal in the IKE - http://tools.ietf.org/html/rfc3947
UDP Encpasulation of IPsec ESP Packets - http://tools.ietf.org/html/rfc3948

PPTP:
Microsoft Point-To-Point Encryption (MPPE) Protocol - RFC 3078 - http://tools.ietf.org/html/rfc3078

AH:
RFC4302 IP Authentication Header - http://tools.ietf.org/html/rfc4302
(Obsoletes RFC2402)

3DES
The ESP Triple DES Transform - http://www.ietf.org/rfc/rfc1851.txt

AES:
The Transport Layer Security (TLS) Protocol Version 1.2. - http://tools.ietf.org/html/rfc5246
(Obsoletes RFC3268, RFC4346, RFC4366) (Updates RFC4492)
>> (AES was originally RFC3268)