Saturday, October 11, 2014

ASA 5525-X Reload and VPN Software Bug

I was recently configuring and troubleshooting an IPsec Site-to-Site (IKEv1) VPN between our newly installed ASA 5525-X (my third ASA for this year) and our core ASA 5520 firewalls. I was making some changes to it and wanted to quickly wipe out some of the added command lines. I performed a reload command and a "weird" thing happened. I can't seem to remotely reset the box and tried all possible combinations of the reload command but nothing worked. My remote session was still connected and I can still ping to the box.

So the last resort is to issue a crashinfo force watchdog command from privileged EXEC mode. This command will force the device to crash, generate a crash output and reload the device. It is practically safe to issue this command and you'll still be able to connect remotely after a few minutes (5-7 minutes).


5525-X# reload
Proceed with reload? [confirm]   // STILL CONNECTED

5525-X# reload noconfirm   // STILL CONNECTED

5525-X# reload in 10    // STILL CONNECTED AFTER 10 mins

5525-X# reload
System config has been modified. Save? [Y]es/[N]o:y
Cryptochecksum: c6bd3ee7 cc75760d 6ecf8bd4 d0fe71a2
8505 bytes copied in 0.650 secs
Proceed with reload? [confirm]   // STILL CONNECTED

5525-X# show reload
Shutting down the system right now.

5525-X# debug crypto ikev1 255
Oct 02 22:37:49 [IKEv1]IP = 202.x.x.x, Reboot Underway... dropping new P1 packet.
Oct 02 22:37:57 [IKEv1]IP = 202.x.x.x, Reboot Underway... dropping new P1 packet.
Oct 02 22:38:05 [IKEv1]IP = 202.x.x.x, Reboot Underway... dropping new P1 packet.

5525-X# show crypto isakmp sa
There are no IKEv1 SAs

There are no IKEv2 SAs


5525-X# crashinfo force watchdog     // THIS COMMAND IS A LIFESAVER!
WARNING: This command will force a crash and cause a
         reboot. Do you wish to proceed? [confirm]:y


This seems to be a symptom of a software bug and I was able to resolve (most of the time it does) the reload and debug/show crypto issue by updating the image of the ASA firewall. According to Cisco's website, the recommended and stable code (as of this writing) is the 9.1(5). I can upgrade directly from 9.1(2) to 9.1(5) since it's a minor release code.


5525-X# show flash | inc .bin
  110  38191104    Apr 29 2014 14:51:00  asa912-smp-k8.bin
  111  18097844    Apr 29 2014 14:52:20  asdm-713.bin
  123  37822464    Oct 06 2014 19:21:33  asa915-smp-k8.bin    // DUMPED IMAGE VIA TFTP
5525-X# show run boot
boot system disk0:/asa912-smp-k8.bin

5525-X# configure terminal
5525-X(config)# no boot system disk0:/asa912-smp-k8.bin
5525-X(config)# boot system disk0:/asa915-smp-k8.bin
5525-X(config)# boot system disk0:/asa912-smp-k8.bin     // CAN SPECIFY MULTIPLE BOOT IMAGES IN SEQUENTIAL ORDER
5525-X(config)# end
5525-X# show run boot
boot system disk0:/asa915-smp-k8.bin
boot system disk0:/asa912-smp-k8.bin

5525-X#reload

<OUTPUT TRUNCATED>

5525-X# show version

Cisco Adaptive Security Appliance Software Version 9.1(5)
Device Manager Version 7.1(3)

Compiled on Thu 27-Mar-14 10:19 PDT by builders
System image file is "disk0:/asa915-smp-k8.bin"
Config file at boot was "startup-config"

5525-X up 1 hour 36 mins


5525-X# show crypto isakmp sa

IKEv1 SAs:

   Active SA: 1
    Rekey SA: 0 (A tunnel will report 1 Active and 1 Rekey SA during rekey)
Total IKE SA: 1

1   IKE Peer: 202.x.x.x
    Type    : L2L             Role    : initiator
    Rekey   : no              State   : MM_ACTIVE



This is the debug crypto isakmp output taken from the remote peer/ASA firewall:

(5) + HASH (8) + IOS KEEPALIVE (128) + VENDOR (13) + NONE (0) total length : 92
Oct 07 10:28:07 [IKEv1 DEBUG]: Group = 202.x.x.x, IP = 202.x.x.x, processing ID payload
Oct 07 10:28:07 [IKEv1 DECODE]: Group = 202.x.x.x, IP = 202.x.x.x, ID_IPV4_ADDR ID received 202.x.x.x
Oct 07 10:28:07 [IKEv1 DEBUG]: Group = 202.x.x.x, IP = 202.x.x.x, processing hash payload
Oct 07 10:28:07 [IKEv1 DEBUG]: Group = 202.x.x.x, IP = 202.x.x.x, Computing hash for ISAKMP
Oct 07 10:28:07 [IKEv1 DEBUG]: IP = 202.x.x.x, Processing IOS keep alive payload: proposal=32767/32767 sec.
Oct 07 10:28:07 [IKEv1 DEBUG]: Group = 202.x.x.x IP = 202.x.x.x, processing VID payload
Oct 07 10:28:07 [IKEv1 DEBUG]: Group = 202.x.x.x, IP = 202.x.x.x, Received DPD VID
Oct 07 10:28:07 [IKEv1]: Group = 202.x.x.x, IP = 202.x.x.x, Automatic NAT Detection Status:    
Remote end is NOT behind a NAT device     This   end is NOT behind a NAT device
Oct 07 10:28:07 [IKEv1]: IP = 202.x.x.x, Connection landed on tunnel_group 202.x.x.x
Oct 07 10:28:07 [IKEv1 DEBUG]: Group = 202.x.x.x, IP = 202.x.x.x, constructing ID payload
Oct 07 10:28:07 [IKEv1 DEBUG]: Group = 202.x.x.x, IP = 202.x.x.x, constructing hash payload
Oct 07 10:28:07 [IKEv1 DEBUG]: Group = 202.x.x.x, IP = 202.x.x.x, Computing hash for ISAKMP
Oct 07 10:28:07 [IKEv1 DEBUG]: IP = 202.176.12.2, Constructing IOS keep alive payload: proposal=32767/32767sec.
Oct 07 10:28:07 [IKEv1 DEBUG]: Group = 202.x.x.x, IP = 202.x.x.x, constructing dpd vid payload
Oct 07 10:28:07 [IKEv1]: IP = 202.176.12.2, IKE_DECODE SENDING Message (msgid=0) with payloads : HDR + ID

(5) + HASH (8) + IOS KEEPALIVE (128) + VENDOR (13) + NONE (0) total length : 92

BEFORE ENCRYPTION
RAW PACKET DUMP on SEND
ac 02 de b5 23 50 bb bf d4 ce ec fe d2 3c c2 81    |  ....#P.......<..
05 10 02 00 00 00 00 00 1c 00 00 00 08 00 00 0c    |  ................
01 11 00 00 ca 4e 10 0e 80 00 00 14 ba 65 ad 93    |  .....N.......e..
9a b0 58 4c b3 9f 91 92 af 9b 71 d7 0d 00 00 0c    |  ..XL......q.....
80 00 7f ff 80 00 7f ff 00 00 00 14 af ca d7 13    |  ..............
68 a1 f1 c9 6b 86 96 fc 77 57 01 00                |  h...k...wW..

ISAKMP Header
  Initiator COOKIE: ac 02 de b5 23 50 bb bf
  Responder COOKIE: d4 ce ec fe d2 3c c2 81
  Next Payload: Identification
  Version: 1.0
  Exchange Type: Identity Protection (Main Mode)
  Flags: (none)
  MessageID: 00000000
  Length: 469762048
  Payload Identification
    Next Payload: Hash
    Reserved: 00
    Payload Length: 12
    ID Type: IPv4 Address (1)
    Protocol ID (UDP/TCP, etc...): 17
    Port: 0
    ID Data: 202.x.x.x
  Payload Hash
    Next Payload: IOS Proprietary Keepalive or CHRE
    Reserved: 00
    Payload Length: 20
    Data:
      ba 65 ad 93 9a b0 58 4c b3 9f 91 92 af 9b 71 d7
  Payload IOS Proprietary Keepalive or CHRE
    Next Payload: Vendor ID
    Reserved: 00
    Payload Length: 12
    Default Interval: 32767
    Retry Interval: 32767
  Payload Vendor ID
    Next Payload: None
    Reserved: 00
    Payload Length: 20
    Data (In Hex):
      af ca d7 13 68 a1 f1 c9 6b 86 96 fc 77 57 01 00

SENDING PACKET to 202.x.x.x
ISAKMP Header
  Initiator COOKIE: ac 02 de b5 23 50 bb bf
  Responder COOKIE: d4 ce ec fe d2 3c c2 81
  Next Payload: Identification
  Version: 1.0
  Exchange Type: Identity Protection (Main Mode)
  Flags: (Encryption)
  MessageID: 00000000
  Length: 92
Oct 07 10:28:07 [IKEv1]: Group = 202.x.x.x, IP = 202.x.x.x, PHASE 1 COMPLETED
Oct 07 10:28:07 [IKEv1]: IP = 202.x.x.x, Keep-alive type for this connection: DPD
Oct 07 10:28:07 [IKEv1 DEBUG]: Group = 202.x.x.x, IP = 202.x.x.x, Starting P1 rekey timer: 73440 seconds.

IKE Recv RAW packet dump
ac 02 de b5 23 50 bb bf d4 ce ec fe d2 3c c2 81    |  ....#P.......<..
08 10 20 01 6d d4 d2 0f 00 00 00 bc 11 6c 79 02    |  .. .m........ly.
9f 7d d0 e4 5c 92 48 0f cd 4f 0f d5 57 fe b9 7e    |  .}..\.H..O..W..~
de 1b fd 84 db ea 6e 10 08 93 54 88 fc 10 a6 fd    |  ......n...T.....
33 a3 d8 90 11 6c e1 97 25 9e 5e 0c 8a 47 26 86    |  3....l..%.^..G&.
50 8d 18 c2 e4 9b 74 3e fe 1d d7 28 d5 67 89 a0    |  P.....t>...(.g..
ef ba f3 b1 73 f8 70 cb b9 37 bb 5d 29 28 93 b0    |  ....s.p..7.])(..
eb fd 6e 5c 5e bb 17 e2 0b b3 aa 60 05 76 a2 58    |  ..n\^......`.v.X
fc 66 48 cc 61 07 eb 67 91 6f 3b 9a 05 93 5d 76    |  .fH.a..g.o;...]v
d9 de 0f db 71 22 16 c5 ac 0c 6e 03 34 45 84 1d    |  ....q"....n.4E..
18 55 f1 e1 6b 5f 9b 3d 40 76 05 2f ce 46 f7 73    |  .U..k_.=@v./.F.s
c1 ba 4e c8 31 e8 fb 06 42 3a b6 d0                |  ..N.1...B:..

 RECV PACKET from 202.x.x.x
ISAKMP Header
  Initiator COOKIE: ac 02 de b5 23 50 bb bf
  Responder COOKIE: d4 ce ec fe d2 3c c2 81
  Next Payload: Hash
  Version: 1.0
  Exchange Type: Quick Mode
  Flags: (Encryption)
  MessageID: 6DD4D20F
  Length: 188
Oct 07 10:28:07 [IKEv1 DECODE]: IP = 202.x.x.x, IKE Responder starting QM: msg id = 6dd4d20f

AFTER DECRYPTION
ISAKMP Header
  Initiator COOKIE: ac 02 de b5 23 50 bb bf
  Responder COOKIE: d4 ce ec fe d2 3c c2 81
  Next Payload: Hash
  Version: 1.0
  Exchange Type: Quick Mode
  Flags: (Encryption)
  MessageID: 6DD4D20F
  Length: 188
  Payload Hash
    Next Payload: Security Association
    Reserved: 00
    Payload Length: 20
    Data:
      f8 03 04 ee ae b2 d3 a7 e4 f3 92 67 b5 8d a8 78
  Payload Security Association
    Next Payload: Nonce
    Reserved: 00
    Payload Length: 60
    DOI: IPsec
    Situation:(SIT_IDENTITY_ONLY)
    Payload Proposal
      Next Payload: None
      Reserved: 00
      Payload Length: 48
      Proposal #: 1
      Protocol-Id: PROTO_IPSEC_ESP
      SPI Size: 4
      # of transforms: 1
      SPI: b7 52 63 26
      Payload Transform
        Next Payload: None
        Reserved: 00
        Payload Length: 36
        Transform #: 1
        Transform-Id: ESP_3DES
        Reserved2: 0000
        Life Type: Seconds
        Life Duration (Hex): 70 80
        Life Type: Kilobytes
        Life Duration (Hex): 00 46 50 00
        Encapsulation Mode: Tunnel
        Authentication Algorithm: MD5
  Payload Nonce
    Next Payload: Identification
    Reserved: 00
    Payload Length: 24
    Data:
      67 4f f9 1c ee bf da b1 e0 4a ad eb 89 7f 92 91
      62 15 63 80
  Payload Identification
    Next Payload: Identification
    Reserved: 00
    Payload Length: 12
    ID Type: IPv4 Address (1)
    Protocol ID (UDP/TCP, etc...): 0
    Port: 0
    ID Data: KL-POP
  Payload Identification
    Next Payload: Notification
    Reserved: 00
    Payload Length: 12
    ID Type: IPv4 Address (1)
    Protocol ID (UDP/TCP, etc...): 0
    Port: 0
    ID Data: core01-Loopback2
  Payload Notification
    Next Payload: None
    Reserved: 00
    Payload Length: 28
    DOI: IPsec
    Protocol-ID: PROTO_ISAKMP
    Spi Size: 16
    Notify Type: STATUS_INITIAL_CONTACT
    SPI:
      ac 02 de b5 23 50 bb bf d4 ce ec fe d2 3c c2 81
Oct 07 10:28:07 [IKEv1]: IP = 202.x.x.x, IKE_DECODE RECEIVED Message (msgid=6dd4d20f) with payloads :

HDR + HASH (8) + SA (1) + NONCE (10) + ID (5) + ID (5) + NOTIFY (11) + NONE (0) total length : 184
Oct 07 10:28:07 [IKEv1 DEBUG]: Group =202.x.x.x, IP = 202.x.x.x, processing hash payload
Oct 07 10:28:07 [IKEv1 DEBUG]: Group = 202.x.x.x, IP = 202.x.x.x, processing SA payload
Oct 07 10:28:07 [IKEv1 DEBUG]: Group = 202.x.x.x, IP = 202.x.x.x, processing nonce payload
Oct 07 10:28:07 [IKEv1 DEBUG]: Group = 202.x.x.x, IP = 202.x.x.x, processing ID payload
Oct 07 10:28:07 [IKEv1 DECODE]: Group = 202.x.x.x, IP = 202.x.x.x, ID_IPV4_ADDR ID received 202.x.x.x
Oct 07 10:28:07 [IKEv1]: Group = 202.x.x.x, IP = 202.x.x.x, Received remote Proxy Host data in ID
Payload:  Address 202.176.12.3, Protocol 0, Port 0
Oct 07 10:28:07 [IKEv1 DEBUG]: Group = 202.x.x.x, IP = 202.x.x.x, processing ID payload
Oct 07 10:28:07 [IKEv1 DECODE]: Group = 202.x.x.x, IP = 202.x.x.x, ID_IPV4_ADDR ID received 202.x.x.x
Oct 07 10:28:07 [IKEv1]: Group = 202.x.x.x, IP = 202.x.x.x, Received local Proxy Host data in ID
Payload:  Address 202.78.20.242, Protocol 0, Port 0
Oct 07 10:28:07 [IKEv1 DEBUG]: Group = 202.x.x.x, IP = 202.x.x.x, processing notify payload
Oct 07 10:28:07 [IKEv1]: Group = 202.x.x.x, IP = 202.x.x.x, QM IsRekeyed old sa not found by addr
Oct 07 10:28:07 [IKEv1]: Group = 202.x.x.x, IP = 202.x.x.x, Static Crypto Map check, checking map = vpndes, seq = 10...
Oct 07 10:28:07 [IKEv1]: Group = 202.x.x.x, IP = 202.x.x.x, Static Crypto Map check, map = vpndes,
seq = 10, ACL does not match proxy IDs src:KL-POP dst:core01-Loopback2
Oct 07 10:28:07 [IKEv1]: Group = 202.x.x.x, IP = 202.x.x.x, Static Crypto Map check, checking map = vpndes, seq = 11...
Oct 07 10:28:07 [IKEv1]: Group = 202.x.x.x, IP = 202.x.x.x, Static Crypto Map check, map = vpndes,
seq = 11, ACL does not match proxy IDs src:KL-POP dst:core01-Loopback2
Oct 07 10:28:07 [IKEv1]: Group = 202.x.x.x, IP = 202.x.x.x, Static Crypto Map check, checking map = vpndes, seq = 13...
Oct 07 10:28:07 [IKEv1]: Group = 202.x.x.x, IP = 202.x.x.x, Static Crypto Map check, map = vpndes,
seq = 13, ACL does not match proxy IDs src:KL-POP dst:core01-Loopback2
Oct 07 10:28:07 [IKEv1]: Group = 202.x.x.x, IP = 202.x.x.x, Static Crypto Map check, checking map = vpndes, seq = 20...
Oct 07 10:28:07 [IKEv1]: Group = 202.x.x.x, IP = 202.x.x.x, Static Crypto Map check, map = vpndes, seq = 20, no ACL configured
Oct 07 10:28:07 [IKEv1]: Group = 202.x.x.x, IP = 202.x.x.x, Static Crypto Map check, checking map = vpndes, seq = 22...
Oct 07 10:28:07 [IKEv1]: Group = 202.x.x.x, IP = 202.x.x.x, Static Crypto Map check, map = vpndes, seq = 22, no ACL configured
Oct 07 10:28:07 [IKEv1]: Group = 202.x.x.x, IP = 202.x.x.x, Static Crypto Map check, checking map = vpndes, seq = 45...
Oct 07 10:28:07 [IKEv1]: Group = 202.x.x.x, IP = 202.x.x.x, Static Crypto Map check, map = vpndes, seq = 45, ACL does not match proxy IDs src:KL-POP dst:core01-Loopback2
Oct 07 10:28:07 [IKEv1]: Group = 202.x.x.x, IP = 202.x.x.x, Static Crypto Map check, checking map = vpndes, seq = 88...
Oct 07 10:28:07 [IKEv1]: Group = 202.x.x.x, IP = 202.x.x.x, Static Crypto Map check, map = vpndes, seq = 88, ACL does not match proxy IDs src:KL-POP dst:core01-Loopback2
Oct 07 10:28:07 [IKEv1]: Group = 202.x.x.x, IP = 202.x.x.x, Static Crypto Map check, checking map = vpndes, seq = 103...
Oct 07 10:28:07 [IKEv1]: Group = 202.x.x.x, IP = 202.x.x.x, Static Crypto Map check, map vpndes, seq = 103 is a successful match
Oct 07 10:28:07 [IKEv1]: Group = 202.x.x.x, IP = 202.x.x.x, IKE Remote Peer configured for crypto map: vpndes
Oct 07 10:28:07 [IKEv1 DEBUG]: Group = 202.x.x.x, IP = 202.x.x.x, processing IPSec SA payload
Oct 07 10:28:07 [IKEv1 DEBUG]: Group = 202.x.x.x, IP = 202.x.x.x, IPSec SA Proposal # 1, Transform # 1 acceptable  Matches global IPSec SA entry # 103
Oct 07 10:28:07 [IKEv1]: Group = 202.x.x.x, IP = 202.x.x.x IKE: requesting SPI!
Oct 07 10:28:07 [IKEv1 DEBUG]: Group = 202.x.x.x, IP = 202.x.x.x, IKE got SPI from key engine: SPI = 0xd8f228fb
Oct 07 10:28:07 [IKEv1 DEBUG]: Group = 202.x.x.x, IP = 202.x.x.x, oakley constucting quick mode
Oct 07 10:28:07 [IKEv1 DEBUG]: Group = 202.x.x.x, IP = 202.x.x.x, constructing blank hash payload
Oct 07 10:28:07 [IKEv1 DEBUG]: Group = 202.x.x.x, IP = 202.x.x.x, constructing IPSec SA payload
Oct 07 10:28:07 [IKEv1 DEBUG]: Group = 202.x.x.x, IP = 202.x.x.x2, constructing IPSec nonce payload
Oct 07 10:28:07 [IKEv1 DEBUG]: Group = 202.x.x.x, IP = 202.x.x.x, constructing proxy ID
Oct 07 10:28:07 [IKEv1 DEBUG]: Group = 202.x.x.x, IP = 202.x.x.x, Transmitting Proxy Id:
  Remote host: 202.x.x.x  Protocol 0  Port 0
  Local host:  202.x.x.x  Protocol 0  Port 0
Oct 07 10:28:07 [IKEv1 DEBUG]: Group = 202.x.x.x, IP = 202.x.x.x, constructing qm hash payload
Oct 07 10:28:07 [IKEv1 DECODE]: Group = 202.x.x.x, IP = 202.x.x.x, IKE Responder sending 2nd QM pkt: msg id = 6dd4d20f
Oct 07 10:28:07 [IKEv1]: IP = 202.x.x.x, IKE_DECODE SENDING Message (msgid=6dd4d20f) with payloads : HDR

+ HASH (8) + SA (1) + NONCE (10) + ID (5) + ID (5) + NONE (0) total length : 156

BEFORE ENCRYPTION
RAW PACKET DUMP on SEND
ac 02 de b5 23 50 bb bf d4 ce ec fe d2 3c c2 81    |  ....#P.......<..
08 10 20 00 0f d2 d4 6d 1c 00 00 00 01 00 00 14    |  .. ....m........
2e 99 8f a9 8a 3f 0a 93 eb be d1 c1 8a 9e 5c 34    |  .....?........\4
0a 00 00 3c 00 00 00 01 00 00 00 01 00 00 00 30    |  ...<...........0
01 03 04 01 d8 f2 28 fb 00 00 00 24 01 03 00 00    |  ......(....$....
80 01 00 01 80 02 70 80 80 01 00 02 00 02 00 04    |  ......p.........
00 46 50 00 80 04 00 01 80 05 00 01 05 00 00 18    |  .FP.............
78 13 f7 fa 8c 32 f7 51 56 09 1a 3d 3b 44 28 92    |  x....2.QV..=;D(.
77 cf 0a 56 05 00 00 0c 01 00 00 00 ca b0 0c 03    |  w..V............
00 00 00 0c 01 00 00 00 ca 4e 14 f2                |  .........N..

ISAKMP Header
  Initiator COOKIE: ac 02 de b5 23 50 bb bf
  Responder COOKIE: d4 ce ec fe d2 3c c2 81
  Next Payload: Hash
  Version: 1.0
  Exchange Type: Quick Mode
  Flags: (none)
  MessageID: 0FD2D46D
  Length: 469762048
  Payload Hash
    Next Payload: Security Association
    Reserved: 00
    Payload Length: 20
    Data:
      2e 99 8f a9 8a 3f 0a 93 eb be d1 c1 8a 9e 5c 34
  Payload Security Association
    Next Payload: Nonce
    Reserved: 00
    Payload Length: 60
    DOI: IPsec
    Situation:(SIT_IDENTITY_ONLY)
    Payload Proposal
      Next Payload: None
      Reserved: 00
      Payload Length: 48
      Proposal #: 1
      Protocol-Id: PROTO_IPSEC_ESP
      SPI Size: 4
      # of transforms: 1
      SPI: d8 f2 28 fb
      Payload Transform
        Next Payload: None
        Reserved: 00
        Payload Length: 36
        Transform #: 1
        Transform-Id: ESP_3DES
        Reserved2: 0000
        Life Type: Seconds
        Life Duration (Hex): 70 80
        Life Type: Kilobytes
        Life Duration (Hex): 00 46 50 00
        Encapsulation Mode: Tunnel
        Authentication Algorithm: MD5
  Payload Nonce
    Next Payload: Identification
    Reserved: 00
    Payload Length: 24
    Data:
      78 13 f7 fa 8c 32 f7 51 56 09 1a 3d 3b 44 28 92
      77 cf 0a 56
  Payload Identification
    Next Payload: Identification
    Reserved: 00
    Payload Length: 12
    ID Type: IPv4 Address (1)
    Protocol ID (UDP/TCP, etc...): 0
    Port: 0
    ID Data: KL-POP
  Payload Identification
    Next Payload: None
    Reserved: 00
    Payload Length: 12
    ID Type: IPv4 Address (1)
    Protocol ID (UDP/TCP, etc...): 0
    Port: 0
    ID Data: core01-Loopback2

ISAKMP Header
  Initiator COOKIE: ac 02 de b5 23 50 bb bf
  Responder COOKIE: d4 ce ec fe d2 3c c2 81
  Next Payload: Hash
  Version: 1.0
  Exchange Type: Quick Mode
  Flags: (Encryption)
  MessageID: 6DD4D20F
  Length: 156


IKE Recv RAW packet dump
ac 02 de b5 23 50 bb bf d4 ce ec fe d2 3c c2 81    |  ....#P.......<..
08 10 20 01 6d d4 d2 0f 00 00 00 4c 94 64 33 cf    |  .. .m......L.d3.
a1 6b 7d 9c e8 4c 6c d3 d5 31 b1 8a fe cb ae 93    |  .k}..Ll..1......
44 f5 53 84 51 0d 3e c4 7c 86 c7 62 ca 73 ae 4d    |  D.S.Q.>.|..b.s.M
8d a8 e5 b7 03 75 b7 f6 b0 5e c9 63                |  .....u...^.c

 RECV PACKET from 202.x.x.x
ISAKMP Header
  Initiator COOKIE: ac 02 de b5 23 50 bb bf
  Responder COOKIE: d4 ce ec fe d2 3c c2 81
  Next Payload: Hash
  Version: 1.0
  Exchange Type: Quick Mode
  Flags: (Encryption)
  MessageID: 6DD4D20F
  Length: 76

AFTER DECRYPTION
ISAKMP Header
  Initiator COOKIE: ac 02 de b5 23 50 bb bf
  Responder COOKIE: d4 ce ec fe d2 3c c2 81
  Next Payload: Hash
  Version: 1.0
  Exchange Type: Quick Mode
  Flags: (Encryption)
  MessageID: 6DD4D20F
  Length: 76
  Payload Hash
    Next Payload: None
    Reserved: 00
    Payload Length: 20
    Data:
      00 6b 6b bd aa 80 00 05 84 40 9b 93 2f f8 e8 50
Oct 07 10:28:07 [IKEv1]: IP = 202.x.x.x, IKE_DECODE RECEIVED Message (msgid=6dd4d20f) with payloads :

HDR + HASH (8) + NONE (0) total length : 48
Oct 07 10:28:07 [IKEv1 DEBUG]: Group = 202.x.x.x, IP = 202.x.x.x, processing hash payload
Oct 07 10:28:07 [IKEv1 DEBUG]: Group = 202.x.x.x, IP = 202.x.x.x, loading all IPSEC SAs
Oct 07 10:28:07 [IKEv1 DEBUG]: Group = 202.x.x.x, IP = 202.x.x.x, Generating Quick Mode Key!
Oct 07 10:28:07 [IKEv1 DEBUG]: Group = 202.x.x.x, IP = 202.x.x.x, NP encrypt rule look up for crypto map vpndes 103 matching ACL Tu4: returned cs_id=cf59b008; rule=ce261a70
Oct 07 10:28:07 [IKEv1 DEBUG]: Group = 202.x.x.x, IP = 202.x.x.x, Generating Quick Mode Key!
Oct 07 10:28:07 [IKEv1 DEBUG]: Group = 202.x.x.x, IP = 202.x.x.x, NP encrypt rule look up for crypto

map vpndes 103 matching ACL Tu4: returned cs_id=cf59b008; rule=ce261a70
Oct 07 10:28:07 [IKEv1]: Group = 202.x.x.x, IP = 202.x.x.x, Security negotiation complete for LAN-
to-LAN Group (202.x.x.x)  Responder, Inbound SPI = 0xd8f228fb, Outbound SPI = 0xb7526326
Oct 07 10:28:07 [IKEv1 DEBUG]: Group = 202.x.x.x, IP = 202.x.x.x, IKE got a KEY_ADD msg for SA: SPI = 0xb7526326
Oct 07 10:28:07 [IKEv1 DEBUG]: Group = 202.x.x.x, IP = 202.x.x.x, Pitcher: received KEY_UPDATE, spi
0xd8f228fb
Oct 07 10:28:07 [IKEv1 DEBUG]: Group = 202.x.x.x, IP = 202.x.x.x, Starting P2 rekey timer: 24480 seconds.
Oct 07 10:28:07 [IKEv1]: Group = 202.x.x.x, IP = 202.x.x.x PHASE 2 COMPLETED (msgid=6dd4d20f)