Saturday, May 19, 2018

Troubleshooting Failover on Cisco Active-Standby ASA Firewall

I've encountered failover flapping between an Active and Standby Cisco ASA firewalls which caused an IPSec VPN tunnels to go down. You'll see console messages that cycles between Failover LAN Failed and then Failover LAN became OK on both Active and Standby firewalls. There's also increasing input and CRC errors and interface resets on the failover interface which indicates a cable problem.

Here's a nice PDF troubleshooting guide from Cisco. Although it was written for the PIX firewall, some troubleshooting scenarios and steps are still applicable with the ASA firewall.


ciscoasa/pri/act#
Failover LAN Failed
Failover LAN became OK
Switchover enabled
Failover LAN Failed
Failover LAN became OK
Switchover enabled
Failover LAN Failed
Failover LAN became OK
Switchover enabled

ciscoasa/pri/act# show interface GigabitEthernet0/7
Interface GigabitEthernet0/7 "FAILOVER", is up, line protocol is up
  Hardware is i82574L rev00, BW 1000 Mbps, DLY 10 usec
    Auto-Duplex(Full-duplex), Auto-Speed(1000 Mbps)
    Input flow control is unsupported, output flow control is off
    Description: LAN FAILOVER
    MAC address 4c4e.35ea.ddb7, MTU 1500
    IP address 172.30.5.1, subnet mask 255.255.255.252
    1563433 packets input, 141470716 bytes, 0 no buffer
    Received 2141 broadcasts, 0 runts, 0 giants
    5238 input errors, 5238 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
    0 pause input, 0 resume input
    0 L2 decode drops
    1583757 packets output, 151416596 bytes, 0 underruns
    0 pause output, 0 resume output
    0 output errors, 0 collisions, 477 interface resets
    0 late collisions, 0 deferred
    0 input reset drops, 0 output reset drops
    input queue (blocks free curr/low): hardware (487/461)
    output queue (blocks free curr/low): hardware (508/446)
  Traffic Statistics for "folan":
    1563151 packets input, 112914310 bytes
    1584569 packets output, 122502340 bytes
    0 packets dropped
      1 minute input rate 0 pkts/sec,  23 bytes/sec
      1 minute output rate 0 pkts/sec,  26 bytes/sec 
      1 minute drop rate, 0 pkts/sec
      5 minute input rate 0 pkts/sec,  23 bytes/sec
      5 minute output rate 0 pkts/sec,  26 bytes/sec
      5 minute drop rate, 0 pkts/sec


Notice the show failover on the Standby firewall displays a short Active time (56 sec).

ciscoasa/sec/stby# show failover
Failover On
Failover unit Secondary
Failover LAN Interface: FAILOVER GigabitEthernet0/7 (up)
Unit Poll frequency 1 seconds, holdtime 15 seconds
Interface Poll frequency 5 seconds, holdtime 25 seconds
Interface Policy 1
Monitored Interfaces 3 of 216 maximum
Version: Ours 9.1(7)13, Mate 9.1(7)13
Last Failover at: 10:37:46 SA Oct 17 2017
    This host: Secondary - Standby Ready
        Active time: 56 (sec)
        slot 0: ASA5525 hw/sw rev (1.0/9.1(7)13) status (Up Sys)
          Interface outside (62.2.4.7): Normal (Monitored)
          Interface inside (172.30.0.5): Normal (Monitored)
          Interface management (172.30.0.8): Normal (Monitored)
    Other host: Primary - Active
        Active time: 1063719 (sec)
        slot 0: ASA5525 hw/sw rev (1.0/9.1(7)13) status (Up Sys)
          Interface outside (62.2.4.8): Normal (Monitored)
          Interface inside (172.30.0.3): Normal (Monitored)
          Interface management (172.30.0.7): Normal (Monitored)

Stateful Failover Logical Update Statistics
    Link : folink GigabitEthernet0/6 (up)
    Stateful Obj     xmit       xerr       rcv        rerr     
    General        137696     0          2372002    57       
    sys cmd      137696     0          137654     57       
    up time      0          0          0          0        
    RPC services      0          0          0          0        
    TCP conn     0          0          215467     0        
    UDP conn     0          0          1829472    0        
    ARP tbl      0          0          186949     0        
    Xlate_Timeout      0          0          0          0        
    IPv6 ND tbl      0          0          0          0        
    VPN IKEv1 SA     0          0          474        0        
    VPN IKEv1 P2     0          0          1978       0        
    VPN IKEv2 SA     0          0          0          0        
    VPN IKEv2 P2     0          0          0          0        
    VPN CTCP upd     0          0          0          0        
    VPN SDI upd     0          0          0          0        
    VPN DHCP upd     0          0          0          0        
    SIP Session     0          0          0          0        
    Route Session     0          0          0          0        
    User-Identity     0          0          8          0        
    CTS SGTNAME     0          0          0          0        
    CTS PAC     0          0          0          0        
    TrustSec-SXP     0          0          0          0        
    IPv6 Route     0          0          0          0        
    Logical Update Queue Information
              Cur     Max     Total
    Recv Q:     0     25     6737580
    Xmit Q:     0     1     137695


The syslogs also showed flapping Failover interface.

ciscoasa/sec/stby# show log

<SNIP>

Oct 17 2017 10:57:27 ciscoasa : %ASA-1-105043: (Secondary) Failover interface failed
Oct 17 2017 10:57:27 ciscoasa : %ASA-6-720032: (VPN-Secondary) HA status callback: id=3,seq=200,grp=0,event=401,op=0,my=Standby Ready,peer=Active.
Oct 17 2017 10:57:27 ciscoasa : %ASA-6-720024: (VPN-Secondary) HA status callback: Control channel is down.
Oct 17 2017 10:57:27 ciscoasa : %ASA-6-721002: (WebVPN-Secondary) HA status change: event HA_STATUS_PEER_CTL_COMM, my state Standby Ready, peer state Active.
Oct 17 2017 10:57:33 ciscoasa : %ASA-6-720032: (VPN-Secondary) HA status callback: id=3,seq=200,grp=0,event=401,op=1,my=Standby Ready,peer=Active.
Oct 17 2017 10:57:33 ciscoasa : %ASA-6-720024: (VPN-Secondary) HA status callback: Control channel is up.
Oct 17 2017 10:57:33 ciscoasa : %ASA-6-721002: (WebVPN-Secondary) HA status change: event HA_STATUS_PEER_CTL_COMM, my state Standby Ready, peer state Active.
Oct 17 2017 10:57:33 ciscoasa : %ASA-6-720032: (VPN-Secondary) HA status callback: id=3,seq=200,grp=0,event=411,op=61,my=Standby Ready,peer=Active.
Oct 17 2017 10:57:33 ciscoasa : %ASA-6-721002: (WebVPN-Secondary) HA status change: event HA_STATUS_CLIENT_NEGOTIATED_VERSION, my state Standby Ready, peer state Active.
Oct 17 2017 10:57:34 ciscoasa : %ASA-1-105042: (Secondary) Failover interface OK
Oct 17 2017 10:57:42 ciscoasa : %ASA-1-105003: (Secondary) Monitoring on interface outside waiting
Oct 17 2017 10:57:42 ciscoasa : %ASA-1-105003: (Secondary) Monitoring on interface inside waiting
Oct 17 2017 10:57:42ciscoasa : %ASA-1-105003: (Secondary) Monitoring on interface management waiting
Oct 17 2017 10:57:57 ciscoasa : %ASA-1-105004: (Secondary) Monitoring on interface outside normal
Oct 17 2017 10:57:57 ciscoasa : %ASA-1-105004: (Secondary) Monitoring on interface inside normal
Oct 17 2017 10:58:07 ciscoasa : %ASA-1-105004: (Secondary) Monitoring on interface management normal
Oct 17 2017 11:03:16 ciscoasa: %ASA-1-105043: (Secondary) Failover interface failed


ciscoasa/sec/stby# Failover LAN Failed
Failover LAN became OK
Switchover enabled
Failover LAN Failed
Failover LAN became OK
Switchover enabled
Failover LAN Failed
Failover LAN became OK
Switchover enabled


You can use the show failover command to verify ASA failover pair operation and activity/history. The show failover state indicates an Interface failure.

ciscoasa/sec/stby# show failover ?

  descriptor  Show failover interface descriptors. Two numbers are shown for
              each interface. When exchanging information regarding a
              particular interface, this unit uses the first number in messages
              it sends to its peer. And it expects the second number in
              messages it receives from its peer. For trouble shooting, collect
              the show output from both units and verify that the numbers
              match.
  exec        Show failover command execution information
  history     Show failover switching history
  interface   Show failover command interface information
  state       Show failover internal state information
  statistics  Show failover command interface statistics information
  |           Output modifiers
  <cr>

ciscoasa/sec/stby# show failover state

               State          Last Failure Reason      Date/Time
This host  -   Secondary
               Standby Ready  Ifc Failure              04:42:50 SA Oct 17 2017
Other host -   Primary
               Active         Comm Failure             10:37:48 SA Oct 17 2017

====Configuration State===
    Sync Done - STANDBY
====Communication State===
    Mac set

ciscoasa/sec/stby# show failover history
==========================================================================
From State                 To State                   Reason
==========================================================================
00:41:34 SA Oct 17 2017
Bulk Sync                  Standby Ready              No Error

04:42:50 SA Oct 17 2017
Standby Ready              Failed                     Interface check

04:42:52 SA Oct 17 2017
Failed                     Standby Ready              Interface check

10:37:46 SA Oct 17 2017
Standby Ready              Just Active                HELLO not heard from mate

10:37:46 SA Oct 17 2017
Just Active                Active Drain               HELLO not heard from mate

10:37:46 SA Oct 17 2017
Active Drain               Active Applying Config     HELLO not heard from mate

10:37:46 SA Oct 17 2017
Active Applying Config     Active Config Applied      HELLO not heard from mate

10:37:46 SA Oct 17 2017

Active Config Applied      Active                     HELLO not heard from mate

10:38:42 SA Oct 17 2017
Active                     Cold Standby               Failover state check

10:38:44 SA Oct 17 2017
Cold Standby               Sync Config                Failover state check

10:38:51 SA Oct 17 2017
Sync Config                Sync File System           Failover state check

10:38:51 SA Oct 17 2017
Sync File System           Bulk Sync                  Failover state check

10:39:10 SA Oct 17 2017
Bulk Sync                  Standby Ready              Failover state check

12:14:06 SA Oct 17 2017
Standby Ready              Disabled                   Set by the config command   // DISABLED FAILOVER

16:09:19 SA Oct 17 2017
Disabled                   Negotiation                Set by the config command

16:09:20 SA Oct 17 2017

Negotiation                Cold Standby               Detected an Active mate

16:09:22 SA Oct 17 2017
Cold Standby               Sync Config                Detected an Active mate

16:09:29 SA Oct 17 2017
Sync Config                Sync File System           Detected an Active mate

16:09:29 SA Oct 17 2017
Sync File System           Bulk Sync                  Detected an Active mate

16:09:44 SA Oct 17 2017
Bulk Sync                  Standby Ready              Detected an Active mate

==========================================================================


I've temporarily disabled failover on the Standby firewall using the no failover command. The flapping had stopped and the hostname displayed NoFailover.

ciscoasa/sec/stby# show run failover
failover
failover lan unit secondary
failover lan interface FAILOVER GigabitEthernet0/7
failover interface ip FAILOVER 172.30.5.1 255.255.255.252 standby 172.30.5.2
ciscoasa/sec/stby# configure terminal
**** WARNING ****
        Configuration Replication is NOT performed from Standby unit to Active unit.
        Configurations are no longer synchronized.
ciscoasa/sec/stby(config)# no failover
INFO: This unit is currently in standby state. By disabling failover, this unit will remain in standby state.

ciscoasa/sec/stbyNoFailover# show interface GigabitEthernet0/7
Interface GigabitEthernet0/7 "FAILOVER", is up, line protocol is up
  Hardware is i82574L rev00, BW 1000 Mbps, DLY 10 usec
    Auto-Duplex(Full-duplex), Auto-Speed(1000 Mbps)
    Input flow control is unsupported, output flow control is off
    Description: LAN FAILOVER
    MAC address f07f.0645.12bd, MTU 1500
    IP address 172.30.5.2, subnet mask 255.255.255.252
    1581714 packets input, 151114208 bytes, 0 no buffer
    Received 11527 broadcasts, 0 runts, 0 giants
    506 input errors, 506 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
    0 pause input, 0 resume input
    0 L2 decode drops
    1591204 packets output, 144014372 bytes, 0 underruns
    0 pause output, 0 resume output
    0 output errors, 0 collisions, 481 interface resets
    0 late collisions, 0 deferred
    11 input reset drops, 0 output reset drops
    input queue (blocks free curr/low): hardware (475/461)
    output queue (blocks free curr/low): hardware (453/447)
  Traffic Statistics for "FAILOVER":
    23352 packets input, 1474664 bytes
    15992 packets output, 1061134 bytes
    0 packets dropped
      1 minute input rate 0 pkts/sec,  35 bytes/sec
      1 minute output rate 0 pkts/sec,  14 bytes/sec     
      1 minute drop rate, 0 pkts/sec
      5 minute input rate 0 pkts/sec,  35 bytes/sec
      5 minute output rate 0 pkts/sec,  14 bytes/sec
      5 minute drop rate, 0 pkts/sec


I swapped the cable (straight cable) between the ASA G0/7 ports (failover link) and cleared the interface counters.

ciscoasa/pri/act# clear interface ?

Current available interface(s):
  GigabitEthernet  GigabitEthernet IEEE 802.3z
  Management       Management interface
  Port-channel     Ethernet Channel of interfaces
  Redundant        Redundant Interface
  inside           Name of interface GigabitEthernet0/1
  management       Name of interface Management0/0
  outside          Name of interface GigabitEthernet0/0
  <cr>

ciscoasa/pri/act# clear interface GigabitEthernet0/7

ciscoasa/pri/act# show interface GigabitEthernet0/7
Interface GigabitEthernet0/7 "folan", is up, line protocol is up
  Hardware is i82574L rev00, BW 1000 Mbps, DLY 10 usec
    Auto-Duplex(Full-duplex), Auto-Speed(1000 Mbps)
    Input flow control is unsupported, output flow control is off
    Description: LAN FAILOVER
    MAC address 4c4e.35ea.ddb7, MTU 1500
    IP address 172.30.5.1, subnet mask 255.255.255.252
    2014 packets input, 177860 bytes, 0 no buffer
    Received 1 broadcasts, 0 runts, 0 giants
    0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
    0 pause input, 0 resume input
    0 L2 decode drops
    2093 packets output, 214728 bytes, 0 underruns
    0 pause output, 0 resume output
    0 output errors, 0 collisions, 0 interface resets
    0 late collisions, 0 deferred
    0 input reset drops, 0 output reset drops
    input queue (blocks free curr/low): hardware (495/459)
    output queue (blocks free curr/low): hardware (454/442)
  Traffic Statistics for "folan":
    2014 packets input, 141204 bytes
    2093 packets output, 172924 bytes
    0 packets dropped
      1 minute input rate 1 pkts/sec,  104 bytes/sec
      1 minute output rate 1 pkts/sec,  112 bytes/sec
      1 minute drop rate, 0 pkts/sec
      5 minute input rate 1 pkts/sec,  105 bytes/sec
      5 minute output rate 1 pkts/sec,  113 bytes/sec
      5 minute drop rate, 0 pkts/sec


ciscoasa/sec/stbyNoFailover# clear interface GigabitEthernet0/7

ciscoasa/sec/stbyNoFailover# show interface GigabitEthernet0/7
Interface GigabitEthernet0/7 "FAILOVER", is up, line protocol is up
  Hardware is i82574L rev00, BW 1000 Mbps, DLY 10 usec
    Auto-Duplex(Full-duplex), Auto-Speed(1000 Mbps)
    Input flow control is unsupported, output flow control is off
    Description: LAN FAILOVER
    MAC address f07f.0645.12bd, MTU 1500
    IP address 172.30.5.2, subnet mask 255.255.255.252
    255 packets input, 16320 bytes, 0 no buffer
    Received 153 broadcasts, 0 runts, 0 giants
    0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
    0 pause input, 0 resume input
    0 L2 decode drops
    153 packets output, 9792 bytes, 0 underruns
    0 pause output, 0 resume output
    0 output errors, 0 collisions, 0 interface resets
    0 late collisions, 0 deferred
    0 input reset drops, 0 output reset drops
    input queue (blocks free curr/low): hardware (468/461)
    output queue (blocks free curr/low): hardware (495/447)
  Traffic Statistics for "folan":
    255 packets input, 10914 bytes
    153 packets output, 4284 bytes
    0 packets dropped
      1 minute input rate 0 pkts/sec,  35 bytes/sec
      1 minute output rate 0 pkts/sec,  14 bytes/sec
      1 minute drop rate, 0 pkts/sec
      5 minute input rate 0 pkts/sec,  28 bytes/sec
      5 minute output rate 0 pkts/sec,  11 bytes/sec
      5 minute drop rate, 0 pkts/sec


I didn't observe incrementing input and CRC errors after the cable swap, so I've enabled failover back on the Standbyfirewall.

ciscoasa/sec/stbyNoFailover# configure terminal
ciscoasa/sec/stbyNoFailover(config)# failover   // SSH SESSION HANGED; NEED RE-LOGIN

No comments:

Post a Comment