I've encountered failover flapping between an Active and Standby Cisco ASA firewalls which caused an IPSec VPN tunnels to go down. You'll see console messages that cycles between Failover LAN Failed and then Failover LAN became OK on both Active and Standby firewalls. There's also increasing input and CRC errors and interface resets on the failover interface which indicates a cable problem.
Here's a nice PDF troubleshooting guide from Cisco. Although it was written for the PIX firewall, some troubleshooting scenarios and steps are still applicable with the ASA firewall.
ciscoasa/pri/act#
Failover LAN Failed
Failover LAN became OK
Switchover enabled
Failover LAN Failed
Failover LAN became OK
Switchover enabled
Failover LAN Failed
Failover LAN became OK
Switchover enabled
ciscoasa/pri/act# show interface GigabitEthernet0/7
Interface GigabitEthernet0/7 "FAILOVER", is up, line protocol is up
Hardware is i82574L rev00, BW 1000 Mbps, DLY 10 usec
Auto-Duplex(Full-duplex), Auto-Speed(1000 Mbps)
Input flow control is unsupported, output flow control is off
Description: LAN FAILOVER
MAC address 4c4e.35ea.ddb7, MTU 1500
IP address 172.30.5.1, subnet mask 255.255.255.252
1563433 packets input, 141470716 bytes, 0 no buffer
Received 2141 broadcasts, 0 runts, 0 giants
5238 input errors, 5238 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
0 pause input, 0 resume input
0 L2 decode drops
1583757 packets output, 151416596 bytes, 0 underruns
0 pause output, 0 resume output
0 output errors, 0 collisions, 477 interface resets
0 late collisions, 0 deferred
0 input reset drops, 0 output reset drops
input queue (blocks free curr/low): hardware (487/461)
output queue (blocks free curr/low): hardware (508/446)
Traffic Statistics for "folan":
1563151 packets input, 112914310 bytes
1584569 packets output, 122502340 bytes
0 packets dropped
1 minute input rate 0 pkts/sec, 23 bytes/sec
1 minute output rate 0 pkts/sec, 26 bytes/sec
1 minute drop rate, 0 pkts/sec
5 minute input rate 0 pkts/sec, 23 bytes/sec
5 minute output rate 0 pkts/sec, 26 bytes/sec
5 minute drop rate, 0 pkts/sec
Notice the show failover on the Standby firewall displays a short Active time (56 sec).
ciscoasa/sec/stby# show failover
Failover On
Failover unit Secondary
Failover LAN Interface: FAILOVER GigabitEthernet0/7 (up)
Unit Poll frequency 1 seconds, holdtime 15 seconds
Interface Poll frequency 5 seconds, holdtime 25 seconds
Interface Policy 1
Monitored Interfaces 3 of 216 maximum
Version: Ours 9.1(7)13, Mate 9.1(7)13
Last Failover at: 10:37:46 SA Oct 17 2017
This host: Secondary - Standby Ready
Active time: 56 (sec)
slot 0: ASA5525 hw/sw rev (1.0/9.1(7)13) status (Up Sys)
Interface outside (62.2.4.7): Normal (Monitored)
Interface inside (172.30.0.5): Normal (Monitored)
Interface management (172.30.0.8): Normal (Monitored)
Other host: Primary - Active
Active time: 1063719 (sec)
slot 0: ASA5525 hw/sw rev (1.0/9.1(7)13) status (Up Sys)
Interface outside (62.2.4.8): Normal (Monitored)
Interface inside (172.30.0.3): Normal (Monitored)
Interface management (172.30.0.7): Normal (Monitored)
Stateful Failover Logical Update Statistics
Link : folink GigabitEthernet0/6 (up)
Stateful Obj xmit xerr rcv rerr
General 137696 0 2372002 57
sys cmd 137696 0 137654 57
up time 0 0 0 0
RPC services 0 0 0 0
TCP conn 0 0 215467 0
UDP conn 0 0 1829472 0
ARP tbl 0 0 186949 0
Xlate_Timeout 0 0 0 0
IPv6 ND tbl 0 0 0 0
VPN IKEv1 SA 0 0 474 0
VPN IKEv1 P2 0 0 1978 0
VPN IKEv2 SA 0 0 0 0
VPN IKEv2 P2 0 0 0 0
VPN CTCP upd 0 0 0 0
VPN SDI upd 0 0 0 0
VPN DHCP upd 0 0 0 0
SIP Session 0 0 0 0
Route Session 0 0 0 0
User-Identity 0 0 8 0
CTS SGTNAME 0 0 0 0
CTS PAC 0 0 0 0
TrustSec-SXP 0 0 0 0
IPv6 Route 0 0 0 0
Logical Update Queue Information
Cur Max Total
Recv Q: 0 25 6737580
Xmit Q: 0 1 137695
The syslogs also showed flapping Failover interface.
ciscoasa/sec/stby# show log
<SNIP>
Oct 17 2017 10:57:27 ciscoasa : %ASA-1-105043: (Secondary) Failover interface failed
Oct 17 2017 10:57:27 ciscoasa : %ASA-6-720032: (VPN-Secondary) HA status callback: id=3,seq=200,grp=0,event=401,op=0,my=Standby Ready,peer=Active.
Oct 17 2017 10:57:27 ciscoasa : %ASA-6-720024: (VPN-Secondary) HA status callback: Control channel is down.
Oct 17 2017 10:57:27 ciscoasa : %ASA-6-721002: (WebVPN-Secondary) HA status change: event HA_STATUS_PEER_CTL_COMM, my state Standby Ready, peer state Active.
Oct 17 2017 10:57:33 ciscoasa : %ASA-6-720032: (VPN-Secondary) HA status callback: id=3,seq=200,grp=0,event=401,op=1,my=Standby Ready,peer=Active.
Oct 17 2017 10:57:33 ciscoasa : %ASA-6-720024: (VPN-Secondary) HA status callback: Control channel is up.
Oct 17 2017 10:57:33 ciscoasa : %ASA-6-721002: (WebVPN-Secondary) HA status change: event HA_STATUS_PEER_CTL_COMM, my state Standby Ready, peer state Active.
Oct 17 2017 10:57:33 ciscoasa : %ASA-6-720032: (VPN-Secondary) HA status callback: id=3,seq=200,grp=0,event=411,op=61,my=Standby Ready,peer=Active.
Oct 17 2017 10:57:33 ciscoasa : %ASA-6-721002: (WebVPN-Secondary) HA status change: event HA_STATUS_CLIENT_NEGOTIATED_VERSION, my state Standby Ready, peer state Active.
Oct 17 2017 10:57:34 ciscoasa : %ASA-1-105042: (Secondary) Failover interface OK
Oct 17 2017 10:57:42 ciscoasa : %ASA-1-105003: (Secondary) Monitoring on interface outside waiting
Oct 17 2017 10:57:42 ciscoasa : %ASA-1-105003: (Secondary) Monitoring on interface inside waiting
Oct 17 2017 10:57:42ciscoasa : %ASA-1-105003: (Secondary) Monitoring on interface management waiting
Oct 17 2017 10:57:57 ciscoasa : %ASA-1-105004: (Secondary) Monitoring on interface outside normal
Oct 17 2017 10:57:57 ciscoasa : %ASA-1-105004: (Secondary) Monitoring on interface inside normal
Oct 17 2017 10:58:07 ciscoasa : %ASA-1-105004: (Secondary) Monitoring on interface management normal
Oct 17 2017 11:03:16 ciscoasa: %ASA-1-105043: (Secondary) Failover interface failed
ciscoasa/sec/stby# Failover LAN Failed
Failover LAN became OK
Switchover enabled
Failover LAN Failed
Failover LAN became OK
Switchover enabled
Failover LAN Failed
Failover LAN became OK
Switchover enabled
You can use the show failover command to verify ASA failover pair operation and activity/history. The show failover state indicates an Interface failure.
ciscoasa/sec/stby# show failover ?
descriptor Show failover interface descriptors. Two numbers are shown for
each interface. When exchanging information regarding a
particular interface, this unit uses the first number in messages
it sends to its peer. And it expects the second number in
messages it receives from its peer. For trouble shooting, collect
the show output from both units and verify that the numbers
match.
exec Show failover command execution information
history Show failover switching history
interface Show failover command interface information
state Show failover internal state information
statistics Show failover command interface statistics information
| Output modifiers
<cr>
ciscoasa/sec/stby# show failover state
State Last Failure Reason Date/Time
This host - Secondary
Standby Ready Ifc Failure 04:42:50 SA Oct 17 2017
Other host - Primary
Active Comm Failure 10:37:48 SA Oct 17 2017
====Configuration State===
Sync Done - STANDBY
====Communication State===
Mac set
ciscoasa/sec/stby# show failover history
==========================================================================
From State To State Reason
==========================================================================
00:41:34 SA Oct 17 2017
Bulk Sync Standby Ready No Error
04:42:50 SA Oct 17 2017
Standby Ready Failed Interface check
04:42:52 SA Oct 17 2017
Failed Standby Ready Interface check
10:37:46 SA Oct 17 2017
Standby Ready Just Active HELLO not heard from mate
10:37:46 SA Oct 17 2017
Just Active Active Drain HELLO not heard from mate
10:37:46 SA Oct 17 2017
Active Drain Active Applying Config HELLO not heard from mate
10:37:46 SA Oct 17 2017
Active Applying Config Active Config Applied HELLO not heard from mate
10:37:46 SA Oct 17 2017
Active Config Applied Active HELLO not heard from mate
10:38:42 SA Oct 17 2017
Active Cold Standby Failover state check
10:38:44 SA Oct 17 2017
Cold Standby Sync Config Failover state check
10:38:51 SA Oct 17 2017
Sync Config Sync File System Failover state check
10:38:51 SA Oct 17 2017
Sync File System Bulk Sync Failover state check
10:39:10 SA Oct 17 2017
Bulk Sync Standby Ready Failover state check
12:14:06 SA Oct 17 2017
Standby Ready Disabled Set by the config command // DISABLED FAILOVER
16:09:19 SA Oct 17 2017
Disabled Negotiation Set by the config command
16:09:20 SA Oct 17 2017
Negotiation Cold Standby Detected an Active mate
16:09:22 SA Oct 17 2017
Cold Standby Sync Config Detected an Active mate
16:09:29 SA Oct 17 2017
Sync Config Sync File System Detected an Active mate
16:09:29 SA Oct 17 2017
Sync File System Bulk Sync Detected an Active mate
16:09:44 SA Oct 17 2017
Bulk Sync Standby Ready Detected an Active mate
==========================================================================
I've temporarily disabled failover on the Standby firewall using the no failover command. The flapping had stopped and the hostname displayed NoFailover.
ciscoasa/sec/stby# show run failover
failover
failover lan unit secondary
failover lan interface FAILOVER GigabitEthernet0/7
failover interface ip FAILOVER 172.30.5.1 255.255.255.252 standby 172.30.5.2
ciscoasa/sec/stby# configure terminal
**** WARNING ****
Configuration Replication is NOT performed from Standby unit to Active unit.
Configurations are no longer synchronized.
ciscoasa/sec/stby(config)# no failover
INFO: This unit is currently in standby state. By disabling failover, this unit will remain in standby state.
ciscoasa/sec/stbyNoFailover# show interface GigabitEthernet0/7
Interface GigabitEthernet0/7 "FAILOVER", is up, line protocol is up
Hardware is i82574L rev00, BW 1000 Mbps, DLY 10 usec
Auto-Duplex(Full-duplex), Auto-Speed(1000 Mbps)
Input flow control is unsupported, output flow control is off
Description: LAN FAILOVER
MAC address f07f.0645.12bd, MTU 1500
IP address 172.30.5.2, subnet mask 255.255.255.252
1581714 packets input, 151114208 bytes, 0 no buffer
Received 11527 broadcasts, 0 runts, 0 giants
506 input errors, 506 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
0 pause input, 0 resume input
0 L2 decode drops
1591204 packets output, 144014372 bytes, 0 underruns
0 pause output, 0 resume output
0 output errors, 0 collisions, 481 interface resets
0 late collisions, 0 deferred
11 input reset drops, 0 output reset drops
input queue (blocks free curr/low): hardware (475/461)
output queue (blocks free curr/low): hardware (453/447)
Traffic Statistics for "FAILOVER":
23352 packets input, 1474664 bytes
15992 packets output, 1061134 bytes
0 packets dropped
1 minute input rate 0 pkts/sec, 35 bytes/sec
1 minute output rate 0 pkts/sec, 14 bytes/sec
1 minute drop rate, 0 pkts/sec
5 minute input rate 0 pkts/sec, 35 bytes/sec
5 minute output rate 0 pkts/sec, 14 bytes/sec
5 minute drop rate, 0 pkts/sec
I swapped the cable (straight cable) between the ASA G0/7 ports (failover link) and cleared the interface counters.
ciscoasa/pri/act# clear interface ?
Current available interface(s):
GigabitEthernet GigabitEthernet IEEE 802.3z
Management Management interface
Port-channel Ethernet Channel of interfaces
Redundant Redundant Interface
inside Name of interface GigabitEthernet0/1
management Name of interface Management0/0
outside Name of interface GigabitEthernet0/0
<cr>
ciscoasa/pri/act# clear interface GigabitEthernet0/7
ciscoasa/pri/act# show interface GigabitEthernet0/7
Interface GigabitEthernet0/7 "folan", is up, line protocol is up
Hardware is i82574L rev00, BW 1000 Mbps, DLY 10 usec
Auto-Duplex(Full-duplex), Auto-Speed(1000 Mbps)
Input flow control is unsupported, output flow control is off
Description: LAN FAILOVER
MAC address 4c4e.35ea.ddb7, MTU 1500
IP address 172.30.5.1, subnet mask 255.255.255.252
2014 packets input, 177860 bytes, 0 no buffer
Received 1 broadcasts, 0 runts, 0 giants
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
0 pause input, 0 resume input
0 L2 decode drops
2093 packets output, 214728 bytes, 0 underruns
0 pause output, 0 resume output
0 output errors, 0 collisions, 0 interface resets
0 late collisions, 0 deferred
0 input reset drops, 0 output reset drops
input queue (blocks free curr/low): hardware (495/459)
output queue (blocks free curr/low): hardware (454/442)
Traffic Statistics for "folan":
2014 packets input, 141204 bytes
2093 packets output, 172924 bytes
0 packets dropped
1 minute input rate 1 pkts/sec, 104 bytes/sec
1 minute output rate 1 pkts/sec, 112 bytes/sec
1 minute drop rate, 0 pkts/sec
5 minute input rate 1 pkts/sec, 105 bytes/sec
5 minute output rate 1 pkts/sec, 113 bytes/sec
5 minute drop rate, 0 pkts/sec
ciscoasa/sec/stbyNoFailover# clear interface GigabitEthernet0/7
ciscoasa/sec/stbyNoFailover# show interface GigabitEthernet0/7
Interface GigabitEthernet0/7 "FAILOVER", is up, line protocol is up
Hardware is i82574L rev00, BW 1000 Mbps, DLY 10 usec
Auto-Duplex(Full-duplex), Auto-Speed(1000 Mbps)
Input flow control is unsupported, output flow control is off
Description: LAN FAILOVER
MAC address f07f.0645.12bd, MTU 1500
IP address 172.30.5.2, subnet mask 255.255.255.252
255 packets input, 16320 bytes, 0 no buffer
Received 153 broadcasts, 0 runts, 0 giants
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
0 pause input, 0 resume input
0 L2 decode drops
153 packets output, 9792 bytes, 0 underruns
0 pause output, 0 resume output
0 output errors, 0 collisions, 0 interface resets
0 late collisions, 0 deferred
0 input reset drops, 0 output reset drops
input queue (blocks free curr/low): hardware (468/461)
output queue (blocks free curr/low): hardware (495/447)
Traffic Statistics for "folan":
255 packets input, 10914 bytes
153 packets output, 4284 bytes
0 packets dropped
1 minute input rate 0 pkts/sec, 35 bytes/sec
1 minute output rate 0 pkts/sec, 14 bytes/sec
1 minute drop rate, 0 pkts/sec
5 minute input rate 0 pkts/sec, 28 bytes/sec
5 minute output rate 0 pkts/sec, 11 bytes/sec
5 minute drop rate, 0 pkts/sec
I didn't observe incrementing input and CRC errors after the cable swap, so I've enabled failover back on the Standbyfirewall.
ciscoasa/sec/stbyNoFailover# configure terminal
ciscoasa/sec/stbyNoFailover(config)# failover // SSH SESSION HANGED; NEED RE-LOGIN
Here's a nice PDF troubleshooting guide from Cisco. Although it was written for the PIX firewall, some troubleshooting scenarios and steps are still applicable with the ASA firewall.
ciscoasa/pri/act#
Failover LAN Failed
Failover LAN became OK
Switchover enabled
Failover LAN Failed
Failover LAN became OK
Switchover enabled
Failover LAN Failed
Failover LAN became OK
Switchover enabled
ciscoasa/pri/act# show interface GigabitEthernet0/7
Interface GigabitEthernet0/7 "FAILOVER", is up, line protocol is up
Hardware is i82574L rev00, BW 1000 Mbps, DLY 10 usec
Auto-Duplex(Full-duplex), Auto-Speed(1000 Mbps)
Input flow control is unsupported, output flow control is off
Description: LAN FAILOVER
MAC address 4c4e.35ea.ddb7, MTU 1500
IP address 172.30.5.1, subnet mask 255.255.255.252
1563433 packets input, 141470716 bytes, 0 no buffer
Received 2141 broadcasts, 0 runts, 0 giants
5238 input errors, 5238 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
0 pause input, 0 resume input
0 L2 decode drops
1583757 packets output, 151416596 bytes, 0 underruns
0 pause output, 0 resume output
0 output errors, 0 collisions, 477 interface resets
0 late collisions, 0 deferred
0 input reset drops, 0 output reset drops
input queue (blocks free curr/low): hardware (487/461)
output queue (blocks free curr/low): hardware (508/446)
Traffic Statistics for "folan":
1563151 packets input, 112914310 bytes
1584569 packets output, 122502340 bytes
0 packets dropped
1 minute input rate 0 pkts/sec, 23 bytes/sec
1 minute output rate 0 pkts/sec, 26 bytes/sec
1 minute drop rate, 0 pkts/sec
5 minute input rate 0 pkts/sec, 23 bytes/sec
5 minute output rate 0 pkts/sec, 26 bytes/sec
5 minute drop rate, 0 pkts/sec
Notice the show failover on the Standby firewall displays a short Active time (56 sec).
ciscoasa/sec/stby# show failover
Failover On
Failover unit Secondary
Failover LAN Interface: FAILOVER GigabitEthernet0/7 (up)
Unit Poll frequency 1 seconds, holdtime 15 seconds
Interface Poll frequency 5 seconds, holdtime 25 seconds
Interface Policy 1
Monitored Interfaces 3 of 216 maximum
Version: Ours 9.1(7)13, Mate 9.1(7)13
Last Failover at: 10:37:46 SA Oct 17 2017
This host: Secondary - Standby Ready
Active time: 56 (sec)
slot 0: ASA5525 hw/sw rev (1.0/9.1(7)13) status (Up Sys)
Interface outside (62.2.4.7): Normal (Monitored)
Interface inside (172.30.0.5): Normal (Monitored)
Interface management (172.30.0.8): Normal (Monitored)
Other host: Primary - Active
Active time: 1063719 (sec)
slot 0: ASA5525 hw/sw rev (1.0/9.1(7)13) status (Up Sys)
Interface outside (62.2.4.8): Normal (Monitored)
Interface inside (172.30.0.3): Normal (Monitored)
Interface management (172.30.0.7): Normal (Monitored)
Stateful Failover Logical Update Statistics
Link : folink GigabitEthernet0/6 (up)
Stateful Obj xmit xerr rcv rerr
General 137696 0 2372002 57
sys cmd 137696 0 137654 57
up time 0 0 0 0
RPC services 0 0 0 0
TCP conn 0 0 215467 0
UDP conn 0 0 1829472 0
ARP tbl 0 0 186949 0
Xlate_Timeout 0 0 0 0
IPv6 ND tbl 0 0 0 0
VPN IKEv1 SA 0 0 474 0
VPN IKEv1 P2 0 0 1978 0
VPN IKEv2 SA 0 0 0 0
VPN IKEv2 P2 0 0 0 0
VPN CTCP upd 0 0 0 0
VPN SDI upd 0 0 0 0
VPN DHCP upd 0 0 0 0
SIP Session 0 0 0 0
Route Session 0 0 0 0
User-Identity 0 0 8 0
CTS SGTNAME 0 0 0 0
CTS PAC 0 0 0 0
TrustSec-SXP 0 0 0 0
IPv6 Route 0 0 0 0
Logical Update Queue Information
Cur Max Total
Recv Q: 0 25 6737580
Xmit Q: 0 1 137695
The syslogs also showed flapping Failover interface.
ciscoasa/sec/stby# show log
<SNIP>
Oct 17 2017 10:57:27 ciscoasa : %ASA-1-105043: (Secondary) Failover interface failed
Oct 17 2017 10:57:27 ciscoasa : %ASA-6-720032: (VPN-Secondary) HA status callback: id=3,seq=200,grp=0,event=401,op=0,my=Standby Ready,peer=Active.
Oct 17 2017 10:57:27 ciscoasa : %ASA-6-720024: (VPN-Secondary) HA status callback: Control channel is down.
Oct 17 2017 10:57:27 ciscoasa : %ASA-6-721002: (WebVPN-Secondary) HA status change: event HA_STATUS_PEER_CTL_COMM, my state Standby Ready, peer state Active.
Oct 17 2017 10:57:33 ciscoasa : %ASA-6-720032: (VPN-Secondary) HA status callback: id=3,seq=200,grp=0,event=401,op=1,my=Standby Ready,peer=Active.
Oct 17 2017 10:57:33 ciscoasa : %ASA-6-720024: (VPN-Secondary) HA status callback: Control channel is up.
Oct 17 2017 10:57:33 ciscoasa : %ASA-6-721002: (WebVPN-Secondary) HA status change: event HA_STATUS_PEER_CTL_COMM, my state Standby Ready, peer state Active.
Oct 17 2017 10:57:33 ciscoasa : %ASA-6-720032: (VPN-Secondary) HA status callback: id=3,seq=200,grp=0,event=411,op=61,my=Standby Ready,peer=Active.
Oct 17 2017 10:57:33 ciscoasa : %ASA-6-721002: (WebVPN-Secondary) HA status change: event HA_STATUS_CLIENT_NEGOTIATED_VERSION, my state Standby Ready, peer state Active.
Oct 17 2017 10:57:34 ciscoasa : %ASA-1-105042: (Secondary) Failover interface OK
Oct 17 2017 10:57:42 ciscoasa : %ASA-1-105003: (Secondary) Monitoring on interface outside waiting
Oct 17 2017 10:57:42 ciscoasa : %ASA-1-105003: (Secondary) Monitoring on interface inside waiting
Oct 17 2017 10:57:42ciscoasa : %ASA-1-105003: (Secondary) Monitoring on interface management waiting
Oct 17 2017 10:57:57 ciscoasa : %ASA-1-105004: (Secondary) Monitoring on interface outside normal
Oct 17 2017 10:57:57 ciscoasa : %ASA-1-105004: (Secondary) Monitoring on interface inside normal
Oct 17 2017 10:58:07 ciscoasa : %ASA-1-105004: (Secondary) Monitoring on interface management normal
Oct 17 2017 11:03:16 ciscoasa: %ASA-1-105043: (Secondary) Failover interface failed
ciscoasa/sec/stby# Failover LAN Failed
Failover LAN became OK
Switchover enabled
Failover LAN Failed
Failover LAN became OK
Switchover enabled
Failover LAN Failed
Failover LAN became OK
Switchover enabled
You can use the show failover command to verify ASA failover pair operation and activity/history. The show failover state indicates an Interface failure.
ciscoasa/sec/stby# show failover ?
descriptor Show failover interface descriptors. Two numbers are shown for
each interface. When exchanging information regarding a
particular interface, this unit uses the first number in messages
it sends to its peer. And it expects the second number in
messages it receives from its peer. For trouble shooting, collect
the show output from both units and verify that the numbers
match.
exec Show failover command execution information
history Show failover switching history
interface Show failover command interface information
state Show failover internal state information
statistics Show failover command interface statistics information
| Output modifiers
<cr>
ciscoasa/sec/stby# show failover state
State Last Failure Reason Date/Time
This host - Secondary
Standby Ready Ifc Failure 04:42:50 SA Oct 17 2017
Other host - Primary
Active Comm Failure 10:37:48 SA Oct 17 2017
====Configuration State===
Sync Done - STANDBY
====Communication State===
Mac set
ciscoasa/sec/stby# show failover history
==========================================================================
From State To State Reason
==========================================================================
00:41:34 SA Oct 17 2017
Bulk Sync Standby Ready No Error
04:42:50 SA Oct 17 2017
Standby Ready Failed Interface check
04:42:52 SA Oct 17 2017
Failed Standby Ready Interface check
10:37:46 SA Oct 17 2017
Standby Ready Just Active HELLO not heard from mate
10:37:46 SA Oct 17 2017
Just Active Active Drain HELLO not heard from mate
10:37:46 SA Oct 17 2017
Active Drain Active Applying Config HELLO not heard from mate
10:37:46 SA Oct 17 2017
Active Applying Config Active Config Applied HELLO not heard from mate
10:37:46 SA Oct 17 2017
Active Config Applied Active HELLO not heard from mate
10:38:42 SA Oct 17 2017
Active Cold Standby Failover state check
10:38:44 SA Oct 17 2017
Cold Standby Sync Config Failover state check
10:38:51 SA Oct 17 2017
Sync Config Sync File System Failover state check
10:38:51 SA Oct 17 2017
Sync File System Bulk Sync Failover state check
10:39:10 SA Oct 17 2017
Bulk Sync Standby Ready Failover state check
12:14:06 SA Oct 17 2017
Standby Ready Disabled Set by the config command // DISABLED FAILOVER
16:09:19 SA Oct 17 2017
Disabled Negotiation Set by the config command
16:09:20 SA Oct 17 2017
Negotiation Cold Standby Detected an Active mate
16:09:22 SA Oct 17 2017
Cold Standby Sync Config Detected an Active mate
16:09:29 SA Oct 17 2017
Sync Config Sync File System Detected an Active mate
16:09:29 SA Oct 17 2017
Sync File System Bulk Sync Detected an Active mate
16:09:44 SA Oct 17 2017
Bulk Sync Standby Ready Detected an Active mate
==========================================================================
I've temporarily disabled failover on the Standby firewall using the no failover command. The flapping had stopped and the hostname displayed NoFailover.
ciscoasa/sec/stby# show run failover
failover
failover lan unit secondary
failover lan interface FAILOVER GigabitEthernet0/7
failover interface ip FAILOVER 172.30.5.1 255.255.255.252 standby 172.30.5.2
ciscoasa/sec/stby# configure terminal
**** WARNING ****
Configuration Replication is NOT performed from Standby unit to Active unit.
Configurations are no longer synchronized.
ciscoasa/sec/stby(config)# no failover
INFO: This unit is currently in standby state. By disabling failover, this unit will remain in standby state.
ciscoasa/sec/stbyNoFailover# show interface GigabitEthernet0/7
Interface GigabitEthernet0/7 "FAILOVER", is up, line protocol is up
Hardware is i82574L rev00, BW 1000 Mbps, DLY 10 usec
Auto-Duplex(Full-duplex), Auto-Speed(1000 Mbps)
Input flow control is unsupported, output flow control is off
Description: LAN FAILOVER
MAC address f07f.0645.12bd, MTU 1500
IP address 172.30.5.2, subnet mask 255.255.255.252
1581714 packets input, 151114208 bytes, 0 no buffer
Received 11527 broadcasts, 0 runts, 0 giants
506 input errors, 506 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
0 pause input, 0 resume input
0 L2 decode drops
1591204 packets output, 144014372 bytes, 0 underruns
0 pause output, 0 resume output
0 output errors, 0 collisions, 481 interface resets
0 late collisions, 0 deferred
11 input reset drops, 0 output reset drops
input queue (blocks free curr/low): hardware (475/461)
output queue (blocks free curr/low): hardware (453/447)
Traffic Statistics for "FAILOVER":
23352 packets input, 1474664 bytes
15992 packets output, 1061134 bytes
0 packets dropped
1 minute input rate 0 pkts/sec, 35 bytes/sec
1 minute output rate 0 pkts/sec, 14 bytes/sec
1 minute drop rate, 0 pkts/sec
5 minute input rate 0 pkts/sec, 35 bytes/sec
5 minute output rate 0 pkts/sec, 14 bytes/sec
5 minute drop rate, 0 pkts/sec
I swapped the cable (straight cable) between the ASA G0/7 ports (failover link) and cleared the interface counters.
ciscoasa/pri/act# clear interface ?
Current available interface(s):
GigabitEthernet GigabitEthernet IEEE 802.3z
Management Management interface
Port-channel Ethernet Channel of interfaces
Redundant Redundant Interface
inside Name of interface GigabitEthernet0/1
management Name of interface Management0/0
outside Name of interface GigabitEthernet0/0
<cr>
ciscoasa/pri/act# clear interface GigabitEthernet0/7
ciscoasa/pri/act# show interface GigabitEthernet0/7
Interface GigabitEthernet0/7 "folan", is up, line protocol is up
Hardware is i82574L rev00, BW 1000 Mbps, DLY 10 usec
Auto-Duplex(Full-duplex), Auto-Speed(1000 Mbps)
Input flow control is unsupported, output flow control is off
Description: LAN FAILOVER
MAC address 4c4e.35ea.ddb7, MTU 1500
IP address 172.30.5.1, subnet mask 255.255.255.252
2014 packets input, 177860 bytes, 0 no buffer
Received 1 broadcasts, 0 runts, 0 giants
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
0 pause input, 0 resume input
0 L2 decode drops
2093 packets output, 214728 bytes, 0 underruns
0 pause output, 0 resume output
0 output errors, 0 collisions, 0 interface resets
0 late collisions, 0 deferred
0 input reset drops, 0 output reset drops
input queue (blocks free curr/low): hardware (495/459)
output queue (blocks free curr/low): hardware (454/442)
Traffic Statistics for "folan":
2014 packets input, 141204 bytes
2093 packets output, 172924 bytes
0 packets dropped
1 minute input rate 1 pkts/sec, 104 bytes/sec
1 minute output rate 1 pkts/sec, 112 bytes/sec
1 minute drop rate, 0 pkts/sec
5 minute input rate 1 pkts/sec, 105 bytes/sec
5 minute output rate 1 pkts/sec, 113 bytes/sec
5 minute drop rate, 0 pkts/sec
ciscoasa/sec/stbyNoFailover# clear interface GigabitEthernet0/7
ciscoasa/sec/stbyNoFailover# show interface GigabitEthernet0/7
Interface GigabitEthernet0/7 "FAILOVER", is up, line protocol is up
Hardware is i82574L rev00, BW 1000 Mbps, DLY 10 usec
Auto-Duplex(Full-duplex), Auto-Speed(1000 Mbps)
Input flow control is unsupported, output flow control is off
Description: LAN FAILOVER
MAC address f07f.0645.12bd, MTU 1500
IP address 172.30.5.2, subnet mask 255.255.255.252
255 packets input, 16320 bytes, 0 no buffer
Received 153 broadcasts, 0 runts, 0 giants
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
0 pause input, 0 resume input
0 L2 decode drops
153 packets output, 9792 bytes, 0 underruns
0 pause output, 0 resume output
0 output errors, 0 collisions, 0 interface resets
0 late collisions, 0 deferred
0 input reset drops, 0 output reset drops
input queue (blocks free curr/low): hardware (468/461)
output queue (blocks free curr/low): hardware (495/447)
Traffic Statistics for "folan":
255 packets input, 10914 bytes
153 packets output, 4284 bytes
0 packets dropped
1 minute input rate 0 pkts/sec, 35 bytes/sec
1 minute output rate 0 pkts/sec, 14 bytes/sec
1 minute drop rate, 0 pkts/sec
5 minute input rate 0 pkts/sec, 28 bytes/sec
5 minute output rate 0 pkts/sec, 11 bytes/sec
5 minute drop rate, 0 pkts/sec
I didn't observe incrementing input and CRC errors after the cable swap, so I've enabled failover back on the Standbyfirewall.
ciscoasa/sec/stbyNoFailover# configure terminal
ciscoasa/sec/stbyNoFailover(config)# failover // SSH SESSION HANGED; NEED RE-LOGIN
No comments:
Post a Comment