Saturday, May 18, 2019

Troubleshooting Cisco ASA Firewall Active-Standby Failover

I was configuring a new pair of Cisco ASA 5555-X and tried to make failover work. I tried removing the failover pre-shared key, used the failover ipsec pre-shared key <KEY> command, re-configured failover on both the Primary and Secondary firewalls and re-configured the Secondary firewall from scratch but no luck.

It kept looping with the error: "REPLICATION OF CONFIGURATION FROM ACTIVE TO STANDBY UNIT IS INCOMPLETE, TO PREVENT THE STANDBY UNIT TAKING OVER AS ACTIVE WITH A PARTIAL CONFIGURATION,THE STANDBY UNIT WILL NOW REBOOT"


ciscoasa#    // SECONDARY FW
Beginning configuration replication from mate.   

ciscoasa# show interface ip brief
Interface                  IP-Address      OK? Method Status                Protocol
GigabitEthernet0/0         unassigned      YES unset  up                    up 
GigabitEthernet0/1         unassigned      YES unset  up                    up 
GigabitEthernet0/2         unassigned      YES unset  administratively down down
GigabitEthernet0/3         unassigned      YES unset  administratively down down
GigabitEthernet0/4         unassigned      YES unset  administratively down down
GigabitEthernet0/5         unassigned      YES unset  administratively down down
GigabitEthernet0/6         unassigned      YES unset  administratively down down
GigabitEthernet0/7         192.168.1.1     YES unset  up                    up 
Internal-Control0/0        127.0.1.1       YES unset  up                    up 
Internal-Data0/0           unassigned      YES unset  up                    up 
Internal-Data0/1           unassigned      YES unset  down                  down
Internal-Data0/2           unassigned      YES unset  up                    up 
Internal-Data0/3           169.254.1.1     YES unset  up                    up 
Management0/0              unassigned      YES unset  up                    up 
ciscoasa#
******REPLICATION OF CONFIGURATION FROM ACTIVE TO STANDBY UNIT IS INCOMPLETE,
 TO PREVENT THE STANDBY UNIT TAKING OVER AS ACTIVE WITH A PARTIAL CONFIGURATION,
 THE STANDBY UNIT WILL NOW REBOOT*******
             
Link : Unconfigured.

INFO: FirePower module is detected running.  ASA will be reloaded gracefully.


***
*** --- SHUTDOWN NOW ---
***
*** Message to all terminals:
***
***   failover reset
Process shutdown finished
Rebooting... (status 0x9)
..
INIT: Sending processes the TERM signal
Deconfiguring network interfaces... done.
Sending all processes the TERM signal...


ciscoasa/pri/act# show run failover
failover
failover lan unit primary
failover lan interface FAILOVER GigabitEthernet0/7
failover key *****
failover replication http
failover interface ip FAILOVER 192.168.1.1 255.255.255.252 standby 192.168.1.2


I noticed a high rate of ping drops towards the Secondary failover IP and also a high input and CRC errors, which indicates a Layer 1 (cable) issue. So I swapped the failover (straight) cable between the firewall pair and the Secondary firewall started to sync its config with the Primary firewall.

ciscoasa/pri/act# ping 192.168.1.2 rep 100
Type escape sequence to abort.
Sending 100, 100-byte ICMP Echos to 192.168.1.2, timeout is 2 seconds:
!?!????!?!!??!???!!??!!????!?!!?!!????!????!???!???!????!?!!????!?!???
!!!?!?
Success rate is 36 percent (28/76), round-trip min/avg/max = 1/1/1 ms

ciscoasa/pri/act# show interface g0/7   // HIGH INPUT AND CRC ERRORS DETECTED                           
Interface GigabitEthernet0/7 "FAILOVER", is up, line protocol is up
  Hardware is i82574L rev00, BW 1000 Mbps, DLY 10 usec
        Auto-Duplex(Full-duplex), Auto-Speed(1000 Mbps)
        Input flow control is unsupported, output flow control is off
        Description: LAN Failover Interface
        MAC address 5087.89b7.5593, MTU 1500
        IP address 192.168.1.1, subnet mask 255.255.255.252
        1058 packets input, 119647 bytes, 0 no buffer
        Received 114 broadcasts, 0 runts, 0 giants
        830 input errors, 830 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
        0 pause input, 0 resume input
        0 L2 decode drops
        1373 packets output, 304203 bytes, 0 underruns
        0 pause output, 0 resume output
        0 output errors, 0 collisions, 9 interface resets
        0 late collisions, 0 deferred
        0 input reset drops, 0 output reset drops
        input queue (blocks free curr/low): hardware (487/461)
        output queue (blocks free curr/low): hardware (453/446)
  Traffic Statistics for "FAILOVER":
        566 packets input, 67308 bytes
        695 packets output, 204164 bytes
        0 packets dropped
      1 minute input rate 2 pkts/sec,  279 bytes/sec
      1 minute output rate 1 pkts/sec,  640 bytes/sec
      1 minute drop rate, 0 pkts/sec
      5 minute input rate 0 pkts/sec,  64 bytes/sec
      5 minute output rate 1 pkts/sec,  304 bytes/sec
      5 minute drop rate, 0 pkts/sec


Troubleshooting the Secondary (Standby) firewall:

ciscoasa# show run failover
failover
failover lan unit secondary
failover lan interface FAILOVER GigabitEthernet0/7
failover key *****
failover replication http
failover interface ip FAILOVER 192.168.1.1 255.255.255.252 standby 192.168.1.2

ciscoasa# show interface g0/7 
Interface GigabitEthernet0/7 "FAILOVER", is up, line protocol is up
  Hardware is i82574L rev00, BW 1000 Mbps, DLY 10 usec
        Auto-Duplex(Full-duplex), Auto-Speed(1000 Mbps)
        Input flow control is unsupported, output flow control is off
        Description: LAN Failover Interface
        MAC address f40f.1b1e.1405, MTU 1500
        IP address 192.168.1.2, subnet mask 255.255.255.252
        277 packets input, 45654 bytes, 0 no buffer
        Received 26 broadcasts, 0 runts, 0 giants
        193 input errors, 193 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
        0 pause input, 0 resume input
        0 L2 decode drops
        992 packets output, 138768 bytes, 0 underruns
        0 pause output, 0 resume output
        0 output errors, 0 collisions, 4 interface resets
        0 late collisions, 0 deferred
        3 input reset drops, 0 output reset drops
        input queue (blocks free curr/low): hardware (503/461)
        output queue (blocks free curr/low): hardware (492/447)
  Traffic Statistics for "FAILOVER":
        274 packets input, 40530 bytes
        992 packets output, 120192 bytes
        0 packets dropped
      1 minute input rate 1 pkts/sec,  255 bytes/sec
      1 minute output rate 4 pkts/sec,  519 bytes/sec
      1 minute drop rate, 0 pkts/sec
      5 minute input rate 0 pkts/sec,  0 bytes/sec
      5 minute output rate 0 pkts/sec,  0 bytes/sec
      5 minute drop rate, 0 pkts/sec

ciscoasa# Failover LAN became OK
Switchover enabled
Configuration has changed, replicate from mate.
..

        Detected an Active mate
Beginning configuration replication from mate.
WARNING: Local user database is empty and there are still 'aaa' commands for 'LOCAL'.
ciscoasa/sec/stby# End configuration replication from mate.

No comments:

Post a Comment