I needed to add the failover key <KEY> command in a Cisco ASA firewall pair. The failover pair is working but the previous admin forgot to add this command. I used two Cisco ASAv firewalls in my GNS3 lab to simulate if it's relatively safe to add the said command and wouldn't break the failover pair. It's still advisable to do this in an approved maintenance window.
ASAv-1# show version
Cisco Adaptive Security Appliance Software Version 9.8(1)
Firepower Extensible Operating System Version 2.2(1.47)
Device Manager Version 7.8(1)
Compiled on Wed 10-May-17 15:38 PDT by builders
System image file is "boot:/asa981-smp-k8.bin"
Config file at boot was "startup-config"
ASAv-1 up 16 mins 35 secs
Hardware: ASAv, 2048 MB RAM, CPU Pentium II 1900 MHz,
Model Id: ASAv10
Internal ATA Compact Flash, 8192MB
Slot 1: ATA Compact Flash, 8192MB
BIOS Flash Firmware Hub @ 0x0, 0KB
0: Ext: Management0/0 : address is 0ce5.a655.f500, irq 11
1: Ext: GigabitEthernet0/0 : address is 0ce5.a655.f501, irq 11
2: Ext: GigabitEthernet0/1 : address is 0ce5.a655.f502, irq 10
3: Ext: GigabitEthernet0/2 : address is 0ce5.a655.f503, irq 10
4: Ext: GigabitEthernet0/3 : address is 0ce5.a655.f504, irq 11
5: Ext: GigabitEthernet0/4 : address is 0ce5.a655.f505, irq 11
6: Ext: GigabitEthernet0/5 : address is 0ce5.a655.f506, irq 10
7: Ext: GigabitEthernet0/6 : address is 0ce5.a655.f507, irq 10
License mode: Smart Licensing
ASAv Platform License State: Unlicensed
No active entitlement: no feature tier and no throughput level configured
*Memory resource allocation is more than the permitted limit.
Licensed features for this platform:
Maximum VLANs : 50
Inside Hosts : Unlimited
Failover : Active/Standby // ASAv SUPPORTS ACTIVE/STANDBY BY DEFAULT
Encryption-DES : Enabled
Encryption-3DES-AES : Enabled
Security Contexts : 0
Carrier : Disabled
AnyConnect Premium Peers : 2
AnyConnect Essentials : Disabled
Other VPN Peers : 250
Total VPN Peers : 250
AnyConnect for Mobile : Disabled
AnyConnect for Cisco VPN Phone : Disabled
Advanced Endpoint Assessment : Disabled
Shared License : Disabled
Total TLS Proxy Sessions : 2
Botnet Traffic Filter : Enabled
Cluster : Disabled
Serial Number: 9A81V5LKN5F
Image type : Release
Key version : A
Configuration last modified by enable_15 at 03:03:53.969 UTC Tue Jan 25 2022
-----
ASAv-2# show version
Cisco Adaptive Security Appliance Software Version 9.8(1)
Firepower Extensible Operating System Version 2.2(1.47)
Device Manager Version 7.8(1)
Compiled on Wed 10-May-17 15:38 PDT by builders
System image file is "boot:/asa981-smp-k8.bin"
Config file at boot was "startup-config"
ASAv-2 up 1 min 53 secs
Hardware: ASAv, 2048 MB RAM, CPU Pentium II 1900 MHz,
Model Id: ASAv10
Internal ATA Compact Flash, 8192MB
Slot 1: ATA Compact Flash, 8192MB
BIOS Flash Firmware Hub @ 0x0, 0KB
0: Ext: Management0/0 : address is 0ce5.a6d4.2800, irq 11
1: Ext: GigabitEthernet0/0 : address is 0ce5.a6d4.2801, irq 11
2: Ext: GigabitEthernet0/1 : address is 0ce5.a6d4.2802, irq 10
3: Ext: GigabitEthernet0/2 : address is 0ce5.a6d4.2803, irq 10
4: Ext: GigabitEthernet0/3 : address is 0ce5.a6d4.2804, irq 11
5: Ext: GigabitEthernet0/4 : address is 0ce5.a6d4.2805, irq 11
6: Ext: GigabitEthernet0/5 : address is 0ce5.a6d4.2806, irq 10
7: Ext: GigabitEthernet0/6 : address is 0ce5.a6d4.2807, irq 10
License mode: Smart Licensing
ASAv Platform License State: Unlicensed
No active entitlement: no feature tier and no throughput level configured
*Memory resource allocation is more than the permitted limit.
Licensed features for this platform:
Maximum VLANs : 50
Inside Hosts : Unlimited
Failover : Active/Standby
Encryption-DES : Enabled
Encryption-3DES-AES : Enabled
Security Contexts : 0
Carrier : Disabled
AnyConnect Premium Peers : 2
AnyConnect Essentials : Disabled
Other VPN Peers : 250
Total VPN Peers : 250
AnyConnect for Mobile : Disabled
AnyConnect for Cisco VPN Phone : Disabled
Advanced Endpoint Assessment : Disabled
Shared License : Disabled
Total TLS Proxy Sessions : 2
Botnet Traffic Filter : Enabled
Cluster : Disabled
Serial Number: 9AK137KWDWB
Image type : Release
Key version : A
Configuration last modified by enable_15 at 03:17:50.569 UTC Tue Jan 25 2022
Configure failover on ASAv-1 and ASAv-2 without the failover key command.
ASAv-1# configure terminal
ASAv-1(config)#
interface GigabitEthernet0/6 // DIRECT CABLE TO ASAv-2 G0/6
ASAv-1(config-if)# no shutdown
ASAv-1(config-if)# failover lan unit primary
ASAv-1(config)# failover lan interface FAILOVER GigabitEthernet0/6
INFO: Non-failover interface config is cleared on GigabitEthernet0/6 and its sub-interfaces
ASAv-1(config)# failover link FAILOVER GigabitEthernet0/6
ASAv-1(config)#
failover interface ip failover 10.0.0.1 255.255.255.252 standby 10.0.0.2
ASAv-1(config)# failover
ASAv-1(config)# .
No Active mate detected
Beginning configuration replication: Sending to mate.
End Configuration Replication to mate
The ASAv pair started to sync and form the failover pair (or cluster) even without the failover key command.
ASAv-2# configure terminal
ASAv-2(config)# interface GigabitEthernet0/6
ASAv-2(config-if)# no shutdown
ASAv-2(config-if)# failover lan unit secondary
ASAv-2(config)# failover lan interface FAILOVER GigabitEthernet0/6
INFO: Non-failover interface config is cleared on GigabitEthernet0/6 and its sub-interfaces
ASAv-2(config)#
failover interface ip failover 10.0.0.1 255.255.255.252 standby 10.0.0.2
ASAv-2(config)# failover
ASAv-2(config)# ..
Detected an Active mate
Beginning configuration replication from mate.
WARNING: Disabling auto import may affect Smart Licensing
WARNING: Local user database is empty and there are still 'aaa' commands for 'LOCAL'.
Creating trustpoint "_SmartCallHome_ServerCA" and installing certificate...
Trustpoint CA certificate accepted.
WARNING: Failover is enabled but standby IP address is not configured for this interface.
WARNING: Failover is enabled but standby IP address is not configured for this interface.
WARNING: Failover is enabled but standby IP address is not configured for this interface.
WARNING: Trustpoint _SmartCallHome_ServerCA is already authenticated.
WARNING: This command will not take effect until interface 'inside' has been assigned an IPv4 address
End configuration replication from mate.
The failover failed due to the monitored interfaces on ASAv-2. The only interface connected in ASAv-2 is a direct cable to ASAv-1 used for the failover interface (G0/6).
ASAv-1# ping 10.0.0.2
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.0.0.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/8/20 ms
ASAv-1# show failover
Failover On
Failover unit Primary
Failover LAN Interface: FAILOVER GigabitEthernet0/6 (up)
Reconnect timeout 0:00:00
Unit Poll frequency 1 seconds, holdtime 15 seconds
Interface Poll frequency 5 seconds, holdtime 25 seconds
Interface Policy 1
Monitored Interfaces 3 of 61 maximum
MAC Address Move Notification Interval not set
Version: Ours 9.8(1), Mate 9.8(1)
Serial Number: Ours 9A81V5LKN5F, Mate 9AK137KWDWB
Last Failover at: 03:20:54 UTC Jan 25 2022
This host: Primary - Active
Active time: 52 (sec)
slot 0: empty
Interface outside (200.1.1.1): Normal (Waiting)
Interface inside (172.16.1.1): Normal (Waiting)
Interface dmz (10.1.1.1): Normal (Waiting)
Other host: Secondary - Failed
Active time: 0 (sec)
Interface outside (0.0.0.0): No Link (Waiting)
Interface inside (0.0.0.0): No Link (Waiting)
Interface dmz (0.0.0.0): No Link (Waiting)
<OUTPUT TRUNCATED>
ASAv-1# show failover state
State Last Failure Reason Date/Time
This host - Primary
Active None
Other host - Secondary
Failed Ifc Failure 03:21:27 UTC Jan 25 2022
outside: No Link
inside: No Link
dmz: No Link
====Configuration State===
Sync Done
====Communication State===
Mac set
ASAv-1# show failover history
==========================================================================
From State To State Reason
==========================================================================
03:02:35 UTC Jan 25 2022
Not Detected Disabled No Error
03:20:19 UTC Jan 25 2022
Disabled Negotiation Set by the config command
03:20:54 UTC Jan 25 2022
Negotiation Just Active No Active unit found
03:20:54 UTC Jan 25 2022
Just Active Active Drain No Active unit found
03:20:54 UTC Jan 25 2022
Active Drain Active Applying Config No Active unit found
03:20:54 UTC Jan 25 2022
Active Applying Config Active Config Applied No Active unit found
03:20:54 UTC Jan 25 2022
Active Config Applied Active No Active unit found
==========================================================================
I've temporarily removed monitoring on all interfaces using the no monitor-interface <INTERFACE> and added the prompt hostname priority command to distinguish between Primary and Secondary ASAv. The failover worked afterwards and Secondary ASAv status changed to Standby Ready.
ASAv-1(config)# no monitor-interface outside
ASAv-1(config)# no monitor-interface inside
ASAv-1(config)# no monitor-interface dmz
ASAv-1(config)# prompt hostname priority
ASAv-1/pri(config)#
ASAv-1/sec#
Switching to Ok for reason Interface check.
ASAv-1/pri# show failover
Failover On
Failover unit Primary
Failover LAN Interface: FAILOVER GigabitEthernet0/6 (up)
Reconnect timeout 0:00:00
Unit Poll frequency 1 seconds, holdtime 15 seconds
Interface Poll frequency 5 seconds, holdtime 25 seconds
Interface Policy 1
Monitored Interfaces 0 of 61 maximum
MAC Address Move Notification Interval not set
Version: Ours 9.8(1), Mate 9.8(1)
Serial Number: Ours 9A81V5LKN5F, Mate 9AK137KWDWB
Last Failover at: 03:20:54 UTC Jan 25 2022
This host: Primary - Active
Active time: 277 (sec)
slot 0: empty
Interface outside (200.1.1.1): Normal (Not-Monitored)
Interface inside (172.16.1.1): Normal (Not-Monitored)
Interface dmz (10.1.1.1): Normal (Not-Monitored)
Other host: Secondary - Standby Ready
Active time: 0 (sec)
Interface outside (0.0.0.0): No Link (Not-Monitored)
Interface inside (0.0.0.0): No Link (Not-Monitored)
Interface dmz (0.0.0.0): No Link (Not-Monitored)
<OUTPUT TRUNCATED>
ASAv-1/pri# show failover state
State Last Failure Reason Date/Time
This host - Primary
Active None
Other host - Secondary
Standby Ready Ifc Failure 03:21:27 UTC Jan 25 2022
outside: No Link
inside: No Link
dmz: No Link
====Configuration State===
Sync Done
====Communication State===
Mac set
ASAv-1/pri# failover exec mate show run // VERIFY CONFIG IN ASAv-2
: Saved
:
: Serial Number: 9AK137KWDWB
: Hardware: ASAv, 2048 MB RAM, CPU Pentium II 1900 MHz
:
ASA Version 9.8(1)
!
hostname ASAv-1
enable password $sha512$5000$5bps8k/6inHnfDTz/HO44A==$PjXAl3nW8pK5BfcT6tJYwA== pbkdf2
xlate per-session deny tcp any4 any4
xlate per-session deny tcp any4 any6
xlate per-session deny tcp any6 any4
xlate per-session deny tcp any6 any6
xlate per-session deny udp any4 any4 eq domain
xlate per-session deny udp any4 any6 eq domain
xlate per-session deny udp any6 any4 eq domain
xlate per-session deny udp any6 any6 eq domain
passwd 2KFQnbNIdI.2KYOU encrypted
names
!
interface GigabitEthernet0/0
description ### TO R1 F1/0: INTERNET ###
nameif outside
security-level 0
ip address 200.1.1.1 255.255.255.0
!
interface GigabitEthernet0/1
description ### TO IOU_SW01 E0/0: INSIDE ###
nameif inside
security-level 100
ip address 172.16.1.1 255.255.255.0
!
interface GigabitEthernet0/2
description ### TO R2 DMZ SERVER ###
nameif dmz
security-level 50
<OUTPUT TRUNCATED>
ASAv-1/pri# failover exec mate show run failover
failover
failover lan unit secondary
failover lan interface FAILOVER GigabitEthernet0/6
failover link FAILOVER GigabitEthernet0/6
failover interface ip FAILOVER 10.0.0.1 255.255.255.252 standby 10.0.0.2
So I added the failover key command and confirmed it didn't "break" the ASA failover pair. The command was also immediately sync'd to ASAv-2.
ASAv-1/pri# configure terminal
ASAv-1/pri(config)# failover key cisco
ASAv-1/pri(config)# end
ASAv-1/pri# show run failover
failover
failover lan unit primary
failover lan interface FAILOVER GigabitEthernet0/6
failover key *****
failover link FAILOVER GigabitEthernet0/6
failover interface ip FAILOVER 10.0.0.1 255.255.255.252 standby 10.0.0.2
ASAv-1/pri# failover exec mate show run failover
failover
failover lan unit secondary
failover lan interface FAILOVER GigabitEthernet0/6
failover key *****
failover link FAILOVER GigabitEthernet0/6
failover interface ip FAILOVER 10.0.0.1 255.255.255.252 standby 10.0.0.2
ASAv-1/pri# sh failover
Failover On
Failover unit Primary
Failover LAN Interface: FAILOVER GigabitEthernet0/6 (up)
Reconnect timeout 0:00:00
Unit Poll frequency 1 seconds, holdtime 15 seconds
Interface Poll frequency 5 seconds, holdtime 25 seconds
Interface Policy 1
Monitored Interfaces 0 of 61 maximum
MAC Address Move Notification Interval not set
Version: Ours 9.8(1), Mate 9.8(1)
Serial Number: Ours 9A81V5LKN5F, Mate 9AK137KWDWB
Last Failover at: 03:20:54 UTC Jan 25 2022
This host: Primary - Active
Active time: 416 (sec)
slot 0: empty
Interface outside (200.1.1.1): Normal (Not-Monitored)
Interface inside (172.16.1.1): Normal (Not-Monitored)
Interface dmz (10.1.1.1): Normal (Not-Monitored)
Other host: Secondary - Standby Ready
Active time: 0 (sec)
Interface outside (0.0.0.0): No Link (Not-Monitored)
Interface inside (0.0.0.0): No Link (Not-Monitored)
Interface dmz (0.0.0.0): No Link (Not-Monitored)
<OUTPUT TRUNCATED>
