Below is a summary of the steps in order to perform a zero downtime upgrade on a Cisco ASA Active/Standby firewall pair.
1) TFTP the ASA image (and ASDM) on both ASA firewalls.
2) Change the boot variable on the Active ASA (ASA1).
3) Save the config by issuing a write memory on ASA1.
4) On the Active ASA (ASA1) issue the failover reload-standby command to reboot ASA2.
5) Wait for ASA2 to reboot and sync its configuration.
6) Issue the no failover active command on the Active ASA (ASA1)
7) SSH into the new Active ASA (ASA2) and issue the failover reload-standby to reboot ASA1.
8) Wait for ASA1 to reboot and sync its configuration.
9) Issue the no failover active command on the Active ASA (ASA2)
So I've TFTP'd first the image and ASDM on both the ASA firewalls using the copy tftp flash (or copy tftp disk0) command.
ASA1/pri/act# copy tftp://asa944-2-smp-k8.bin disk0:/asa944-2-smp-k8.bin
Address or name of remote host [172.27.25.254]?
Source filename [asa944-2-smp-k8.bin]?
Destination filename [asa944-2-smp-k8.bin?
Accessing tftp://172.27.25.254/asa944-2-smp-k8.bin...!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
<OUTPUT TRUNCATED>
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
24827904 bytes copied in 9477.840 secs (2619 bytes/sec)
ASA1/pri/act# failover ?
active Make this system to be the active unit of the failover pair
exec Execute command on the designated unit
reload-standby Force standby unit to reboot
reset Force an unit or failover group to an unfailed state
ciscoasa/pri/act# failover exec ?
active Execute command on the active unit
mate Execute command on the peer unit
standby Execute command on the standby unit
ciscoasa1/pri/act# failover exec mate ?
LINE Command String
ASA1/pri/act# failover exec mate dir // CHECK THE DIRECTORY AND FILES ON ASA2 FLASH
Directory of disk0:/
11 drwx 4096 14:59:42 Aug 28 2014 log
22 drwx 4096 15:00:10 Aug 28 2014 crypto_archive
25 drwx 4096 15:00:18 Aug 28 2014 coredumpinfo
45 -rwx 38191104 15:02:16 Aug 28 2014 asa912-smp-k8.bin
46 -rwx 18097844 15:03:48 Aug 28 2014 asdm-713.bin
47 drwx 4096 09:53:44 Jan 23 2015 tmp
48 -rwx 100 10:12:26 Jan 23 2015 upgrade_startup_errors_201501231012.log
49 -rwx 12998641 15:06:32 Aug 28 2014 csd_3.5.2008-k9.pkg
50 drwx 4096 15:06:34 Aug 28 2014 sdesktop
51 -rwx 6487517 15:06:34 Aug 28 2014 anyconnect-macosx-i386-2.5.2014-k9.pkg
52 -rwx 6689498 15:06:36 Aug 28 2014 anyconnect-linux-2.5.2014-k9.pkg
53 -rwx 4678691 15:06:36 Aug 28 2014 anyconnect-win-2.5.2014-k9.pkg
54 -rwx 100 10:22:20 Jan 23 2015 upgrade_startup_errors_201501231022.log
55 -rwx 100 10:29:06 Jan 23 2015 upgrade_startup_errors_201501231029.log
23 drwx 4096 10:56:34 Jan 23 2015 snmp
56 -rwx 100 12:05:07 Mar 15 2017 upgrade_startup_errors_201703151205.log
ASA1/pri/act# failover exec mate copy /noconfirm tftp://172.27.25.254/asa944-2-smp-k8.bin disk0:/asa944-2-smp-k8.bin
Connection closed by foreign host. // NO TFTP TRANSFER SEEN ON 3CDAEMON EVEN WHEN ASA1 WAS DISCONNECTED
The remote access to ASA1 (Active) was disconnected and TFTP wasn't successful using the failover exec mate copy command. So I SSH directly to AS2 management IP address and used the copy tftp flash command instead.
ASA2/sec/stby# copy tftp://172.27.25.254/asa944-2-smp-k8.bin disk0:/asa944-2-smp-k8.bin
Address or name of remote host [172.27.25.254]?
Source filename [asa944-2-smp-k8.bin]?
Destination filename [asa944-2-smp-k8.bin]?
Accessing tftp://172.27.25.254/asa944-2-smp-k8.bin...!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
<OUTPUT TRUNCATED>
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
24827904 bytes copied in 9496.960 secs (2614 bytes/sec)
ASA1/pri/act# failover exec standby dir // YOU CAN EITHER USE KEYWORD mate OR standby
Directory of disk0:/
11 drwx 4096 14:59:42 Aug 28 2014 log
22 drwx 4096 15:00:10 Aug 28 2014 crypto_archive
25 drwx 4096 15:00:18 Aug 28 2014 coredumpinfo
45 -rwx 38191104 15:02:16 Aug 28 2014 asa912-smp-k8.bin
46 -rwx 18097844 15:03:48 Aug 28 2014 asdm-713.bin
47 drwx 4096 09:53:44 Jan 23 2015 tmp
48 -rwx 100 10:12:26 Jan 23 2015 upgrade_startup_errors_201501231012.log
49 -rwx 12998641 15:06:32 Aug 28 2014 csd_3.5.2008-k9.pkg
50 drwx 4096 15:06:34 Aug 28 2014 sdesktop
51 -rwx 6487517 15:06:34 Aug 28 2014 anyconnect-macosx-i386-2.5.2014-k9.pkg
52 -rwx 6689498 15:06:36 Aug 28 2014 anyconnect-linux-2.5.2014-k9.pkg
53 -rwx 4678691 15:06:36 Aug 28 2014 anyconnect-win-2.5.2014-k9.pkg
54 -rwx 100 10:22:20 Jan 23 2015 upgrade_startup_errors_201501231022.log
55 -rwx 100 10:29:06 Jan 23 2015 upgrade_startup_errors_201501231029.log
23 drwx 4096 10:56:34 Jan 23 2015 snmp
56 -rwx 100 12:05:07 Mar 15 2017 upgrade_startup_errors_201703151205.log
125 -rwx 73635840 22:49:24 Mar 17 2017 asa944-2-smp-k8.bin
126 -rwx 26729944 03:25:37 Mar 18 2017 asdm-771-150.bin
Below are the failover and current boot variable config.
ASA1/act/pri# show run failover
failover
failover lan unit primary
failover lan interface folink GigabitEthernet0/2
failover key cisco
failover link folink GigabitEthernet0/2
failover interface ip folink 192.168.1.1 255.255.255.252 standby 192.168.1.2
ASA1/act/pri# show run boot
boot system disk0:/asa912-smp-k8.bin
ASA1/act/pri# show run asdm
asdm image disk0:/asdm-713.bin
no asdm history enable
I needed to upgrade the ASA pair to 9.4(4) image and ASDM 7.7.1.
ASA1/act/pri# dir
Directory of disk0:/
11 drwx 4096 14:59:42 Aug 28 2014 log
22 drwx 4096 15:00:10 Aug 28 2014 crypto_archive
25 drwx 4096 15:00:18 Aug 28 2014 coredumpinfo
45 -rwx 38191104 15:02:16 Aug 28 2014 asa912-smp-k8.bin
46 -rwx 18097844 15:03:48 Aug 28 2014 asdm-713.bin
47 drwx 4096 09:53:44 Jan 23 2015 tmp
48 -rwx 100 10:12:26 Jan 23 2015 upgrade_startup_errors_201501231012.log
49 -rwx 12998641 15:06:32 Aug 28 2014 csd_3.5.2008-k9.pkg
50 drwx 4096 15:06:34 Aug 28 2014 sdesktop
51 -rwx 6487517 15:06:34 Aug 28 2014 anyconnect-macosx-i386-2.5.2014-k9.pkg
52 -rwx 6689498 15:06:36 Aug 28 2014 anyconnect-linux-2.5.2014-k9.pkg
53 -rwx 4678691 15:06:36 Aug 28 2014 anyconnect-win-2.5.2014-k9.pkg
54 -rwx 100 10:22:20 Jan 23 2015 upgrade_startup_errors_201501231022.log
55 -rwx 100 10:29:06 Jan 23 2015 upgrade_startup_errors_201501231029.log
23 drwx 4096 10:56:34 Jan 23 2015 snmp
56 -rwx 100 12:05:07 Mar 15 2017 upgrade_startup_errors_201703151205.log
125 -rwx 73635840 22:49:24 Mar 17 2017 asa944-2-smp-k8.bin
126 -rwx 26729944 03:25:37 Mar 18 2017 asdm-771-150.bin // TFTP TRANSFER OF ASDM IMAGE WASN'T SHOWN.
8238202880 bytes total (8049881088 bytes free)
Change the boot variable using the boot system and asdm image commands.
ASA1/act/pri# configure terminal
ASA1/act/pri(config)# no boot system disk0:/asa912-smp-k8.bin
ASA1/act/pri(config)# no asdm image disk0:/asdm-713.bin
ASA1/act/pri(config)# boot system disk0:/asa944-2-smp-k8.bin
ASA1/act/pri(config)# asdm image disk0:/asdm-771-150.bin
ASA1/act/pri(config)# end
ASA1/act/pri# write memory
Building configuration...
Cryptochecksum: aeb34eaf d1e8a03e 39884930 0d00e844
7125 bytes copied in 0.720 secs
[OK]
Issue a failover reload-standby command on ASA1 (Active) to reboot the ASA2 (Standby) firewall. A log message will display that the mate (ASA2) image version is not identical with ASA1 (Active). You can verify the new image (and ASDM) on ASA2 by issuing failover exec mate show version on ASA1.
ASA1/act/pri# failover reload-standby
<OUTPUT TRUNCATED>
ASA1/act/pri#
************WARNING****WARNING****WARNING********************************
Mate version 9.4(4)2 is not identical with ours 9.1(2)
************WARNING****WARNING****WARNING********************************
Beginning configuration replication: Sending to mate.
End Configuration Replication to mate
ASA1/act/pri# failover exec mate show version
Cisco Adaptive Security Appliance Software Version 9.4(4)2
Device Manager Version 7.7(1)150
Compiled on Thu 16-Feb-17 09:09 PST by builders
System image file is "disk0:/asa944-2-smp-k8.bin"
Config file at boot was "startup-config"
ASA2 up 4 mins 41 secs
failover cluster up 4 days 7 hours
<OUTPUT TRUNCATED>
ASA1/act/pri# show failover
Failover On
Failover unit Primary
Failover LAN Interface: folink GigabitEthernet0/2 (up)
Unit Poll frequency 1 seconds, holdtime 15 seconds
Interface Poll frequency 5 seconds, holdtime 25 seconds
Interface Policy 1
Monitored Interfaces 1 of 216 maximum
Version: Ours 9.1(2), Mate 9.4(4)2
Last Failover at: 08:38:27 UTC Mar 16 2017
This host: Primary - Active
Active time: 371156 (sec)
slot 0: ASA5525 hw/sw rev (1.0/9.1(2)) status (Up Sys)
Interface inside (10.108.12.50): Normal (Monitored)
Other host: Secondary - Standby Ready
Active time: 0 (sec)
slot 0: ASA5525 hw/sw rev (1.0/9.4(4)2) status (Up Sys)
Interface inside (10.108.12.51): Normal (Monitored)
Stateful Failover Logical Update Statistics
Link : folink GigabitEthernet0/2 (up)
Stateful Obj xmit xerr rcv rerr
General 74820 0 39623 0
sys cmd 39624 0 39623 0
up time 0 0 0 0
RPC services 0 0 0 0
TCP conn 0 0 0 0
UDP conn 0 0 0 0
ARP tbl 20978 0 0 0
Xlate_Timeout 0 0 0 0
IPv6 ND tbl 0 0 0 0
VPN IKEv1 SA 0 0 0 0
VPN IKEv1 P2 0 0 0 0
VPN IKEv2 SA 0 0 0 0
VPN IKEv2 P2 0 0 0 0
VPN CTCP upd 0 0 0 0
VPN SDI upd 0 0 0 0
VPN DHCP upd 0 0 0 0
SIP Session 0 0 0 0
Route Session 14216 0 0 0
User-Identity 2 0 0 0
CTS SGTNAME 0 0 0 0
CTS PAC 0 0 0 0
TrustSec-SXP 0 0 0 0
IPv6 Route 0 0 0 0
Logical Update Queue Information
Cur Max Total
Recv Q: 0 15 39625
Xmit Q: 0 30 217324
ASA1/act/pri# no failover active // FORCE ASA2 (STANDBY) TO BECOME ACTIVE
<SSH ASA1 AND ASA2 WERE DISCONNECTED>
ASA2/stby/sec#
ASA2/stby/sec# Connection closed by foreign host.
ASA2/stby/pri# show failover
Failover On
Failover unit Primary
Failover LAN Interface: folink GigabitEthernet0/2 (up)
Unit Poll frequency 1 seconds, holdtime 15 seconds
Interface Poll frequency 5 seconds, holdtime 25 seconds
Interface Policy 1
Monitored Interfaces 1 of 216 maximum
Version: Ours 9.1(2), Mate 9.4(4)2
Last Failover at: 02:37:14 UTC Mar 20 2017
This host: Primary - Standby Ready
Active time: 371276 (sec)
slot 0: ASA5525 hw/sw rev (1.0/9.1(2)) status (Up Sys)
Interface inside (10.108.12.51): Normal (Monitored)
Other host: Secondary - Active
Active time: 80 (sec)
slot 0: ASA5525 hw/sw rev (1.0/9.4(4)2) status (Up Sys)
Interface inside (10.108.12.50): Normal (Monitored)
<OUTPUT TRUNCATED>
ASA2/act/sec# failover reload-standby // FORCE ASA1 (FORMER ACTIVE) TO REBOOT
ASA2/stby/pri#
************WARNING****WARNING****WARNING********************************
Mate version 9.4(4)2 is not identical with ours 9.1(2)
************WARNING****WARNING****WARNING********************************
***
*** --- SHUTDOWN NOW ---
***
*** Message to all terminals:
***
*** requested by active unit
ASA2/act/sec#
Beginning configuration replication: Sending to mate.
End Configuration Replication to mate
ASA2/act/sec# failover exec mate show version // VERIFY ASA1 HAS THE SAME 9.4 AND 7.7 ASDM IMAGE
Cisco Adaptive Security Appliance Software Version 9.4(4)2
Device Manager Version 7.7(1)150
Compiled on Thu 16-Feb-17 09:09 PST by builders
System image file is "disk0:/asa944-2-smp-k8.bin"
Config file at boot was "startup-config"
ASA2 up 2 mins 20 secs
failover cluster up 4 days 7 hours
<OUTPUT TRUNCATED>
ASA2/act/sec# show failover
Failover On
Failover unit Secondary
Failover LAN Interface: folink GigabitEthernet0/2 (up)
Reconnect timeout 0:00:00
Unit Poll frequency 1 seconds, holdtime 15 seconds
Interface Poll frequency 5 seconds, holdtime 25 seconds
Interface Policy 1
Monitored Interfaces 1 of 216 maximum
MAC Address Move Notification Interval not set
Version: Ours 9.4(4)2, Mate 9.4(4)2 // ASA1 AND ASA2 HAVE THE SAME IMAGE
Last Failover at: 02:37:14 UTC Mar 20 2017
This host: Secondary - Active
Active time: 1178 (sec)
slot 0: ASA5525 hw/sw rev (1.0/9.4(4)2) status (Up Sys)
Interface inside (10.108.12.50): Normal (Monitored)
Other host: Primary - Standby Ready
Active time: 0 (sec)
slot 0: ASA5525 hw/sw rev (1.0/9.4(4)2) status (Up Sys)
Interface inside (10.108.12.51): Normal (Monitored)
<OUTPUT TRUNCATED>
ASA2/act/sec# no failover active // FORCE ASA1 TO BECOME ACTIVE AGAIN
Connection closed by foreign host
ASA1/act/pri# show version
Cisco Adaptive Security Appliance Software Version 9.4(4)2
Device Manager Version 7.7(1)150
Compiled on Thu 16-Feb-17 09:09 PST by builders
System image file is "disk0:/asa944-2-smp-k8.bin"
Config file at boot was "startup-config"
ASA1 up 10 mins 39 secs
failover cluster up 4 days 7 hours
<OUTPUT TRUNCATED>
ASA1/act/pri# show failover
Failover On
Failover unit Primary
Failover LAN Interface: folink GigabitEthernet0/2 (up)
Reconnect timeout 0:00:00
Unit Poll frequency 1 seconds, holdtime 15 seconds
Interface Poll frequency 5 seconds, holdtime 25 seconds
Interface Policy 1
Monitored Interfaces 1 of 216 maximum
MAC Address Move Notification Interval not set
Version: Ours 9.4(4)2, Mate 9.4(4)2
Last Failover at: 02:57:32 UTC Mar 20 2017
This host: Primary - Active
Active time: 48 (sec)
slot 0: ASA5525 hw/sw rev (1.0/9.4(4)2) status (Up Sys)
Interface inside (10.108.12.50): Normal (Monitored)
Other host: Secondary - Standby Ready
Active time: 1219 (sec)
slot 0: ASA5525 hw/sw rev (1.0/9.4(4)2) status (Up Sys)
Interface inside (10.108.12.51): Normal (Monitored)
<OUTPUT TRUNCATED>
ASA1/act/pri# failover exec mate show version
Cisco Adaptive Security Appliance Software Version 9.4(4)2
Device Manager Version 7.7(1)150
Compiled on Thu 16-Feb-17 09:09 PST by builders
System image file is "disk0:/asa944-2-smp-k8.bin"
Config file at boot was "startup-config"
ASA2 up 29 mins 18 secs
failover cluster up 4 days 7 hours
<OUTPUT TRUNCATED>
I've performed a zero downtime upgrade on a Cisco ASA 5520 Active/Standby pair and only used write memory. The Standby firewall kept rebooting in a loop since it didn't find the image configured using the boot system disk0:/ and I should've specified a fallback boot image. You also need to use the write standby in order to replicate the config to the Standby firewall. I also didn't see this log message on the Active firewall.
Beginning configuration replication: Sending to mate.
End Configuration Replication to mate
ASA5520/pri/act# show failover
Failover On
Failover unit Primary
Failover LAN Interface: failover GigabitEthernet0/2 (up)
Unit Poll frequency 1 seconds, holdtime 15 seconds
Interface Poll frequency 5 seconds, holdtime 25 seconds
Interface Policy 1
Monitored Interfaces 2 of 160 maximum
Version: Ours 8.4(3), Mate 8.4(6)
Last Failover at: 12:32:40 UTC Aug 20 2015
This host: Primary - Active
Active time: 52035347 (sec)
slot 0: ASA5520 hw/sw rev (2.0/8.4(3)) status (Up Sys)
Interface outside (1.1.1.1): Normal (Waiting)
Interface inside (192.168.1.1): Normal (Waiting)
slot 1: empty
Other host: Secondary - Cold Standby
Active time: 0 (sec)
slot 0: ASA5520 hw/sw rev (2.0/8.4(6)) status (Up Sys)
Interface outside (1.1.1.2): Unknown (Monitored)
Interface inside (192.168.1.2): Unknown (Monitored)
slot 1: empty
ASA5520/pri/act# show failover state
State Last Failure Reason Date/Time
This host - Primary
Active Ifc Failure 15:16:36 UTC Jun 4 2015
inside: No Link
Other host - Secondary
Cold Standby Comm Failure 16:27:27 UTC Mar 23 2017
====Configuration State===
Sync Done // YOU SHOULD ALSO SEE Sync Done - STANDBY
====Communication State===
// YOU SHOULD SEE Mac set
ASA5520/pri/act# write ?
erase Clear flash memory configuration
memory Save active configuration to the flash
net Save the active configuration to the tftp server
standby Save the active configuration on the active unit to the flash on
the standby unit
terminal Display the current active configuration
<cr>
ASA5520/pri/act# write memory
Building configuration...
Cryptochecksum: aac8cc62 f0d002b1 20122d02 0499661b
36859 bytes copied in 3.290 secs (12286 bytes/sec)
[OK]
ASA5520/pri/act# write standby // DIDN'T USE THIS ON A CISCO 5500-X
Building configuration...
[OK]
ciscoasa5520/pri/act#
Beginning configuration replication: Sending to mate.
End Configuration Replication to mate
1) TFTP the ASA image (and ASDM) on both ASA firewalls.
2) Change the boot variable on the Active ASA (ASA1).
3) Save the config by issuing a write memory on ASA1.
4) On the Active ASA (ASA1) issue the failover reload-standby command to reboot ASA2.
5) Wait for ASA2 to reboot and sync its configuration.
6) Issue the no failover active command on the Active ASA (ASA1)
7) SSH into the new Active ASA (ASA2) and issue the failover reload-standby to reboot ASA1.
8) Wait for ASA1 to reboot and sync its configuration.
9) Issue the no failover active command on the Active ASA (ASA2)
So I've TFTP'd first the image and ASDM on both the ASA firewalls using the copy tftp flash (or copy tftp disk0) command.
ASA1/pri/act# copy tftp://asa944-2-smp-k8.bin disk0:/asa944-2-smp-k8.bin
Address or name of remote host [172.27.25.254]?
Source filename [asa944-2-smp-k8.bin]?
Destination filename [asa944-2-smp-k8.bin?
Accessing tftp://172.27.25.254/asa944-2-smp-k8.bin...!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
<OUTPUT TRUNCATED>
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
24827904 bytes copied in 9477.840 secs (2619 bytes/sec)
ASA1/pri/act# failover ?
active Make this system to be the active unit of the failover pair
exec Execute command on the designated unit
reload-standby Force standby unit to reboot
reset Force an unit or failover group to an unfailed state
ciscoasa/pri/act# failover exec ?
active Execute command on the active unit
mate Execute command on the peer unit
standby Execute command on the standby unit
ciscoasa1/pri/act# failover exec mate ?
LINE Command String
ASA1/pri/act# failover exec mate dir // CHECK THE DIRECTORY AND FILES ON ASA2 FLASH
Directory of disk0:/
11 drwx 4096 14:59:42 Aug 28 2014 log
22 drwx 4096 15:00:10 Aug 28 2014 crypto_archive
25 drwx 4096 15:00:18 Aug 28 2014 coredumpinfo
45 -rwx 38191104 15:02:16 Aug 28 2014 asa912-smp-k8.bin
46 -rwx 18097844 15:03:48 Aug 28 2014 asdm-713.bin
47 drwx 4096 09:53:44 Jan 23 2015 tmp
48 -rwx 100 10:12:26 Jan 23 2015 upgrade_startup_errors_201501231012.log
49 -rwx 12998641 15:06:32 Aug 28 2014 csd_3.5.2008-k9.pkg
50 drwx 4096 15:06:34 Aug 28 2014 sdesktop
51 -rwx 6487517 15:06:34 Aug 28 2014 anyconnect-macosx-i386-2.5.2014-k9.pkg
52 -rwx 6689498 15:06:36 Aug 28 2014 anyconnect-linux-2.5.2014-k9.pkg
53 -rwx 4678691 15:06:36 Aug 28 2014 anyconnect-win-2.5.2014-k9.pkg
54 -rwx 100 10:22:20 Jan 23 2015 upgrade_startup_errors_201501231022.log
55 -rwx 100 10:29:06 Jan 23 2015 upgrade_startup_errors_201501231029.log
23 drwx 4096 10:56:34 Jan 23 2015 snmp
56 -rwx 100 12:05:07 Mar 15 2017 upgrade_startup_errors_201703151205.log
ASA1/pri/act# failover exec mate copy /noconfirm tftp://172.27.25.254/asa944-2-smp-k8.bin disk0:/asa944-2-smp-k8.bin
Connection closed by foreign host. // NO TFTP TRANSFER SEEN ON 3CDAEMON EVEN WHEN ASA1 WAS DISCONNECTED
The remote access to ASA1 (Active) was disconnected and TFTP wasn't successful using the failover exec mate copy command. So I SSH directly to AS2 management IP address and used the copy tftp flash command instead.
ASA2/sec/stby# copy tftp://172.27.25.254/asa944-2-smp-k8.bin disk0:/asa944-2-smp-k8.bin
Address or name of remote host [172.27.25.254]?
Source filename [asa944-2-smp-k8.bin]?
Destination filename [asa944-2-smp-k8.bin]?
Accessing tftp://172.27.25.254/asa944-2-smp-k8.bin...!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
<OUTPUT TRUNCATED>
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
24827904 bytes copied in 9496.960 secs (2614 bytes/sec)
ASA1/pri/act# failover exec standby dir // YOU CAN EITHER USE KEYWORD mate OR standby
Directory of disk0:/
11 drwx 4096 14:59:42 Aug 28 2014 log
22 drwx 4096 15:00:10 Aug 28 2014 crypto_archive
25 drwx 4096 15:00:18 Aug 28 2014 coredumpinfo
45 -rwx 38191104 15:02:16 Aug 28 2014 asa912-smp-k8.bin
46 -rwx 18097844 15:03:48 Aug 28 2014 asdm-713.bin
47 drwx 4096 09:53:44 Jan 23 2015 tmp
48 -rwx 100 10:12:26 Jan 23 2015 upgrade_startup_errors_201501231012.log
49 -rwx 12998641 15:06:32 Aug 28 2014 csd_3.5.2008-k9.pkg
50 drwx 4096 15:06:34 Aug 28 2014 sdesktop
51 -rwx 6487517 15:06:34 Aug 28 2014 anyconnect-macosx-i386-2.5.2014-k9.pkg
52 -rwx 6689498 15:06:36 Aug 28 2014 anyconnect-linux-2.5.2014-k9.pkg
53 -rwx 4678691 15:06:36 Aug 28 2014 anyconnect-win-2.5.2014-k9.pkg
54 -rwx 100 10:22:20 Jan 23 2015 upgrade_startup_errors_201501231022.log
55 -rwx 100 10:29:06 Jan 23 2015 upgrade_startup_errors_201501231029.log
23 drwx 4096 10:56:34 Jan 23 2015 snmp
56 -rwx 100 12:05:07 Mar 15 2017 upgrade_startup_errors_201703151205.log
125 -rwx 73635840 22:49:24 Mar 17 2017 asa944-2-smp-k8.bin
126 -rwx 26729944 03:25:37 Mar 18 2017 asdm-771-150.bin
Below are the failover and current boot variable config.
ASA1/act/pri# show run failover
failover
failover lan unit primary
failover lan interface folink GigabitEthernet0/2
failover key cisco
failover link folink GigabitEthernet0/2
failover interface ip folink 192.168.1.1 255.255.255.252 standby 192.168.1.2
ASA1/act/pri# show run boot
boot system disk0:/asa912-smp-k8.bin
ASA1/act/pri# show run asdm
asdm image disk0:/asdm-713.bin
no asdm history enable
I needed to upgrade the ASA pair to 9.4(4) image and ASDM 7.7.1.
ASA1/act/pri# dir
Directory of disk0:/
11 drwx 4096 14:59:42 Aug 28 2014 log
22 drwx 4096 15:00:10 Aug 28 2014 crypto_archive
25 drwx 4096 15:00:18 Aug 28 2014 coredumpinfo
45 -rwx 38191104 15:02:16 Aug 28 2014 asa912-smp-k8.bin
46 -rwx 18097844 15:03:48 Aug 28 2014 asdm-713.bin
47 drwx 4096 09:53:44 Jan 23 2015 tmp
48 -rwx 100 10:12:26 Jan 23 2015 upgrade_startup_errors_201501231012.log
49 -rwx 12998641 15:06:32 Aug 28 2014 csd_3.5.2008-k9.pkg
50 drwx 4096 15:06:34 Aug 28 2014 sdesktop
51 -rwx 6487517 15:06:34 Aug 28 2014 anyconnect-macosx-i386-2.5.2014-k9.pkg
52 -rwx 6689498 15:06:36 Aug 28 2014 anyconnect-linux-2.5.2014-k9.pkg
53 -rwx 4678691 15:06:36 Aug 28 2014 anyconnect-win-2.5.2014-k9.pkg
54 -rwx 100 10:22:20 Jan 23 2015 upgrade_startup_errors_201501231022.log
55 -rwx 100 10:29:06 Jan 23 2015 upgrade_startup_errors_201501231029.log
23 drwx 4096 10:56:34 Jan 23 2015 snmp
56 -rwx 100 12:05:07 Mar 15 2017 upgrade_startup_errors_201703151205.log
125 -rwx 73635840 22:49:24 Mar 17 2017 asa944-2-smp-k8.bin
126 -rwx 26729944 03:25:37 Mar 18 2017 asdm-771-150.bin // TFTP TRANSFER OF ASDM IMAGE WASN'T SHOWN.
8238202880 bytes total (8049881088 bytes free)
Change the boot variable using the boot system and asdm image commands.
ASA1/act/pri# configure terminal
ASA1/act/pri(config)# no boot system disk0:/asa912-smp-k8.bin
ASA1/act/pri(config)# no asdm image disk0:/asdm-713.bin
ASA1/act/pri(config)# boot system disk0:/asa944-2-smp-k8.bin
ASA1/act/pri(config)# asdm image disk0:/asdm-771-150.bin
ASA1/act/pri(config)# end
ASA1/act/pri# write memory
Building configuration...
Cryptochecksum: aeb34eaf d1e8a03e 39884930 0d00e844
7125 bytes copied in 0.720 secs
[OK]
Issue a failover reload-standby command on ASA1 (Active) to reboot the ASA2 (Standby) firewall. A log message will display that the mate (ASA2) image version is not identical with ASA1 (Active). You can verify the new image (and ASDM) on ASA2 by issuing failover exec mate show version on ASA1.
ASA1/act/pri# failover reload-standby
<OUTPUT TRUNCATED>
ASA1/act/pri#
************WARNING****WARNING****WARNING********************************
Mate version 9.4(4)2 is not identical with ours 9.1(2)
************WARNING****WARNING****WARNING********************************
Beginning configuration replication: Sending to mate.
End Configuration Replication to mate
ASA1/act/pri# failover exec mate show version
Cisco Adaptive Security Appliance Software Version 9.4(4)2
Device Manager Version 7.7(1)150
Compiled on Thu 16-Feb-17 09:09 PST by builders
System image file is "disk0:/asa944-2-smp-k8.bin"
Config file at boot was "startup-config"
ASA2 up 4 mins 41 secs
failover cluster up 4 days 7 hours
<OUTPUT TRUNCATED>
ASA1/act/pri# show failover
Failover On
Failover unit Primary
Failover LAN Interface: folink GigabitEthernet0/2 (up)
Unit Poll frequency 1 seconds, holdtime 15 seconds
Interface Poll frequency 5 seconds, holdtime 25 seconds
Interface Policy 1
Monitored Interfaces 1 of 216 maximum
Version: Ours 9.1(2), Mate 9.4(4)2
Last Failover at: 08:38:27 UTC Mar 16 2017
This host: Primary - Active
Active time: 371156 (sec)
slot 0: ASA5525 hw/sw rev (1.0/9.1(2)) status (Up Sys)
Interface inside (10.108.12.50): Normal (Monitored)
Other host: Secondary - Standby Ready
Active time: 0 (sec)
slot 0: ASA5525 hw/sw rev (1.0/9.4(4)2) status (Up Sys)
Interface inside (10.108.12.51): Normal (Monitored)
Stateful Failover Logical Update Statistics
Link : folink GigabitEthernet0/2 (up)
Stateful Obj xmit xerr rcv rerr
General 74820 0 39623 0
sys cmd 39624 0 39623 0
up time 0 0 0 0
RPC services 0 0 0 0
TCP conn 0 0 0 0
UDP conn 0 0 0 0
ARP tbl 20978 0 0 0
Xlate_Timeout 0 0 0 0
IPv6 ND tbl 0 0 0 0
VPN IKEv1 SA 0 0 0 0
VPN IKEv1 P2 0 0 0 0
VPN IKEv2 SA 0 0 0 0
VPN IKEv2 P2 0 0 0 0
VPN CTCP upd 0 0 0 0
VPN SDI upd 0 0 0 0
VPN DHCP upd 0 0 0 0
SIP Session 0 0 0 0
Route Session 14216 0 0 0
User-Identity 2 0 0 0
CTS SGTNAME 0 0 0 0
CTS PAC 0 0 0 0
TrustSec-SXP 0 0 0 0
IPv6 Route 0 0 0 0
Logical Update Queue Information
Cur Max Total
Recv Q: 0 15 39625
Xmit Q: 0 30 217324
ASA1/act/pri# no failover active // FORCE ASA2 (STANDBY) TO BECOME ACTIVE
<SSH ASA1 AND ASA2 WERE DISCONNECTED>
ASA2/stby/sec#
ASA2/stby/sec# Connection closed by foreign host.
ASA2/stby/pri# show failover
Failover On
Failover unit Primary
Failover LAN Interface: folink GigabitEthernet0/2 (up)
Unit Poll frequency 1 seconds, holdtime 15 seconds
Interface Poll frequency 5 seconds, holdtime 25 seconds
Interface Policy 1
Monitored Interfaces 1 of 216 maximum
Version: Ours 9.1(2), Mate 9.4(4)2
Last Failover at: 02:37:14 UTC Mar 20 2017
This host: Primary - Standby Ready
Active time: 371276 (sec)
slot 0: ASA5525 hw/sw rev (1.0/9.1(2)) status (Up Sys)
Interface inside (10.108.12.51): Normal (Monitored)
Other host: Secondary - Active
Active time: 80 (sec)
slot 0: ASA5525 hw/sw rev (1.0/9.4(4)2) status (Up Sys)
Interface inside (10.108.12.50): Normal (Monitored)
<OUTPUT TRUNCATED>
ASA2/act/sec# failover reload-standby // FORCE ASA1 (FORMER ACTIVE) TO REBOOT
ASA2/stby/pri#
************WARNING****WARNING****WARNING********************************
Mate version 9.4(4)2 is not identical with ours 9.1(2)
************WARNING****WARNING****WARNING********************************
***
*** --- SHUTDOWN NOW ---
***
*** Message to all terminals:
***
*** requested by active unit
ASA2/act/sec#
Beginning configuration replication: Sending to mate.
End Configuration Replication to mate
ASA2/act/sec# failover exec mate show version // VERIFY ASA1 HAS THE SAME 9.4 AND 7.7 ASDM IMAGE
Cisco Adaptive Security Appliance Software Version 9.4(4)2
Device Manager Version 7.7(1)150
Compiled on Thu 16-Feb-17 09:09 PST by builders
System image file is "disk0:/asa944-2-smp-k8.bin"
Config file at boot was "startup-config"
ASA2 up 2 mins 20 secs
failover cluster up 4 days 7 hours
<OUTPUT TRUNCATED>
ASA2/act/sec# show failover
Failover On
Failover unit Secondary
Failover LAN Interface: folink GigabitEthernet0/2 (up)
Reconnect timeout 0:00:00
Unit Poll frequency 1 seconds, holdtime 15 seconds
Interface Poll frequency 5 seconds, holdtime 25 seconds
Interface Policy 1
Monitored Interfaces 1 of 216 maximum
MAC Address Move Notification Interval not set
Version: Ours 9.4(4)2, Mate 9.4(4)2 // ASA1 AND ASA2 HAVE THE SAME IMAGE
Last Failover at: 02:37:14 UTC Mar 20 2017
This host: Secondary - Active
Active time: 1178 (sec)
slot 0: ASA5525 hw/sw rev (1.0/9.4(4)2) status (Up Sys)
Interface inside (10.108.12.50): Normal (Monitored)
Other host: Primary - Standby Ready
Active time: 0 (sec)
slot 0: ASA5525 hw/sw rev (1.0/9.4(4)2) status (Up Sys)
Interface inside (10.108.12.51): Normal (Monitored)
<OUTPUT TRUNCATED>
ASA2/act/sec# no failover active // FORCE ASA1 TO BECOME ACTIVE AGAIN
Connection closed by foreign host
ASA1/act/pri# show version
Cisco Adaptive Security Appliance Software Version 9.4(4)2
Device Manager Version 7.7(1)150
Compiled on Thu 16-Feb-17 09:09 PST by builders
System image file is "disk0:/asa944-2-smp-k8.bin"
Config file at boot was "startup-config"
ASA1 up 10 mins 39 secs
failover cluster up 4 days 7 hours
<OUTPUT TRUNCATED>
ASA1/act/pri# show failover
Failover On
Failover unit Primary
Failover LAN Interface: folink GigabitEthernet0/2 (up)
Reconnect timeout 0:00:00
Unit Poll frequency 1 seconds, holdtime 15 seconds
Interface Poll frequency 5 seconds, holdtime 25 seconds
Interface Policy 1
Monitored Interfaces 1 of 216 maximum
MAC Address Move Notification Interval not set
Version: Ours 9.4(4)2, Mate 9.4(4)2
Last Failover at: 02:57:32 UTC Mar 20 2017
This host: Primary - Active
Active time: 48 (sec)
slot 0: ASA5525 hw/sw rev (1.0/9.4(4)2) status (Up Sys)
Interface inside (10.108.12.50): Normal (Monitored)
Other host: Secondary - Standby Ready
Active time: 1219 (sec)
slot 0: ASA5525 hw/sw rev (1.0/9.4(4)2) status (Up Sys)
Interface inside (10.108.12.51): Normal (Monitored)
<OUTPUT TRUNCATED>
ASA1/act/pri# failover exec mate show version
Cisco Adaptive Security Appliance Software Version 9.4(4)2
Device Manager Version 7.7(1)150
Compiled on Thu 16-Feb-17 09:09 PST by builders
System image file is "disk0:/asa944-2-smp-k8.bin"
Config file at boot was "startup-config"
ASA2 up 29 mins 18 secs
failover cluster up 4 days 7 hours
<OUTPUT TRUNCATED>
I've performed a zero downtime upgrade on a Cisco ASA 5520 Active/Standby pair and only used write memory. The Standby firewall kept rebooting in a loop since it didn't find the image configured using the boot system disk0:/ and I should've specified a fallback boot image. You also need to use the write standby in order to replicate the config to the Standby firewall. I also didn't see this log message on the Active firewall.
Beginning configuration replication: Sending to mate.
End Configuration Replication to mate
ASA5520/pri/act# show failover
Failover On
Failover unit Primary
Failover LAN Interface: failover GigabitEthernet0/2 (up)
Unit Poll frequency 1 seconds, holdtime 15 seconds
Interface Poll frequency 5 seconds, holdtime 25 seconds
Interface Policy 1
Monitored Interfaces 2 of 160 maximum
Version: Ours 8.4(3), Mate 8.4(6)
Last Failover at: 12:32:40 UTC Aug 20 2015
This host: Primary - Active
Active time: 52035347 (sec)
slot 0: ASA5520 hw/sw rev (2.0/8.4(3)) status (Up Sys)
Interface outside (1.1.1.1): Normal (Waiting)
Interface inside (192.168.1.1): Normal (Waiting)
slot 1: empty
Other host: Secondary - Cold Standby
Active time: 0 (sec)
slot 0: ASA5520 hw/sw rev (2.0/8.4(6)) status (Up Sys)
Interface outside (1.1.1.2): Unknown (Monitored)
Interface inside (192.168.1.2): Unknown (Monitored)
slot 1: empty
ASA5520/pri/act# show failover state
State Last Failure Reason Date/Time
This host - Primary
Active Ifc Failure 15:16:36 UTC Jun 4 2015
inside: No Link
Other host - Secondary
Cold Standby Comm Failure 16:27:27 UTC Mar 23 2017
====Configuration State===
Sync Done // YOU SHOULD ALSO SEE Sync Done - STANDBY
====Communication State===
// YOU SHOULD SEE Mac set
ASA5520/pri/act# write ?
erase Clear flash memory configuration
memory Save active configuration to the flash
net Save the active configuration to the tftp server
standby Save the active configuration on the active unit to the flash on
the standby unit
terminal Display the current active configuration
<cr>
ASA5520/pri/act# write memory
Building configuration...
Cryptochecksum: aac8cc62 f0d002b1 20122d02 0499661b
36859 bytes copied in 3.290 secs (12286 bytes/sec)
[OK]
ASA5520/pri/act# write standby // DIDN'T USE THIS ON A CISCO 5500-X
Building configuration...
[OK]
ciscoasa5520/pri/act#
Beginning configuration replication: Sending to mate.
End Configuration Replication to mate
Great article.
ReplyDelete