Thursday, June 8, 2017

Zero Downtime Upgrade on a Cisco ASA Active-Standby Firewall Pair

Below is a summary of the steps in order to perform a zero downtime upgrade on a Cisco ASA Active/Standby firewall pair.

1) TFTP the ASA image (and ASDM) on both ASA firewalls.

2) Change the boot variable on the Active ASA (ASA1).

3) Save the config by issuing a write memory on ASA1.

4) On the Active ASA (ASA1) issue the failover reload-standby command to reboot ASA2.

5) Wait for ASA2 to reboot and sync its configuration.

6) Issue the no failover active command on the Active ASA (ASA1)

7) SSH into the new Active ASA (ASA2) and issue the failover reload-standby to reboot ASA1.

8) Wait for ASA1 to reboot and sync its configuration.

9) Issue the no failover active command on the Active ASA (ASA2)


So I've TFTP'd first the image and ASDM on both the ASA firewalls using the copy tftp flash (or copy tftp disk0) command.

ASA1/pri/act# copy tftp://asa944-2-smp-k8.bin disk0:/asa944-2-smp-k8.bin

Address or name of remote host [172.27.25.254]?

Source filename [asa944-2-smp-k8.bin]?

Destination filename [asa944-2-smp-k8.bin?

Accessing tftp://172.27.25.254/asa944-2-smp-k8.bin...!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

<OUTPUT TRUNCATED>

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
24827904 bytes copied in 9477.840 secs (2619 bytes/sec)

ASA1/pri/act# failover ?

  active          Make this system to be the active unit of the failover pair
  exec            Execute command on the designated unit
  reload-standby  Force standby unit to reboot
  reset           Force an unit or failover group to an unfailed state
ciscoasa/pri/act# failover exec ?

  active   Execute command on the active unit
  mate     Execute command on the peer unit
  standby  Execute command on the standby unit
ciscoasa1/pri/act# failover exec mate ?

  LINE  Command String

ASA1/pri/act# failover exec mate dir      // CHECK THE DIRECTORY AND FILES ON ASA2 FLASH

Directory of disk0:/

11     drwx  4096         14:59:42 Aug 28 2014  log
22     drwx  4096         15:00:10 Aug 28 2014  crypto_archive
25     drwx  4096         15:00:18 Aug 28 2014  coredumpinfo
45     -rwx  38191104     15:02:16 Aug 28 2014  asa912-smp-k8.bin
46     -rwx  18097844     15:03:48 Aug 28 2014  asdm-713.bin
47     drwx  4096         09:53:44 Jan 23 2015  tmp
48     -rwx  100          10:12:26 Jan 23 2015  upgrade_startup_errors_201501231012.log
49     -rwx  12998641     15:06:32 Aug 28 2014  csd_3.5.2008-k9.pkg
50     drwx  4096         15:06:34 Aug 28 2014  sdesktop
51     -rwx  6487517      15:06:34 Aug 28 2014  anyconnect-macosx-i386-2.5.2014-k9.pkg
52     -rwx  6689498      15:06:36 Aug 28 2014  anyconnect-linux-2.5.2014-k9.pkg
53     -rwx  4678691      15:06:36 Aug 28 2014  anyconnect-win-2.5.2014-k9.pkg
54     -rwx  100          10:22:20 Jan 23 2015  upgrade_startup_errors_201501231022.log
55     -rwx  100          10:29:06 Jan 23 2015  upgrade_startup_errors_201501231029.log
23     drwx  4096         10:56:34 Jan 23 2015  snmp
56     -rwx  100          12:05:07 Mar 15 2017  upgrade_startup_errors_201703151205.log

ASA1/pri/act# failover exec mate copy /noconfirm tftp://172.27.25.254/asa944-2-smp-k8.bin disk0:/asa944-2-smp-k8.bin
Connection closed by foreign host.       // NO TFTP TRANSFER SEEN ON 3CDAEMON EVEN WHEN ASA1 WAS DISCONNECTED


The remote access to ASA1 (Active) was disconnected and TFTP wasn't successful using the failover exec mate copy command. So I SSH directly to AS2 management IP address and used the copy tftp flash command instead.

ASA2/sec/stby# copy tftp://172.27.25.254/asa944-2-smp-k8.bin disk0:/asa944-2-smp-k8.bin

Address or name of remote host [172.27.25.254]?

Source filename [asa944-2-smp-k8.bin]?

Destination filename [asa944-2-smp-k8.bin]?

Accessing tftp://172.27.25.254/asa944-2-smp-k8.bin...!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!


<OUTPUT TRUNCATED>


!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
24827904 bytes copied in 9496.960 secs (2614 bytes/sec)

ASA1/pri/act# failover exec standby dir      // YOU CAN EITHER USE KEYWORD mate OR standby

Directory of disk0:/

11     drwx  4096         14:59:42 Aug 28 2014  log
22     drwx  4096         15:00:10 Aug 28 2014  crypto_archive
25     drwx  4096         15:00:18 Aug 28 2014  coredumpinfo
45     -rwx  38191104     15:02:16 Aug 28 2014  asa912-smp-k8.bin
46     -rwx  18097844     15:03:48 Aug 28 2014  asdm-713.bin
47     drwx  4096         09:53:44 Jan 23 2015  tmp
48     -rwx  100          10:12:26 Jan 23 2015  upgrade_startup_errors_201501231012.log
49     -rwx  12998641     15:06:32 Aug 28 2014  csd_3.5.2008-k9.pkg
50     drwx  4096         15:06:34 Aug 28 2014  sdesktop
51     -rwx  6487517      15:06:34 Aug 28 2014  anyconnect-macosx-i386-2.5.2014-k9.pkg
52     -rwx  6689498      15:06:36 Aug 28 2014  anyconnect-linux-2.5.2014-k9.pkg
53     -rwx  4678691      15:06:36 Aug 28 2014  anyconnect-win-2.5.2014-k9.pkg
54     -rwx  100          10:22:20 Jan 23 2015  upgrade_startup_errors_201501231022.log
55     -rwx  100          10:29:06 Jan 23 2015  upgrade_startup_errors_201501231029.log
23     drwx  4096         10:56:34 Jan 23 2015  snmp
56     -rwx  100          12:05:07 Mar 15 2017  upgrade_startup_errors_201703151205.log
125    -rwx  73635840     22:49:24 Mar 17 2017  asa944-2-smp-k8.bin
126    -rwx  26729944     03:25:37 Mar 18 2017  asdm-771-150.bin


Below are the failover and current boot variable config.

ASA1/act/pri#  show run failover
failover
failover lan unit primary
failover lan interface folink GigabitEthernet0/2
failover key cisco
failover link folink GigabitEthernet0/2
failover interface ip folink 192.168.1.1 255.255.255.252 standby 192.168.1.2

ASA1/act/pri# show run boot
boot system disk0:/asa912-smp-k8.bin

ASA1/act/pri# show run asdm
asdm image disk0:/asdm-713.bin
no asdm history enable


I needed to upgrade the ASA pair to 9.4(4) image and ASDM 7.7.1.

ASA1/act/pri# dir

Directory of disk0:/

11     drwx  4096         14:59:42 Aug 28 2014  log
22     drwx  4096         15:00:10 Aug 28 2014  crypto_archive
25     drwx  4096         15:00:18 Aug 28 2014  coredumpinfo
45     -rwx  38191104     15:02:16 Aug 28 2014  asa912-smp-k8.bin
46     -rwx  18097844     15:03:48 Aug 28 2014  asdm-713.bin
47     drwx  4096         09:53:44 Jan 23 2015  tmp
48     -rwx  100          10:12:26 Jan 23 2015  upgrade_startup_errors_201501231012.log
49     -rwx  12998641     15:06:32 Aug 28 2014  csd_3.5.2008-k9.pkg
50     drwx  4096         15:06:34 Aug 28 2014  sdesktop
51     -rwx  6487517      15:06:34 Aug 28 2014  anyconnect-macosx-i386-2.5.2014-k9.pkg
52     -rwx  6689498      15:06:36 Aug 28 2014  anyconnect-linux-2.5.2014-k9.pkg
53     -rwx  4678691      15:06:36 Aug 28 2014  anyconnect-win-2.5.2014-k9.pkg
54     -rwx  100          10:22:20 Jan 23 2015  upgrade_startup_errors_201501231022.log
55     -rwx  100          10:29:06 Jan 23 2015  upgrade_startup_errors_201501231029.log
23     drwx  4096         10:56:34 Jan 23 2015  snmp
56     -rwx  100          12:05:07 Mar 15 2017  upgrade_startup_errors_201703151205.log
125    -rwx  73635840     22:49:24 Mar 17 2017  asa944-2-smp-k8.bin
126    -rwx  26729944     03:25:37 Mar 18 2017  asdm-771-150.bin         // TFTP TRANSFER OF ASDM IMAGE WASN'T SHOWN.

8238202880 bytes total (8049881088 bytes free)


Change the boot variable using the boot system and asdm image commands.

ASA1/act/pri# configure terminal
ASA1/act/pri(config)# no boot system disk0:/asa912-smp-k8.bin
ASA1/act/pri(config)# no asdm image disk0:/asdm-713.bin
ASA1/act/pri(config)# boot system disk0:/asa944-2-smp-k8.bin
ASA1/act/pri(config)# asdm image disk0:/asdm-771-150.bin
ASA1/act/pri(config)# end
ASA1/act/pri# write memory
Building configuration...
Cryptochecksum: aeb34eaf d1e8a03e 39884930 0d00e844

7125 bytes copied in 0.720 secs
[OK]


Issue a failover reload-standby command on ASA1 (Active) to reboot the ASA2 (Standby) firewall. A log message will display that the mate (ASA2) image version is not identical with ASA1 (Active). You can verify the new image (and ASDM) on ASA2 by issuing failover exec mate show version on ASA1.

ASA1/act/pri# failover reload-standby


<OUTPUT TRUNCATED>


ASA1/act/pri#
************WARNING****WARNING****WARNING********************************
   Mate version 9.4(4)2 is not identical with ours 9.1(2)
************WARNING****WARNING****WARNING********************************
Beginning configuration replication: Sending to mate.
End Configuration Replication to mate       

ASA1/act/pri# failover exec mate show version

Cisco Adaptive Security Appliance Software Version 9.4(4)2
Device Manager Version 7.7(1)150

Compiled on Thu 16-Feb-17 09:09 PST by builders
System image file is "disk0:/asa944-2-smp-k8.bin"
Config file at boot was "startup-config"

ASA2 up 4 mins 41 secs
failover cluster up 4 days 7 hours


<OUTPUT TRUNCATED>


ASA1/act/pri# show failover
Failover On
Failover unit Primary
Failover LAN Interface: folink GigabitEthernet0/2 (up)
Unit Poll frequency 1 seconds, holdtime 15 seconds
Interface Poll frequency 5 seconds, holdtime 25 seconds
Interface Policy 1
Monitored Interfaces 1 of 216 maximum
Version: Ours 9.1(2), Mate 9.4(4)2
Last Failover at: 08:38:27 UTC Mar 16 2017
        This host: Primary - Active
                Active time: 371156 (sec)
                slot 0: ASA5525 hw/sw rev (1.0/9.1(2)) status (Up Sys)
                  Interface inside (10.108.12.50): Normal (Monitored)
        Other host: Secondary - Standby Ready
                Active time: 0 (sec)
                slot 0: ASA5525 hw/sw rev (1.0/9.4(4)2) status (Up Sys)
                  Interface inside (10.108.12.51): Normal (Monitored)

Stateful Failover Logical Update Statistics
        Link : folink GigabitEthernet0/2 (up)
        Stateful Obj    xmit       xerr       rcv        rerr
        General         74820      0          39623      0
        sys cmd         39624      0          39623      0
        up time         0          0          0          0
        RPC services    0          0          0          0
        TCP conn        0          0          0          0
        UDP conn        0          0          0          0
        ARP tbl         20978      0          0          0
        Xlate_Timeout   0          0          0          0
        IPv6 ND tbl     0          0          0          0
        VPN IKEv1 SA    0          0          0          0
        VPN IKEv1 P2    0          0          0          0
        VPN IKEv2 SA    0          0          0          0
        VPN IKEv2 P2    0          0          0          0
        VPN CTCP upd    0          0          0          0
        VPN SDI upd     0          0          0          0
        VPN DHCP upd    0          0          0          0
        SIP Session     0          0          0          0
        Route Session   14216      0          0          0
        User-Identity   2          0          0          0
        CTS SGTNAME     0          0          0          0
        CTS PAC         0          0          0          0
        TrustSec-SXP    0          0          0          0
        IPv6 Route      0          0          0          0

        Logical Update Queue Information
                        Cur     Max     Total
        Recv Q:         0       15      39625
        Xmit Q:         0       30      217324


ASA1/act/pri# no failover active     // FORCE ASA2 (STANDBY) TO BECOME ACTIVE


<SSH ASA1 AND ASA2 WERE DISCONNECTED>

ASA2/stby/sec#
ASA2/stby/sec# Connection closed by foreign host.


ASA2/stby/pri# show failover
Failover On
Failover unit Primary
Failover LAN Interface: folink GigabitEthernet0/2 (up)
Unit Poll frequency 1 seconds, holdtime 15 seconds
Interface Poll frequency 5 seconds, holdtime 25 seconds
Interface Policy 1
Monitored Interfaces 1 of 216 maximum
Version: Ours 9.1(2), Mate 9.4(4)2
Last Failover at: 02:37:14 UTC Mar 20 2017
        This host: Primary - Standby Ready
                Active time: 371276 (sec)
                slot 0: ASA5525 hw/sw rev (1.0/9.1(2)) status (Up Sys)
                  Interface inside (10.108.12.51): Normal (Monitored)
        Other host: Secondary - Active
                Active time: 80 (sec)
                slot 0: ASA5525 hw/sw rev (1.0/9.4(4)2) status (Up Sys)
                  Interface inside (10.108.12.50): Normal (Monitored)


<OUTPUT TRUNCATED>


ASA2/act/sec# failover reload-standby      // FORCE ASA1 (FORMER ACTIVE) TO REBOOT


ASA2/stby/pri#
************WARNING****WARNING****WARNING********************************
   Mate version 9.4(4)2 is not identical with ours 9.1(2)
************WARNING****WARNING****WARNING********************************

***
*** --- SHUTDOWN NOW ---
***
*** Message to all terminals:
***
***   requested by active unit


ASA2/act/sec#
Beginning configuration replication: Sending to mate.
End Configuration Replication to mate

ASA2/act/sec# failover exec mate show version        // VERIFY ASA1 HAS THE SAME 9.4 AND 7.7 ASDM IMAGE

Cisco Adaptive Security Appliance Software Version 9.4(4)2
Device Manager Version 7.7(1)150

Compiled on Thu 16-Feb-17 09:09 PST by builders
System image file is "disk0:/asa944-2-smp-k8.bin"
Config file at boot was "startup-config"

ASA2 up 2 mins 20 secs
failover cluster up 4 days 7 hours



<OUTPUT TRUNCATED>


ASA2/act/sec# show failover
Failover On
Failover unit Secondary
Failover LAN Interface: folink GigabitEthernet0/2 (up)
Reconnect timeout 0:00:00
Unit Poll frequency 1 seconds, holdtime 15 seconds
Interface Poll frequency 5 seconds, holdtime 25 seconds
Interface Policy 1
Monitored Interfaces 1 of 216 maximum
MAC Address Move Notification Interval not set
Version: Ours 9.4(4)2, Mate 9.4(4)2          // ASA1 AND ASA2 HAVE THE SAME IMAGE
Last Failover at: 02:37:14 UTC Mar 20 2017
        This host: Secondary - Active
                Active time: 1178 (sec)
                slot 0: ASA5525 hw/sw rev (1.0/9.4(4)2) status (Up Sys)
                  Interface inside (10.108.12.50): Normal (Monitored)
        Other host: Primary - Standby Ready
                Active time: 0 (sec)
                slot 0: ASA5525 hw/sw rev (1.0/9.4(4)2) status (Up Sys)
                  Interface inside (10.108.12.51): Normal (Monitored)


<OUTPUT TRUNCATED>


ASA2/act/sec# no failover active        // FORCE ASA1 TO BECOME ACTIVE AGAIN
Connection closed by foreign host

ASA1/act/pri# show version

Cisco Adaptive Security Appliance Software Version 9.4(4)2
Device Manager Version 7.7(1)150

Compiled on Thu 16-Feb-17 09:09 PST by builders
System image file is "disk0:/asa944-2-smp-k8.bin"
Config file at boot was "startup-config"

ASA1 up 10 mins 39 secs
failover cluster up 4 days 7 hours


<OUTPUT TRUNCATED>


ASA1/act/pri# show failover
Failover On
Failover unit Primary
Failover LAN Interface: folink GigabitEthernet0/2 (up)
Reconnect timeout 0:00:00
Unit Poll frequency 1 seconds, holdtime 15 seconds
Interface Poll frequency 5 seconds, holdtime 25 seconds
Interface Policy 1
Monitored Interfaces 1 of 216 maximum
MAC Address Move Notification Interval not set
Version: Ours 9.4(4)2, Mate 9.4(4)2
Last Failover at: 02:57:32 UTC Mar 20 2017
        This host: Primary - Active
                Active time: 48 (sec)
                slot 0: ASA5525 hw/sw rev (1.0/9.4(4)2) status (Up Sys)
                  Interface inside (10.108.12.50): Normal (Monitored)
        Other host: Secondary - Standby Ready
                Active time: 1219 (sec)
                slot 0: ASA5525 hw/sw rev (1.0/9.4(4)2) status (Up Sys)
                  Interface inside (10.108.12.51): Normal (Monitored)


<OUTPUT TRUNCATED>


ASA1/act/pri# failover exec mate show version

Cisco Adaptive Security Appliance Software Version 9.4(4)2
Device Manager Version 7.7(1)150

Compiled on Thu 16-Feb-17 09:09 PST by builders
System image file is "disk0:/asa944-2-smp-k8.bin"
Config file at boot was "startup-config"

ASA2 up 29 mins 18 secs
failover cluster up 4 days 7 hours


<OUTPUT TRUNCATED>


I've performed a zero downtime upgrade on a Cisco ASA 5520 Active/Standby pair and only used write memory. The Standby firewall kept rebooting in a loop since it didn't find the image configured using the boot system disk0:/ and I should've specified a fallback boot image. You also need to use the write standby in order to replicate the config to the Standby firewall. I also didn't see this log message on the Active firewall.

Beginning configuration replication: Sending to mate.
End Configuration Replication to mate



ASA5520/pri/act# show failover
Failover On
Failover unit Primary
Failover LAN Interface: failover GigabitEthernet0/2 (up)
Unit Poll frequency 1 seconds, holdtime 15 seconds
Interface Poll frequency 5 seconds, holdtime 25 seconds
Interface Policy 1
Monitored Interfaces 2 of 160 maximum
Version: Ours 8.4(3), Mate 8.4(6)
Last Failover at: 12:32:40 UTC Aug 20 2015
        This host: Primary - Active
                Active time: 52035347 (sec)
                slot 0: ASA5520 hw/sw rev (2.0/8.4(3)) status (Up Sys)
                  Interface outside (1.1.1.1): Normal (Waiting)
                  Interface inside (192.168.1.1): Normal (Waiting)
                slot 1: empty
        Other host: Secondary - Cold Standby
                Active time: 0 (sec)
                slot 0: ASA5520 hw/sw rev (2.0/8.4(6)) status (Up Sys)
                  Interface outside (1.1.1.2): Unknown (Monitored)
                  Interface inside (192.168.1.2): Unknown (Monitored)
                slot 1: empty

ASA5520/pri/act# show failover state

               State          Last Failure Reason      Date/Time
This host  -   Primary
               Active         Ifc Failure              15:16:36 UTC Jun 4 2015
                              inside: No Link
Other host -   Secondary
               Cold Standby   Comm Failure             16:27:27 UTC Mar 23 2017

====Configuration State===
        Sync Done         // YOU SHOULD ALSO SEE Sync Done - STANDBY
====Communication State===
                                  // YOU SHOULD SEE Mac set


ASA5520/pri/act# write ?  

  erase     Clear flash memory configuration
  memory    Save active configuration to the flash
  net       Save the active configuration to the tftp server
  standby   Save the active configuration on the active unit to the flash on
            the standby unit

  terminal  Display the current active configuration
  <cr>


ASA5520/pri/act# write memory
Building configuration...
Cryptochecksum: aac8cc62 f0d002b1 20122d02 0499661b

36859 bytes copied in 3.290 secs (12286 bytes/sec)
[OK]

ASA5520/pri/act# write standby      // DIDN'T USE THIS ON A CISCO 5500-X
Building configuration...
[OK]
ciscoasa5520/pri/act#
Beginning configuration replication: Sending to mate.
End Configuration Replication to mate

1 comment: