Troubleshoot FortiGuard Server Connectivity

Here's a Fortinet link in troubleshooting FortiGuard server connectivity over the Internet. I was configuring a new FortiGate firewall in Multiple VDOM mode but I can't ping or perform a license update to the FortiGuard server (a cloud service over the Internet).

FGT # config vdom

 

FGT (vdom) # edit root

current vf=root:0

 

FGT (root) # execute ping update.fortiguard.net

Unable to resolve hostname.

 

The FortiGate uses FortiGuard public DNS server IP: 96.45.45.45 and 96.45.46.46 by default. To change DNS server settings, go to Network > DNS > select: Specify > type the usable public DNS server IP (Google DNS 8.8.8.8 or your private DNS server) > enable/toggle: DNS (UDP/53) > click Apply.

FGT (root) # execute ping service.fortiguard.net

PING guard.fortinet.net (208.184.237.61): 56 data bytes

64 bytes from 208.184.237.61: icmp_seq=0 ttl=47 time=255.0 ms

64 bytes from 208.184.237.61: icmp_seq=1 ttl=47 time=254.7 ms

64 bytes from 208.184.237.61: icmp_seq=2 ttl=47 time=254.7 ms

64 bytes from 208.184.237.61: icmp_seq=3 ttl=47 time=254.7 ms

64 bytes from 208.184.237.61: icmp_seq=4 ttl=47 time=254.7 ms

 

--- guard.fortinet.net ping statistics ---

5 packets transmitted, 5 packets received, 0% packet loss

round-trip min/avg/max = 254.7/254.7/255.0 ms

 

FGT (root) # execute ping update.fortiguard.net

PING fds1.fortinet.com (12.34.97.16): 56 data bytes

64 bytes from 12.34.97.16: icmp_seq=0 ttl=46 time=332.9 ms

64 bytes from 12.34.97.16: icmp_seq=1 ttl=46 time=333.5 ms

64 bytes from 12.34.97.16: icmp_seq=2 ttl=46 time=333.4 ms

64 bytes from 12.34.97.16: icmp_seq=3 ttl=46 time=333.5 ms

64 bytes from 12.34.97.16: icmp_seq=4 ttl=46 time=337.0 ms

 

--- fds1.fortinet.com ping statistics ---

5 packets transmitted, 5 packets received, 0% packet loss

round-trip min/avg/max = 332.9/334.0/337.0 ms

 

FGT (root) # execute ping guard.fortinet.net

PING guard.fortinet.net (208.184.237.61): 56 data bytes

64 bytes from 208.184.237.61: icmp_seq=0 ttl=47 time=254.9 ms

64 bytes from 208.184.237.61: icmp_seq=1 ttl=47 time=254.6 ms

64 bytes from 208.184.237.61: icmp_seq=2 ttl=47 time=254.5 ms

64 bytes from 208.184.237.61: icmp_seq=3 ttl=47 time=254.5 ms

64 bytes from 208.184.237.61: icmp_seq=4 ttl=47 time=254.5 ms

 

--- guard.fortinet.net ping statistics ---

5 packets transmitted, 5 packets received, 0% packet loss

round-trip min/avg/max = 254.5/254.6/254.9 ms

 

 

The FortiCare (support and warranty), Next Generation Firewall licenses (Intrusion Prevention, Antivirus and Web filtering) were updated/enabled after a few minutes.

 

The Internet Service Database (ISDB) objects were updated as well. You can view these under Policy & Objects > Internet Service Database.

The ISDB is a comprehensive list of public IP addresses (Geolocation based), service/port numbers, reputation, popularity (Facebook, Amazon, Microsoft, etc.) which can be used in creating a firewall policy or security profile (Antivirus, Web filter, Application Control, etc.) in a FortiGate firewall.

Create a Fortinet Support Ticket

Here's a Fortinet link for device hardening and best practice in a FortiGate firewall.

To create a new Fortinet support ticket, go to this link > select Create a Ticket.

Select a Request Ticket Type (closest to your issue/inquiry). In this case, I selected Customer Service > Submit ticket.

Select a CS category. In this case I selected: Cloud Portal Query.

Put the device Serial Number > Contact Information > Ticket Information.

Add Comment to describe your issue or upload a screenshot of the error in the Attachments.

Click Finish and note the ticket number. The Fortinet ticket number and summary will be sent to your registered email.

Another way to create a Fortinet ticket is via the Asset Management portal. Click Support > FortiCare > Create a Ticket.

Click New Ticket.

Choose: Technical Support Ticket > Submit Ticket.

You can get the FortiGate serial number with the get system status CLI command:

FG# get system status

Version: FortiGate-xx v7x,buildxx

Security Level: 2

Firmware Signature: certified

Virus-DB: 1.00000(2018-04-09 18:07)

Extended DB: 1.00000(2018-04-09 18:07)

Extreme DB: 1.00000(2018-04-09 18:07)

AV AI/ML Model: 0.00000(2001-01-01 00:00)

IPS-DB: 6.00741(2015-12-01 02:30)

IPS-ETDB: 6.00741(2015-12-01 02:30)

APP-DB: 6.00741(2015-12-01 02:30)

INDUSTRIAL-DB: 6.00741(2015-12-01 02:30)

IPS Malicious URL Database: 1.00001(2015-01-01 01:01)

IoT-Detect: 0.00000(2022-08-17 17:31)

Serial-Number: FGxx

BIOS version: 06000008

System Part-Number: Pxx

Log hard disk: Available

Hostname: FG

Private Encryption: Disable

Operation Mode: NAT

Current virtual domain: root

Max number of virtual domains: 10

Virtual domains status: 2 in NAT mode, 0 in TP mode

Virtual domain configuration: multiple

FIPS-CC mode: disable

Current HA mode: a-p, primary

Cluster uptime: 241 days, 23 hours, 33 minutes, 20 seconds

Cluster state change time: 2024-03-06 07:04:47

Branch point: xx

Release Version Information: GA

FortiOS x86-64: Yes

System time: Fri Sep  6 03:20:07 2024

Last reboot reason: warm reboot

 

Or retrieve it via the web GUI under Dashboard > Status.

Under the Product Info > type the device SN > click Go

Fill up the required info > click Next.

Type the Comment (answer the pre-filled questionnaire) or click File Upload to upload a screenshot of the error.

It's also very useful to upload the Debug log which is similar to show tech-support in a Cisco device. Go to System > Settings > Debug logs > click Download.

It only took a few seconds to download the Debug log text file. Here's a snippet of the Debug log output:

----------------------------------------------------------------

  Serial Number: FG4Hxx   Diagnose output

----------------------------------------------------------------

 

### get system status

 

 

Version: FortiGate-xxv7x

Security Level: 2

Firmware Signature: certified

Virus-DB: 1.00000(2018-04-09 18:07)

Extended DB: 1.00000(2018-04-09 18:07)

Extreme DB: 1.00000(2018-04-09 18:07)

AV AI/ML Model: 0.00000(2001-01-01 00:00)

IPS-DB: 6.00741(2015-12-01 02:30)

IPS-ETDB: 6.00741(2015-12-01 02:30)

APP-DB: 6.00741(2015-12-01 02:30)

INDUSTRIAL-DB: 6.00741(2015-12-01 02:30)

IPS Malicious URL Database: 1.00001(2015-01-01 01:01)

IoT-Detect: 0.00000(2022-08-17 17:31)

Serial-Number: FG4xx

BIOS version: 06000008

System Part-Number: P27xx

Log hard disk: Available

Hostname: xx

Private Encryption: Disable

Operation Mode: NAT

Current virtual domain: root

Max number of virtual domains: 10

Virtual domains status: 2 in NAT mode, 0 in TP mode

Virtual domain configuration: multiple

FIPS-CC mode: disable

Current HA mode: a-p, primary

Cluster uptime: 241 days, 23 hours, 40 minutes, 25 seconds

Cluster state change time: 2024-03-06 07:04:47

Branch point: xx

Release Version Information: xx

FortiOS x86-64: Yes

System time: Fri Sep  6 03:27:12 2024

Last reboot reason: warm reboot

 

### get system performance status

 

 

CPU states: 1% user 0% system 0% nice 99% idle 0% iowait 0% irq 0% softirq

CPU0 states: 2% user 0% system 0% nice 98% idle 0% iowait 0% irq 0% softirq

CPU1 states: 0% user 0% system 0% nice 99% idle 0% iowait 0% irq 1% softirq

CPU2 states: 7% user 5% system 0% nice 87% idle 0% iowait 0% irq 1% softirq

CPU3 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq

CPU4 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq

CPU5 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq

CPU6 states: 0% user 0% system 0% nice 99% idle 0% iowait 0% irq 1% softirq

CPU7 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq

CPU8 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq

 

<OUTPUT TRUNCATED>

 

 

Review the ticket summary before submitting > click Confirm to proceed.

 

Cisco GRE Tunnel Keepalive

This Cisco link covers the GRE Tunnel and how a keepalive works. I got a GRE over IPSec VPN configured between Singapore and London. The GRE tunnel only goes up whenever I perform a ping. So I configured the GRE tunnel keepalive so it always stays up. The default keepalive interval is 10 seconds and 3 retries.

SIN#show run interface Tunnel40
Building configuration...

Current configuration : 314 bytes
!
interface Tunnel40
 ip address 10.16.2.194 255.255.255.252
 ip mtu 1400
 tunnel source 192.168.1.18
 tunnel destination 192.168.1.146
end

SIN#configure terminal
Enter configuration commands, one per line.  End with CNTL/Z.
SIN(config)#interface Tunnel40
SIN(config-if)#keepalive ?
  <0-32767>  Keepalive period (default 10 seconds)
  <cr>       <cr>

SIN(config-if)#keepalive
SIN(config-if)#end
SIN#write memory
Building configuration...
[OK]

SIN#show run interface Tunnel40
Building configuration...

Current configuration : 330 bytes
!
interface Tunnel40
 ip address 10.106.192.194 255.255.255.252
 ip mtu 1400
 keepalive 10 3
 tunnel source 192.168.1.18
 tunnel destination 192.168.1.146
end

LON#show run interface Tunnel40
Building configuration...

Current configuration : 322 bytes
!
interface Tunnel40
 ip address 10.16.2.193 255.255.255.252
 ip mtu 1400
 tunnel source 192.168.1.146
 tunnel destination 192.168.1.18
end

LON#configure terminal
Enter configuration commands, one per line.  End with CNTL/Z.
LON(config)#interface Tunnel40
LON(config-if)#keepalive
LON(config-if)#end
LON#write memory
Building configuration...
[OK]

LON#show run interface Tunnel40
Building configuration...

Current configuration : 338 bytes
!
interface Tunnel40
 ip address 10.16.2.193 255.255.255.252
 ip mtu 1400
 keepalive 10 3   // DEFAULT IS 10 SECOND INTERVAL AND 3 RETRIES
 tunnel source 192.168.1.146
 tunnel destination 192.168.1.18
end

LON#ping 10.16.2.194
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.16.2.194, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 181/181/181 ms

 

I've checked the IPSec VPN was already up in the FortiGate firewall even before I did a ping.

Cisco ASA Firewall SNMP OID

There's a "hidden" Cisco ASA CLI command in order to retrieve the SNMP MIB OID info without performing an SNMP walk from a remote server/NMS. The Management Information Base (MIB) is the hierarchical (tree) structure of the SNMP Object Identifier (OID). OID is the long code string or numbers separated by dots. It uniquely identifies an SNMP managed object such as a device interface, CPU, memory, bandwidth/traffic stats, etc.

I had a high CPU alarm in our NMS but there was no high CPU when checked. It was later found out the NMS didn't support the new Firepower ASA platform using multiple Core CPU, so it needed to update its MIB OID database.

FPR2100# show cpu core all

Core              5 sec  1 min  5 min

Core 0             1.4%   0.8%   0.7%

Core 1             0.2%   0.2%   0.2%

Core 2             0.2%   0.2%   0.2%

Core 3             0.2%   0.2%   0.2%

Core 4             0.2%   0.2%   0.2%

Core 5             0.2%   0.2%   0.2%

Core 6             0.2%   0.2%   0.2%

Core 7             0.2%   0.2%   0.2%

Core 8             0.2%   0.2%   0.2%

Core 9             0.2%   0.2%   0.2%

Core 10            0.2%   0.2%   0.2%

Core 11            1.0%   0.6%   0.5%

Core 12            0.2%   0.2%   0.2%

Core 13            0.2%   0.2%   0.2%

Core 14            0.2%   0.2%   0.2%

Core 15            0.2%   0.2%   0.2%

Core 16            0.2%   0.2%   0.2%

Core 17            0.2%   0.2%   0.2%

Core 18            0.2%   0.2%   0.2%

Core 19            0.2%   0.2%   0.2%

Core 20            0.2%   0.2%   0.2%

Core 21            0.2%   0.2%   0.2%

The output below came a Cisco ASA5515-X firewall. You'll need to run this command in the admin context if the ASA is in Multiple Context mode.

ciscoasa# show snmp-server ?      

  engineID    Show snmp engineID
  group       Show snmp groups
  host        Show snmp host's
  statistics  Show snmp-server statistics
  user        Show snmp users 

ciscoasa# show snmp-server oidlist ?   // IT'S A HIDDEN CLI COMMAND
ERROR: % Unrecognized command

ciscoasa# show snmp-server oidlist

-------------------------------------------------
[0]     1.3.6.1.2.1.1.1.        sysDescr
[1]     1.3.6.1.2.1.1.2.        sysObjectID
[2]     1.3.6.1.2.1.1.3.        sysUpTime
[3]     1.3.6.1.2.1.1.4.        sysContact
[4]     1.3.6.1.2.1.1.5.        sysName
[5]     1.3.6.1.2.1.1.6.        sysLocation
[6]     1.3.6.1.2.1.1.7.        sysServices
[7]     1.3.6.1.2.1.1.8.        sysORLastChange
[8]     1.3.6.1.2.1.1.9.1.2.    sysORID
[9]     1.3.6.1.2.1.1.9.1.3.    sysORDescr
[10]    1.3.6.1.2.1.1.9.1.4.    sysORUpTime
[11]    1.3.6.1.2.1.2.1.        ifNumber
[12]    1.3.6.1.2.1.2.2.1.1.    ifIndex
[13]    1.3.6.1.2.1.2.2.1.2.    ifDescr
[14]    1.3.6.1.2.1.2.2.1.3.    ifType
[15]    1.3.6.1.2.1.2.2.1.4.    ifMtu
[16]    1.3.6.1.2.1.2.2.1.5.    ifSpeed
[17]    1.3.6.1.2.1.2.2.1.6.    ifPhysAddress
[18]    1.3.6.1.2.1.2.2.1.7.    ifAdminStatus
[19]    1.3.6.1.2.1.2.2.1.8.    ifOperStatus
[20]    1.3.6.1.2.1.2.2.1.9.    ifLastChange
[21]    1.3.6.1.2.1.2.2.1.10.   ifInOctets
[22]    1.3.6.1.2.1.2.2.1.11.   ifInUcastPkts
<--- More --->

<OUTPUT TRUNCATED>

[1002]  1.3.6.1.6.3.15.1.2.2.1.3.       usmUserSecurityName
[1003]  1.3.6.1.6.3.15.1.2.2.1.4.       usmUserCloneFrom
[1004]  1.3.6.1.6.3.15.1.2.2.1.5.       usmUserAuthProtocol
[1005]  1.3.6.1.6.3.15.1.2.2.1.6.       usmUserAuthKeyChange
[1006]  1.3.6.1.6.3.15.1.2.2.1.7.       usmUserOwnAuthKeyChange
[1007]  1.3.6.1.6.3.15.1.2.2.1.8.       usmUserPrivProtocol
[1008]  1.3.6.1.6.3.15.1.2.2.1.9.       usmUserPrivKeyChange
[1009]  1.3.6.1.6.3.15.1.2.2.1.10.      usmUserOwnPrivKeyChange
[1010]  1.3.6.1.6.3.15.1.2.2.1.11.      usmUserPublic
[1011]  1.3.6.1.6.3.15.1.2.2.1.12.      usmUserStorageType
[1012]  1.3.6.1.6.3.15.1.2.2.1.13.      usmUserStatus
[1013]  1.3.6.1.6.3.16.1.2.1.3. vacmGroupName
[1014]  1.3.6.1.6.3.16.1.2.1.4. vacmSecurityToGroupStorageType
[1015]  1.3.6.1.6.3.16.1.2.1.5. vacmSecurityToGroupStatus
-------------------------------------------------

 

Cisco Firepower FX-OS show tech-support

We've encountered an outage in our Cisco FPR 2100 High Availability (HA) pair running ASA OS. The Primary ASA firewall crashed or auto reload and didn't failover properly to the Secondary ASA firewall. I also had to reload the Secondary ASA firewall in order for HA to synchronize.

Aside from the usual show tech-support in the ASA command, Cisco TAC will ask for the show tech-support fprm detail output (GZ archive file), which can be generated from the FX-OS CLI. This saves time in troubleshooting and would allow TAC to further investigate using their internal database.

Based on the FX-OS show tech, the auto reload was due to a memory bug CSCwk27830. TAC recommended to perform an ASA OS upgrade using the known fixed release.

Threadname: **lina**

| Rip: ****

| Version: **9.xx**

| Hardware: **FPR-21xx**

| 0x00000000019862b8 : ikev2_copy_ike_policy+216 at ikev2/granite/ikev2/core/policy/ikev2_policy.c:1677

| 0x00000000019c1144 : ikev2_initiate_sa+476 at ikev2/granite/ikev2/core/ikev2_sa_management.c:132

| 0x00000000018e300c : asa_connect_continue+136 at ikev2/ikev2_asa_connect.c:663

| 0x000000000193f214 : asa_spi_mgt_callback+1060 at ikev2/ikev2_spi_mgt.c:666

| 0x000000000193dcc0 : ikev2_pitcher+328 at ikev2/ikev2_pitcher.c:880

| 0x000000000193a768 : IKEv2ProcessMsg+140 at ikev2/ikev2_daemon.c:548

| 0x000000000193c9c4 : Ikev2Daemon+1452 at ikev2/ikev2_daemon.c:343

 

ciscoasa/pri/act/admin# connect fxos admin
Configuring session.
.
Connecting to FXOS.
...
Connected to FXOS. Escape character sequence is 'CTRL-^X'.

NOTICE: You have connected to the FXOS CLI with admin privileges.
Config commands and commit-buffer are not supported in appliance mode.


Certain components of this software are licensed under the "GNU General Public
License, version 3" provided with ABSOLUTELY NO WARRANTY under the terms of
"GNU General Public License, Version 3", available here:
http://www.gnu.org/licenses/gpl.html. See User Manual (''Licensing'') for
details.

<OUTPUT TRUNCATED>


firepower-2100# connect
  asa         Connect to ASA Application CLI
  local-mgmt  Connect to Local Management CLI


firepower-2120# connect local-mgmt
Warning: network service is not available when entering 'connect local-mgmt'
firepower-2120(local-mgmt)# show  
  active-connections     Show active TCP/IP connections
  cli                    CLI Information
  clock                  Clock
  consent-token          consent token
  debug                  Debugging functions
  env                    Show environmental monitoring data
  failsafe-params        Show the failsafe mode configuration
  file                   File Commands
  fxos-mode              Fxos-mode
  lacp                   LACP command
  mgmt-ip-debug          IP Debug Info
  npu-accel              Show NPU accelerator data
  ntp                    NTP Status
  open-network-ports     Show open network ports
  pktmgr                 pktmgr command
  platform-sw-processes  Show the state of platform software processes
  pmon                   Pmon
  portchannel            portchannel command
  portmanager            portmanager command
  processes              Processes
  running-config         Running-config
  software               Software
  sshkey                 Sshkey
  tech-support           Tech Support
  version                System version


firepower-2100(local-mgmt)# show tech-support
  fprm  FPRM


firepower-2100(local-mgmt)# show tech-support fprm
  <CR>      
  >       Redirect it to a file
  >>      Redirect it to a file in append mode
  brief   Brief
  detail  Detail
  |       Pipe command output to filter


firepower-2120(local-mgmt)# show tech-support fprm detail

The show tech output is savedin the ASA flash (disk0:) and can be simply transferred to your PC via ASDM and then upload it to the Cisco Support Case portal.

ciscoasa/pri/act/admin# changeto system
ciscoasa/pri/act# show flash
--#--  --length--  -----date/time------  path
44053  98          Apr 06 2023 07:50:39  log
134673345  4096        Jun 26 2023 05:19:00  log/from_tmp
134673346  145713      Jul 31 2024 17:26:02  log/from_tmp/asa-appagent.log
134673347  0           Jul 22 2024 16:46:23  log/from_tmp/asa-fxos_xml.log

<OUTPUT TRUNCATED>

134217933  17421854    Jul 30 2024 05:41:58  fxos/20240730054152_firepower-2100_FPRM.tar.gz

<OUTPUT TRUNCATED>

21475885056 bytes total (20623392768 bytes free)

Changing a Cisco Switchport Mode From Access to Trunk

I had to reconfigure a Cisco switchport mode from access to a trunk in order to run multiple VLANs in a Cisco ASA firewall interface. I configured a new sub-interface on the ASA using VLAN 10.

ciscoasa# show interface ip brief
Interface                  IP-Address      OK? Method Status                Protocol
GigabitEthernet0/0         46.4.4.66    YES CONFIG up                    up  
GigabitEthernet0/1         172.30.3.4    YES CONFIG up                    up  
GigabitEthernet0/1.10     172.20.1.7     YES manual up                    up  
GigabitEthernet0/2         unassigned      YES unset  administratively down down
GigabitEthernet0/3         unassigned      YES unset  administratively down down
GigabitEthernet0/4         unassigned      YES unset  administratively down down
GigabitEthernet0/5         unassigned      YES unset  administratively down down
GigabitEthernet0/6         unassigned      YES unset  administratively down down
GigabitEthernet0/7         172.30.3.254  YES unset  up                    up  
Internal-Control0/0        127.0.1.1       YES unset  up                    up  
Internal-Data0/0           unassigned      YES unset  up                    up  
Internal-Data0/1           unassigned      YES unset  down                  down
Internal-Data0/2           unassigned      YES unset  up                    up  
Internal-Data0/3           169.254.1.1     YES unset  up                    up  
Management0/0              10.10.6.9   YES CONFIG up                    up  

 

ciscoasa# show run interface GigabitEthernet0/1
!
interface GigabitEthernet0/1    <<< VLAN 30 ON SWITCH
 description | SW G1/0/3 : INSIDE |
 nameif inside
 security-level 100
 ip address 172.30.3.4 255.255.255.0

 

ciscoasa# show run interface GigabitEthernet0/1.10
!
interface GigabitEthernet0/1.10
 description | DMZ |
 vlan 10
 nameif dmz
 security-level 100
 ip address 172.20.1.7 255.255.255.248

Since it's a remote site with an IPSec VPN over the Internet via the native "inside" interface, I had to use the reload command to avoid being locked out. I reconfigured the switch port from an access port (single VLAN 30) to a trunk. I also used the switchport trunk native vlan command in order for the original "inside" interface to work (untagged). Once everything resumed and working, I canceled the reload command.

Switch#reload in 10
Reload scheduled for 10:24:35 UTC Thu Feb 8 2024 (in 10 minutes) by john on vty0 (172.30.3.1)Reload command is being issued on Active unit, this will reload the whole stack
Proceed with reload? [confirm]
Switch#configure terminal
Enter configuration commands, one per line.  End with CNTL/Z.
Switch(config)#interface GigabitEthernet1/0/3
Switch(config-if)#switchport trunk native vlan 30
Switch(config-if)#switchport mode trunk

There was about a 5 second outage, then my SSH session resumed.

Switch(config-if)#no switchport access vlan 30
Switch(config-if)#no spanning-tree portfast
Switch(config-if)#end

Switch#show run interface g1/0/3
Building configuration...

Current configuration : 140 bytes
!
interface GigabitEthernet1/0/3
 description | FW G0/1 : INSIDE |
 switchport trunk native vlan 30
 switchport mode trunk
end

Switch#reload cancel


***
*** --- SHUTDOWN ABORTED ---
***
 

The proper design in the ASA should a sub-interface and VLAN configured away from the G0/1 main interface. Then the switchport is plainly configured as a trunk (no native vlan).

interface GigabitEthernet0/1
 no nameif
 no security-level
 no ip address

interface GigabitEthernet0/1.30
 vlan 30
 nameif inside
 security-level 100
 ip address 172.30.3.4 255.255.255.0 

 

Troubleshoot Cisco IPSec Anti-Replay Error

Here's a Cisco link for troubleshooting an IPSec anti-replay errors. 

I was troubleshooting and observed a lot %CRYPTO-4-PKT_REPLAY_ERR: decrypt: replay check failed logs in a Cisco router configured for IPSec VPN:

*Sep 18 00:16:48 UTC: %CRYPTO-4-PKT_REPLAY_ERR: decrypt: replay check failed

        connection id=349, sequence number=4838

 

*Sep 18 00:26:35 UTC: %CRYPTO-4-PKT_REPLAY_ERR: decrypt: replay check failed

        connection id=349, sequence number=499055

 

*Sep 18 00:27:50 UTC: %CRYPTO-4-PKT_REPLAY_ERR: decrypt: replay check failed

        connection id=349, sequence number=591422

 

*Sep 18 00:30:58 UTC: %CRYPTO-4-PKT_REPLAY_ERR: decrypt: replay check failed

        connection id=349, sequence number=884929

 

*Sep 18 00:31:58 UTC: %CRYPTO-4-PKT_REPLAY_ERR: decrypt: replay check failed

        connection id=349, sequence number=969290

 

You can verify if the IPSec Security Association (SA) is from a legitimate peer using the show crypto ipsec sa | in peer|conn id command

2900#show crypto ipsec sa | in peer|conn id

   current_peer 110.17.9.8 port 500

   current_peer 175.23.30.6 port 19302

        conn id: 2349, flow_id: Onboard VPN:349, sibling_flags 80000040, crypto map: VPN-PROFILE-head-1

        conn id: 2350, flow_id: Onboard VPN:350, sibling_flags 80000040, crypto map: VPN-PROFILE-head-1

 

 

You can adjust the replay window size once the SA peer is confirmed and it's not doing a man-in-the-middle (MITM) attack. The default window size is 64 packets.

2900#show run all | inc crypto ipsec

crypto ipsec optional retry 300

crypto ipsec security-association lifetime kilobytes 4608000

crypto ipsec security-association lifetime seconds 3600

no crypto ipsec security-association replay disable

crypto ipsec security-association replay window-size 64

crypto ipsec security-association dummy

crypto ipsec transform-set default esp-aes esp-sha-hmac

crypto ipsec transform-set VPN-TSET esp-aes 256 esp-sha-hmac

crypto ipsec nat-transparency udp-encapsulation

crypto ipsec profile VPN-PROFILE

crypto ipsec profile default

Use the crypto ipsec security-association replay window-size <WINDOW SIZE> command. Start with 512 first, clear and observe the logs again.

2900#configure terminal

Enter configuration commands, one per line.  End with CNTL/Z.

2900(config)#crypto ipsec security-association replay window-size ? 

  1024  Window size of 1024

  128   Window size of 128

  256   Window size of 256

  512   Window size of 512

  64    Window size of 64 (default)

 

2900(config)#crypto ipsec security-association replay window-size 512

2900(config)#end

2900#write memory

Building configuration...

[OK]

 

2900#clear log
Clear logging buffer [confirm] 

 

 

I didn't observe the said logs afterwards (for almost more than 30 mins).

 

*Sep 18 01:08:37 UTC: %CLEAR-5-COUNTERS: Clear counter on all interfaces by admin on vty0 (202.7.6.8)
2900#
2900#show clock
*01:46:12.854 UTC Mon Sep 18 2023 

Troubleshoot Cisco ASA MM_WAIT_MSG2 Stuck Issue

I had to troubleshoot a site-to-site IPSec VPN in a Cisco ASA firewall in Multiple Context mode. The IKE Phase 1 State was stuck in MM_WAIT_MSG2, therefore IKE Phase 1 can't be established. The VPN tunnel was previously working and there were no changes in the ASA firewall. The IKE Phase 1 policy are the same on both ends and debug showed both the Security Association (SA) peer are reachable (ISAKMP/UDP 500 is open).

ciscoasa/VPN# show crypto isakmp sa

IKEv1 SAs:

   Active SA: 1
    Rekey SA: 0 (A tunnel will report 1 Active and 1 Rekey SA during rekey)
Total IKE SA: 1

1   IKE Peer: 202.8.6.4
    Type    : user            Role    : initiator
    Rekey   : no              State   : MM_WAIT_MSG2

 

Using the show crypto isakmp sa detail command revealed something strange which was the Encrypt: aes-256 which is not configured in this context (it's configured on the remote Cisco ASA though) and Lifetime: 0 was displayed (no timer count down).

ciscoasa/VPN# show crypto isakmp sa detail

IKEv1 SAs:

   Active SA: 1
    Rekey SA: 0 (A tunnel will report 1 Active and 1 Rekey SA during rekey)
Total IKE SA: 1

1   IKE Peer: 202.8.6.4
    Type    : user            Role    : initiator
    Rekey   : no              State   : MM_WAIT_MSG2
    Encrypt : aes-256         Hash    : SHA       
    Auth    : preshared       Lifetime: 0

There are no IKEv2 SAs

 

The local Cisco ASA had only a single ISAKMP/IKE Phase 1 policy of AES 128 for encryption. I  searched for a Cisco bug but found none.

ciscoasa/VPN# show run crypto

crypto ikev1 policy 10
 authentication pre-share
 encryption aes
 hash sha
 group 2
 lifetime 28800

 

I tried to re-configure the GRE tunnel, crypto map, add IKE Phase 1 policy using AES-256 for encryption, re-configured the tunnel-group, changed IP address for interesting traffic/ACL but I still got the same Encrypt: aes-256 and Lifetime:0 output.

I was able to fix the issue by re-applying the VPN license for the VPN context resource class/limit under the "system" context.

ciscoasa/VPN# changeto system
ciscoasa#
ciscoasa# show run class
class default
  limit-resource All 0
  limit-resource Mac-addresses 16384
  limit-resource ASDM 5
  limit-resource SSH 5
  limit-resource Telnet 5
!

class VPN
  limit-resource VPN Other 10

ciscoasa# conf t
ciscoasa(config)# context VPN
ciscoasaconfig-ctx)# no member VPN
ciscoasa(config-ctx)# member VPN
ciscoasa(config-ctx)# write memory
Building configuration...
Cryptochecksum: 0474f6fe d84a31ed a692dca2 ab52fd98

7473 bytes copied in 0.960 secs
[OK]

After the VPN resource class was re-applied, the IKE  Phase 1 SA established the correct encryption (aes-128) and "Lifetime Remaining" counter had restarted.

ciscoasa# changeto context VPN
ciscoasa/VPN# show crypto isakmp sa detail

IKEv1 SAs:

   Active SA: 1
    Rekey SA: 0 (A tunnel will report 1 Active and 1 Rekey SA during rekey)
Total IKE SA: 1

1   IKE Peer: 202.8.6.4
    Type    : L2L             Role    : initiator
    Rekey   : no              State   : MM_ACTIVE
    Encrypt : aes             Hash    : SHA       
    Auth    : preshared       Lifetime: 43200
    Lifetime Remaining: 43197

There are no IKEv2 SAs

Cisco ASA Firewall SSH Key Exchange

I had an issue wherein I couldn't SSH to a Cisco ASA firewall and received a key exchange error. I've checked the configured SSH key exchange group and it was configured to use a higher DH group which is the default in newer Cisco ASA version.

 

[john@server01 ~]$ ssh admin@192.168.202.4

Unable to negotiate with 192.168.202.4 port 22: no matching key exchange method found. Their offer: diffie-hellman-group14-sha256

 

 

ciscoasa# show run ssh

ssh stricthostkeycheck

ssh timeout 30

ssh version 2

ssh key-exchange group dh-group14-sha256

ssh 10.10.0.0 255.255.0.0 management

 

 

I configured a lower DH group as an interim solution. The management server needs to be upgraded in order to support newer SSH key exchange protocols as a permanent fix.

 

ciscoasa# configure terminal

ciscoasa(config)# ssh key-exchange group ?

 

configure mode commands/options:

  curve25519-sha256   Diffie-Hellman group-31-sha256

  dh-group1-sha1      Diffie-Hellman group 2 (DEPRECATED)

  dh-group14-sha1     Diffie-Hellman group-14-sha1

  dh-group14-sha256   Diffie-Hellman group-14-sha256

  ecdh-sha2-nistp256  Diffie-Hellman group-19-sha256

ciscoasa(config)# ssh key-exchange group dh-group1-sha1

WARNING: DH group 2 is considered insecure. This option is deprecated and will be removed in a later version.

 

 

I was able to SSH and save the RSA key afterwards.

[john@server01 ~]$ ssh admin@192.168.202.4
The authenticity of host '172.16.22.4 (172.16.22.4)' can't be established.
RSA key fingerprint is 98:c0:6b:42:88:7f:48:68:ae:a1:b4:04:03:12:34:56.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.202.4' (RSA) to the list of known hosts.admin@192.168.202.4's password:<SSH PASSWORD>

ciscoasa# show version

 

Cisco Adaptive Security Appliance Software Version 9.16.x

SSP Operating System Version 2.10.x

Device Manager Version 7.18.x

 

Compiled on Wed 03-Aug-22 05:26 GMT by builders

 

ciscoasa up 23 hours 51 mins

failover cluster up 23 hours 51 mins

 

Cisco Secure Firewall 3100 ASA Smart License

The license feature Encryption-3DES-AES was disabled by default and I needed to add the Standard Smart license (Essential license) for the Cisco Secure Firewall FPR 3110. The Cisco Smart Software Manager (CSSM) has allowed the Export-Controlled for its registration token by default. Refer to this link.

Essentials license: -L-FPR3110-BSE=. The Essentials license is a required license.

Strong Encryption (3DES/AES) license: -L-FPR3K-ENC-K9=. Only required if your account is not authorized for strong encryption. 

ciscoasa# show version

Cisco Adaptive Security Appliance Software Version 9.18(3)56
SSP Operating System Version 2.12(0.519)
Device Manager Version 7.20(1)

Compiled on Tue 12-Sep-23 19:15 GMT by builders
System image file is "disk0:/installables/switch/fxos-k8-fp3k-lfbff.2.12.0.519.SPA"
Config file at boot was "startup-config"

ciscoasa up 1 min 57 secs
Start-up time 8 secs

Hardware:   FPR-3110, 52168 MB RAM, CPU Ryzen Zen 2 2900 MHz, 1 CPU (24 cores)

Encryption hardware device : Cisco ASA Crypto on-board accelerator (revision 0x1)
                             AE microcode        : CNN5x-MC-AE-MAIN-0007
                             SE SSL microcode    : CNN5x-MC-SE-SSL-0018
                             Number of accelerators: 1

 1: Int: Internal-Data0/1    : address is 0000.0041.0004, irq 152
 3: Int: Not licensed        : irq 0
 4: Ext: Management1/1       : address is c47e.e07e.1482, irq 0
 5: Int: Internal-Data1/1    : address is 0000.0100.0001, irq 0

License mode: Smart Licensing
              
Licensed features for this platform:
Maximum Physical Interfaces       : Unlimited      
Maximum VLANs                     : 1024           
Inside Hosts                      : Unlimited      
Failover                          : Active/Active  
Encryption-DES                    : Enabled        
Encryption-3DES-AES               : Disabled       
Security Contexts                 : 2              
Carrier                           : Disabled       
AnyConnect Premium Peers          : 3000           
AnyConnect Essentials             : Disabled       
Other VPN Peers                   : 3000           
Total VPN Peers                   : 3000           
AnyConnect for Mobile             : Enabled        
AnyConnect for Cisco VPN Phone    : Enabled        
Advanced Endpoint Assessment      : Enabled        
Shared License                    : Disabled       
Total TLS Proxy Sessions          : 4000           
Cluster                           : Enabled        

Serial Number: FJZ27231234
Configuration register is 0x1
Configuration has not been modified since last system restart.

ciscoasa# show license summ

Smart Licensing is ENABLED

Registration:
  Status: UNREGISTERED
  Export-Controlled Functionality: NOT ALLOWED

License Authorization:
  Status: EVAL MODE
  Evaluation Period Remaining: 82 days, 6 hours, 37 minutes, 42 seconds

License Usage:
  License                 Entitlement tag               Count Status
  -----------------------------------------------------------------------------
                          (FPR_3110_BASE_STD)               1 EVAL MODE

I configured Smart Call Home (SCH) using the management interface and used http method only.

ciscoasa(config)# dns domain-lookup management
ciscoasa(config)# call-home
ciscoasa(cfg-call-home)#  no profile CiscoTAC-1
INFO: default profile is reset to default configuration.
ciscoasa(cfg-call-home)# profile MY-LICENSE
ciscoasa(cfg-call-home-profile)#   active
ciscoasa(cfg-call-home-profile)#   destination address http http:/<CSSM IP>/Transportgateway/services/DeviceRequestHandler
ciscoasa(cfg-call-home-profile)# destination transport-method http
ciscoasa(cfg-call-home-profile)# license smart
INFO: License(s) corresponding to an entitlement will be activated only after an entitlement request has been authorized.
ciscoasa(config-smart-lic)#  feature tier standard

ciscoasa# license smart register idtoken <CSSM REGISTRATION TOKEN>

ciscoasa# show license summary                        

Smart Licensing is ENABLED

Registration:
  Status: REGISTERED
  Smart Account: MY-ACCOUNT
  Virtual Account: Default
  Export-Controlled Functionality: ALLOWED
  Last Renewal Attempt: None
  Next Renewal Attempt: Apr 29 2024 06:35:22 UTC

License Authorization:
  Status: AUTHORIZED
  Last Communication Attempt: SUCCEEDED
  Next Communication Attempt: Dec 01 2023 06:35:34 UTC

License Usage:
  License                 Entitlement tag               Count Status
  -----------------------------------------------------------------------------
  FPR3110 Base License    (FPR_3110_BASE_STD)               1 AUTHORIZED

ciscoasa# show version

Cisco Adaptive Security Appliance Software Version 9.18(3)56
SSP Operating System Version 2.12(0.519)
Device Manager Version 7.20(1)
Compiled on Tue 12-Sep-23 19:15 GMT by builders
System image file is "disk0:/installables/switch/fxos-k8-fp3k-lfbff.2.12.0.519.SPA"
Config file at boot was "startup-config"

ciscoasa up 6 mins 4 secs
Start-up time 8 secs

Hardware:   FPR-3110, 52168 MB RAM, CPU Ryzen Zen 2 2900 MHz, 1 CPU (24 cores)

Encryption hardware device : Cisco ASA Crypto on-board accelerator (revision 0x1)
                             AE microcode        : CNN5x-MC-AE-MAIN-0007
                             SE SSL microcode    : CNN5x-MC-SE-SSL-0018
                             Number of accelerators: 1

 1: Int: Internal-Data0/1    : address is 0000.0041.0004, irq 43
 3: Int: Not licensed        : irq 0
 4: Ext: Management1/1       : address is c47e.e07e.1482, irq 0
 5: Int: Internal-Data1/1    : address is 0000.0100.0001, irq 0

License mode: Smart Licensing
              
Licensed features for this platform:
Maximum Physical Interfaces       : Unlimited      
Maximum VLANs                     : 1024           
Inside Hosts                      : Unlimited      
Failover                          : Active/Active  
Encryption-DES                    : Enabled        
Encryption-3DES-AES               : Enabled        
Security Contexts                 : 2              
Carrier                           : Disabled       
AnyConnect Premium Peers          : 3000           
AnyConnect Essentials             : Disabled       
Other VPN Peers                   : 3000           
Total VPN Peers                   : 3000           
AnyConnect for Mobile             : Enabled        
AnyConnect for Cisco VPN Phone    : Enabled        
Advanced Endpoint Assessment      : Enabled        
Shared License                    : Disabled       
Total TLS Proxy Sessions          : 4000           
Cluster                           : Enabled      

Cisco Secure Firewall 3100 ASA Software Upgrade

The Cisco Secure Firewall 3100 series is the latest Next-Generation Firewall (NGFW) product from Cisco. The 3100 can be deployed to run either the classic ASA or the latest Firewall Thread Defense (FTD) software.

The 3100 front chassis has a fixed 8x RJ45 ports (Ethernet 1/1 - 1/8) and 8x fiber SFP ports (Ethernet 1/9 - 16).

The 3100 have an RJ45 and USB Console ports which are just beside the Management port (left).

The out of band Management port (Management 1/1) would need a GLC-TE copper SFP.

The 3100 has dual power supplies found in the rear and they're hot-swappable. It also has a power on/off toggle switch found on the left hand side.

 

I ran the classic ASA software and followed the upgrade path. I always choose an ASA software with a Long Term Release (LTR) which will be supported for 36 months (3 years) in terms of TAC support and software patches. LTR is designated by an even number in the second digit of its major release, i.e. FTD 6.4 and ASA 9.12. 

Aside from the ASA upgrade path, you should also follow the ASA and ASDM Compatibility Matrix (Table 2). In this case I chose ASA version 9.18 and its compatible ASDM should be 7.20(1).

You can download the 3100 ASA software and ASDM from the Cisco Software Download page. The upgrade procedure is identical with the classic ASA. Just change the boot variable to point to the new ASA version stored in flash memory (disk0).

The ASA now use Smart License which started around ASA version 9.4. 

First, transfer the ASA and ASDM images to flash memory then verify the MD5 hash.

cisocasa# copy ftp://ftpuser:ftp123@172.16.5.2/cisco-asa-fp3k.9.18.3.56.SPA disk0:

 

Address or name of remote host [172.16.5.2]?

 

Source username [ftpuser]?

 

Source password []? *******

 

Source filename [cisco-asa-fp3k.9.18.3.56.SPA]?

 

Destination filename [cisco-asa-fp3k.9.18.3.56.SPA]?

 

 

Accessing ftp://ftpuser:<password>@172.16.5.2/cisco-asa-fp3k.9.18.3.56.SPA...

!!!!!!!!!!

 

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Verifying file disk0:/cisco-asa-fp3k.9.18.3.56.SPA...

 

Writing file disk0:/cisco-asa-fp3k.9.18.3.56.SPA...

 

738779600 bytes copied in 4515.190 secs (163627 bytes/sec)

 

 

cisocasa# copy ftp://ftpuser:ftpuser@172.16.5.2/asdm-7201.bin disk0:

 

Address or name of remote host [172.16.5.2]?

 

Source username [ftpuser]?

 

Source password []? *******

 

Source filename [asdm-7201.bin]?

 

Destination filename [asdm-7201.bin]?

 

Accessing ftp://ftpuser:<password>@172.16.5.2/asdm-7201.bin...

!

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Verifying file disk0:/asdm-7201.bin...

 

Writing file disk0:/asdm-7201.bin...

 

116798028 bytes copied in 1029.990 secs (113506 bytes/sec)

 

 

ciscoasa# dir

 

Directory of disk0:/

 

<OUTPUT TRUNCATED>

1610981055  -rwx  116798028    02:22:49 Nov 01 2023  asdm-7201.bin

1610973629  -rwx  738779600    12:51:00 Oct 31 2023  cisco-asa-fp3k.9.18.3.56.SPA

 

8 file(s) total size: 972729823 bytes

16106127360 bytes total (14880296960 bytes free/92% free)

 

 

You can compare the hash output with the hash published in the Cisco Software Download website to confirm its authenticity and it's not corrupted during the file transfer.

 

ciscoasa# verify /md5 cisco-asa-fp3k.9.18.3.56.SPA

 

!!!!!!!!!!!!!!Done!

verify /MD5 (disk0:/cisco-asa-fp3k.9.18.3.56.SPA) = f466853bcebf15c81279e956e6c37906

 

 

ciscoasa# verify /md5 asdm-7201.bin

 

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!Done!

verify /MD5 (disk0:/asdm-7201.bin) = ba376c64777461ca587f8a8b5578554e

 

 

The ASA currently runs version 9.17 and ASDM 7.18.

 

ciscoasa# show version

 

Cisco Adaptive Security Appliance Software Version 9.17(1)21

SSP Operating System Version 2.11(1.191)

Device Manager Version 7.18(1)152

 

Compiled on Wed 16-Nov-22 00:04 GMT by builders

System image file is "disk0:/installables/switch/fxos-k8-fp3k-lfbff.2.11.1.191.SPA"

Config file at boot was "startup-config"

 

ciscoasa up 7 days 15 hours

Start-up time 3 secs

 

Hardware:   FPR-3110, 52169 MB RAM, CPU Ryzen Zen 2 2900 MHz, 1 CPU (24 cores)

 

Encryption hardware device : Cisco ASA Crypto on-board accelerator (revision 0x1)

                             AE microcode        : CNN5x-MC-AE-MAIN-0007

                             SE SSL microcode    : CNN5x-MC-SE-SSL-0018

                             Number of accelerators: 1

 

 1: Int: Internal-Data0/1    : address is 0000.0041.0004, irq 239

 3: Int: Not licensed        : irq 0

 4: Ext: Management1/1       : address is c47e.e07e.1482, irq 0

 5: Int: Internal-Data1/1    : address is 0000.0100.0001, irq 0

 

License mode: Smart Licensing

 

Licensed features for this platform:

Maximum Physical Interfaces       : Unlimited     

Maximum VLANs                     : 1024          

Inside Hosts                      : Unlimited     

Failover                          : Active/Active 

Encryption-DES                    : Enabled       

Encryption-3DES-AES               : Disabled      

Security Contexts                 : 2             

Carrier                           : Disabled      

AnyConnect Premium Peers          : 3000          

AnyConnect Essentials             : Disabled      

Other VPN Peers                   : 3000          

Total VPN Peers                   : 3000          

AnyConnect for Mobile             : Enabled       

AnyConnect for Cisco VPN Phone    : Enabled       

Advanced Endpoint Assessment      : Enabled       

Shared License                    : Disabled      

Total TLS Proxy Sessions          : 4000          

Cluster                           : Enabled       

 

Serial Number: FJZ27231234

Configuration register is 0x1

Configuration last modified by enable_15 at 02:47:41.036 UTC Wed Nov 1 2023

 

 

Change to the boot variable to point to the new ASA and ASDM image. Save the config and reload for the new ASA version to take effect. It's highly recommended to monitor the upgrade procedure via the console.

 

ciscoasa# show run boot

ciscoasa#

ciscoasa# show run asdm

no asdm history enable

 

ciscoasa# configure terminal

ciscoasa(config)# boot system disk0:/cisco-asa-fp3k.9.18.3.56.SPA

 

The system is currently installed with security software package 9.17.1.21, which has:

   - The platform version:  2.11.1.191

   - The CSP (asa) version: 9.17.1.21

Preparing new image for install...

!!!!!

Image download complete (Successful unpack the image).

Installation of version 9.18.3.56 will do the following:

   - upgrade to the new platform version 2.12.0.519

   - upgrade to the CSP ASA version 9.18.3.56

After installation is complete, ensure to do write memory and reload to save this config and apply the new image.

Finalizing image install process...

 

Install_status: ready.............................

Install_status: validating-images.

Install_status: upgrading-system

Install_status: upgrading-firmware

Install_status: update-software-pack-completed

 

ciscoasa(config)# asdm image disk0:/asdm-7201.bin

ciscoasai(config)# end

ciscoasa# write memory

Building configuration...

Cryptochecksum: 9db145a0 ceddd2a5 4416d104 91137070

 

14594 bytes copied in 0.260 secs

[OK]

 

 

It took around 2 mins for installing the new ASA software to finish.

 

ciscoasa# show run boot

boot system disk0:/cisco-asa-fp3k.9.18.3.56.SPA

ciscoasa# show run asdm

asdm image disk0:/asdm-7201.bin

no asdm history enable

 

 

ciscoasa# reload

Proceed with reload? [confirm]

ciscoasa#

 

 

***

*** --- START GRACEFUL SHUTDOWN ---

Shutting down Application Agent

Shutting down isakmp

Shutting down webvpn

Shutting down sw-module

Shutting down License Controller

Shutting down File system

 

 

***

*** --- SHUTDOWN NOW ---

Process shutdown finished

Rebooting... (status 0x9)

..

<13>Nov  1 04:08:42 root: FXOS shutdown log started: pid = 955 cmdline = /bin/sh/sbin/fxos_log_shutdown ####

 

Broadcast message from root@firepower-3110 (Wed Nov  1 04:08:42 2023):

 

The system is going down for reboot NOW!

2023 Nov 01 04:08:44 PMLOG: PM IPC UTILITY: Shutting down all ports

Stopping OpenBSD Secure Shell server: sshd

stopped /usr/sbin/sshd (pid 8992)

done.

Stopping Advanced Configuration and Power Interface daemon: stopped /usr/sbin/acpid (pid 1403)

acpid.

Stopping web server: apache2failed

Stopping system message bus: dbus.

Stopping DHCP server: dhcpd3no /usr/sbin/dhcpd found; none killed

.

stopping DNS forwarder and DHCP server: dnsmasq... no /usr/bin/dnsmasq found; none killed

stopping mountd: done

stopping nfsd: .done

Stopping ntpd: start-stop-daemon: warning: killing process 1441: No such process

done

Stopping internet superserver: xinetd.

stopping statd: done

Stopping random number generator daemon.

Stopping domain name service: named.

Stopping crond: OK

Stopping rpcbind daemon...

done.

Stopping fan control daemon: fancontrol... no process in pidfile '/var/run/fancontrol.pid' found; none killed

done.

Stopping sensors logging daemon: sensord... stopped /usr/sbin/sensord (pid 2433)

done.

 * Stopping virtualization library daemon: libvirtd                      [fail]

Deconfiguring network interfaces... done.

Stopping FreeRADIUS daemon radiusd Failed

Wed Nov  1 04:08:45 UTC 2023

SSP-Security-Module is shutting down ...

Wed Nov  1 04:08:45 UTC 2023 SHUTDOWN WARNING: Beginning System Shutdown request for CSP Apps

Wed Nov  1 04:08:45 UTC 2023 SHUTDOWN WARNING: Upgrade process ready for reboot

Wed Nov  1 04:08:45 UTC 2023 SHUTDOWN WARNING: Continue System Shutdown request for CSP Apps

DEBUG-CSPM: Checkpoint: autorun exist

Nov  1 04:08:45 firepower-3110 NVRAM: Confreg value: confreg = 0x1

DEBUG-CSPM: Checkpoint: autorun exist

omit_pids_opt: -o 680,683

Wed Nov  1 04:08:49 UTC 2023

Sending ALL processes the TERM signal ...

Note: SIGKILL_ALL will be triggered after after 1 + 2 secs ...

Wed Nov  1 04:08:51 UTC 2023

Sending ALL processes the KILL signal ...

Wed Nov  1 04:08:52 UTC 2023

Deactivating swap...

Unmounting local filesystems...

Stop Soft RAID

2023 Nov 01 04:08:53:

Soft-RAID configuration started

2023 Nov 01 04:08:53: found 1 devices

2023 Nov 01 04:08:53: enter FSM state PROBE

2023 Nov 01 04:08:53: probe /dev/nvme0n1

2023 Nov 01 04:08:53: superblock meta entries 6

2023 Nov 01 04:08:53: sysfs_probe_md: version=1.0, level=raid1, state=active

2023 Nov 01 04:08:53: enter FSM state DESTROY

2023 Nov 01 04:08:53: /sbin/mdadm --stop /dev/md0: 0

2023 Nov 01 04:08:54: enter FSM state PROBE

2023 Nov 01 04:08:54: probe /dev/nvme0n1

2023 Nov 01 04:08:54: superblock meta entries 6

2023 Nov 01 04:08:54: Soft-RAID configuration exit

2023 Nov 01 04:08:55: retrieving device list ...

2023 Nov 01 04:08:55: /usr/sbin/nvme subsystem-reset /dev/nvme0: 0

Reset TAM device ...

Rebooting... [661655.890391] reboot: Restarting system

 

 

<OUTPUT TRUNCATED>

 

                                                                              

                       Please do not remove the AC power!                     

                                                                              

 

          Insyde H2OFFT (Flash Firmware Tool) Version (SEG) 200.00.00.10

         Copyright (C) 2020 Insyde Software Corp. All Rights Reserved.

 

 

                           Current BIOS Model Name: FPR-3100

                           New     BIOS Model Name: FPR-3100

                           Current System BIOS Version: 1.2.04

                           New     BIOS  Image Version: 1.2.05

 

 

                        Updating Block at FFFF0000h       

          0%          25%         50%          75%         100%

           ****+++*******************************************     100%

                           Update Progress: Completed 

 

 

Checking media [Fail]

Checking media [Fail]

 

To launch ROMMON.

 

Time: 11/01/2023 04:26:47 (LOCAL)

 

 

*******************************************************************************

Cisco System ROMMON, Version 1.2.04, RELEASE SOFTWARE

Copyright (c) 1994-2022  by Cisco Systems, Inc.

Compiled Tue 10/18/2022 19:08:38.69 by Administrator

*******************************************************************************

 

Current image running: Boot ROM1

Last reset cause: ResetRequest (0x00001000)

DIMMs installed: P0 CHANNEL C P0 CHANNEL D

 

Platform FPR-3110 with 65536 MBytes of main memory

switch: bar0=0xd0800000 bar2=0xcc000000 bar4=0xd0000000 cmd=0x6

Switch Microinit: allocated buffer 5b758018, aligned buffer 5c000000

Mgmt port in SGMII mode

INFO: Firmware upgrade state: ROMMON_UPG_START (1)

firmware_upgrade: ROMMON_UPG_START

INFO: Reset code: 0x00001000

firmware_upgrade: ROMMON_UPG_START default

Active ROMMON: Preferred 1, selected 1, booted 1

 

Preparing to launch the new ROMMON upgrade image.

The new ROMMON upgrade image has been detected.

This will be launch attempt (1 of 4) to start the upgraded ROMMON image.

Power cyling the system to start the upgraded ROMMON image...

 

Toggling power on system board...

 

 

Checking media [Fail]

Checking media [Fail]

 

To launch ROMMON.

 

Time: 11/01/2023 04:29:20 (LOCAL)

 

 

*******************************************************************************

Cisco System ROMMON, Version 1.2.05, RELEASE SOFTWARE

Copyright (c) 1994-2022  by Cisco Systems, Inc.

Compiled Thu 12/08/2022 11:19:32.18 by builder

*******************************************************************************

 

Current image running: *Upgrade in progress* Boot ROM0

Last reset cause: BootRomUpgrade (0x00000010)

DIMMs installed: P0 CHANNEL C P0 CHANNEL D

 

Platform FPR-3110 with 65536 MBytes of main memory

switch: bar0=0xd0800000 bar2=0xcc000000 bar4=0xd0000000 cmd=0x6

Switch Microinit: allocated buffer 5bcad018, aligned buffer 5c000000

Mgmt port in SGMII mode

INFO: Firmware upgrade state: ROMMON_UPG_START (1)

firmware_upgrade: ROMMON_UPG_START

INFO: Reset code: 0x00000010

firmware_upgrade: ROMMON_UPG_START PLD_RST_REASON_FLASH

The upgraded ROMMON image has successfully started.

The boot watchdog timer is being stopped.

Active ROMMON: Preferred 1, selected 1, booted 0

INFO: File 'FS0:installables/switch/fxos-k8-fp3k-firmware.1.2.20.SPA' has 231330384 bytes.

fs_fopen_readonly: FileHandle 5d4fd020

Golden FPGA Version     : 0.21.0

New Golden FPGA version : 0.21.0

Golden FPGA image is up-to-date.

INFO: Set the ROMMON upgrade state: ROMMON_UPG_NONE

 

+-----------------------------------------------------------------+

+---------------  ROMMON FIRMWARE UPGRADE SUCCESS  ---------------+

+-----------------------------------------------------------------+

|                                                                 |

| Start the security application to complete the ROMMON upgrade.  |

|                                                                 |

| Rebooting this unit without starting the security application   |

| will cause the ROMMON to default back to the previously running |

| ROMMON version.                                                 |

|                                                                 |

+-----------------------------------------------------------------+

 

MAC Address: c4:7e:e0:7e:12:34

 

Use BREAK or ESC to interrupt boot.

Use SPACE to begin boot immediately.

Boot in 10 seconds.

 

<OUTPUT TRUNCATED>

 

INFO: Configure management0 interface ...

INFO: Configure system files ...

INFO: System Name is: firepower-3110

Create 16 QDMA VFs from PF: 0000:41:00.0

Starting sensors logging daemon: sensord... done.

INFO: fp1000 asa copy appliance mode

INFO: console : ttyS0, speed : 9600

INFO: manager_startup: setting up fxmgr apache ...

INFO: manager_startup: Start manager httpd setup...

INFO: manager_startup: using HTTPD_INFO persistent cache

/bin/rm: cannot remove '/tmp/openssl.conf': No such file or directory

 httpdRegister INFO: [httpd.2689 -s -4 0.0.0.0 -n localhost]

 httpdRegister INFO: SKIP httpd syntax check

 httpdRegister INFO: Starting httpd setup/registration...

 httpdRegister INFO: Completed httpd setup/registration!

 INFO: httpdRegister [httpd.2689 script exit]

INFO: manager_startup: Completed manager httpd setup!

Starting crond: OK

1:/opt/cisco/csp/cores

2:/opt/cisco/csp/packet-capture

/opt/cisco/csp/cores 62914560

/opt/cisco/csp/packet-capture 41943040

System Mode Check: NATIVE mode assigned

System Mode Check: NATIVE mode assigned

System Mode Check: NATIVE mode assigned

 

Cisco ASA: CMD=-bootup, CSP-ID=cisco-asa.9.17.1.21__asa_001_FJC27261SLF3EN1234, FLAG=''

Cisco ASA booting up ...

INFO: starting config regster monitor

 

 

System Mode Check: NATIVE mode assigned

firepower-3110 login: admin (automatic login)

 

Last login: Tue Oct 24 12:22:18 UTC 2023 on ttyS0

Successful login attempts for user 'admin' : 1

INFO: System Disk /dev/md0 present. Status: Operable.

System Mode Check: NATIVE mode assigned

System Mode Check: NATIVE mode assigned

 

Waiting for Application infrastructure to be ready...

Verifying the signature of the Application image...

System Mode Check: NATIVE mode assigned

Creating FXOS swap file ...

Please wait for Cisco ASA to come online...1...

Please wait for Cisco ASA to come online...2...

Please wait for Cisco ASA to come online...3...

Please wait for Cisco ASA to come online...4...

Please wait for Cisco ASA to come online...5...

Please wait for Cisco ASA to come online...6...

Please wait for Cisco ASA to come online...7...

 

Cisco ASA: CMD=-upgrade, CSP-ID=cisco-asa.9.18.3.56__asa_001_FJC27261SLF3EN1234, FLAG='cisco-asa.9.17.1.21__asa_001_FJC27261SLF3EN1234'

Cisco ASA begins upgrade ...

Please wait for Cisco ASA to come online...8...

Verifying signature for cisco-asa.9.18.3.56 ...

Verifying signature for cisco-asa.9.18.3.56 ... success

Please wait for Cisco ASA to come online...9...

 

Cisco ASA: CMD=-start, CSP-ID=cisco-asa.9.18.3.56__asa_001_FJC27261SLF3EN1234, FLAG=''

Cisco ASA starting ...

ASA start done pre

ASA Clear status

Memory allocated to application in kbytes: 54914048

CPU cores allocated to application: 1,13,2,14,3,15,4,16,5,17,6,18,7,19,8,20,9,21,10,22,11,23

Deleting previous CGroup Configuration ...

Registering to process manager ...

Cisco ASA started successfully.

lina_init_env: memif is not enabled.

System Cores 24 Nodes 1 Max Cores 128

IO Memory Nodes: 1

IO Memory Per Node: 2147483648 bytes num_pages = 524288 page_size = 4096

 

Global Reserve Memory Per Node: 2147483648 bytes Nodes=1

 

LCMB: got DMA 2147483648 bytes on numa-id=0, phys=0x0000000180000000, virt=0x00007fa040000000

LCMB: HEAP-CACHE POOL got 2147483648 bytes on numa-id=0, virt=0x00007f9f80000000

 

total_reserved_mem = 2147483648

 

total_heapcache_mem = 2147483648

ERROR: fail to open /var/run/lina/meminfo_new

ERROR: fail to open /var/run/lina/meminfo_old

total mem 54702424064 system 67387310080 kernel 134217728 image 112999912

new 54702424064 old 2260483560 reserve 4294967296 priv new 50541674496 priv old 0

Processor memory:  54702424064

M_MMAP_THRESHOLD 65536, M_MMAP_MAX 834692

POST started...

POST finished, result is 0 (hint: 1 means it failed)

 

Cisco Adaptive Security Appliance Software Version 9.18(3)56

 

Compiled on Tue 12-Sep-23 19:15 GMT by builders

FPR-3110 platformNic assigned 0

 

Total NICs found: 5

cpss_poll_devmain success!!

en_vtun rev00 Backplane Ext-Mgmt Interface     @ index 03 MAC: c47e.e07e.1234

en_vtun rev00 Backplane Tap Interface     @ index 04 MAC: 0000.0100.0001

livecore intialized

Counter ID 'TLS13_DOWNSTREAM_CLIENT_CERTIFICATE_VERIFY' is too long must be 40 characters or less

WARNING: Attribute already exists in the dictionary.

ILK enabled for instance 0 with lane mask 0xF speed 6250 MHz

Init ILK - NPS_CORE_GBL_VFCFG 0X00000000

Configure the GSER registers

ILK configured on QLM 0 with ref_clk 156250000 Hz, baud 6250 MHz, instance 0

QLM0: Lane 0: TX_SWING=16, TX_PRE=0, TX_POST=4, TX_GAIN=-1, TX_VBOOST=-1

QLM0: Lane 1: TX_SWING=16, TX_PRE=0, TX_POST=4, TX_GAIN=-1, TX_VBOOST=-1

QLM0: Lane 2: TX_SWING=16, TX_PRE=0, TX_POST=4, TX_GAIN=-1, TX_VBOOST=-1

QLM0: Lane 3: TX_SWING=16, TX_PRE=0, TX_POST=4, TX_GAIN=-1, TX_VBOOST=-1

ILK configured on QLM 1 with ref_clk 156250000 Hz, baud 6250 MHz, instance 0

Clear TX/TX calendars

Configure the SERDES for all possible lanes

Configure TX / RX Calendars

Enable per lane RX error counts

Bring up the TX side

Configure the RX lanes

RX equalization for speeds > 5G

QLM0: Lane 0 RX equalization complete

QLM0: Lane 1 RX equalization complete

QLM0: Lane 2 RX equalization complete

QLM0: Lane 3 RX equalization complete

Bring up RX link

ILK0: Lane alignment complete

Encryption hardware device : Cisco ASA Crypto on-board accelerator (revision 0x1)

                             AE microcode        : CNN5x-MC-AE-MAIN-0007

                             SE SSL microcode    : CNN5x-MC-SE-SSL-0018

The 3DES/AES algorithms require a Encryption-3DES-AES entitlement.

The 3DES/AES algorithms require a Encryption-3DES-AES entitlement.

 

Cisco Adaptive Security Appliance Software Version 9.18(3)56

 

  ****************************** Warning *******************************

  This product contains cryptographic features and is

  subject to United States and local country laws

  governing, import, export, transfer, and use.

  Delivery of Cisco cryptographic products does not

  imply third-party authority to import, export,

  distribute, or use encryption. Importers, exporters,

  distributors and users are responsible for compliance

  with U.S. and local country laws. By using this

  product you agree to comply with applicable laws and

  regulations. If you are unable to comply with U.S.

  and local laws, return the enclosed items immediately.

 

  A summary of U.S. laws governing Cisco cryptographic

  products may be found at:

  http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

 

  If you require further assistance please contact us by

  sending email to export@cisco.com.

  ******************************* Warning *******************************

Cisco Adaptive Security Appliance Software, version 9.18

Copyright (c) 1996-2023 by Cisco Systems, Inc.

For licenses and notices for open source software used in this product, please visit

http://www.cisco.com/go/asa-opensource

 

                Restricted Rights Legend

Use, duplication, or disclosure by the Government is

subject to restrictions as set forth in subparagraph

(c) of the Commercial Computer Software - Restricted

Rights clause at FAR sec. 52.227-19 and subparagraph

(c) (1) (ii) of the Rights in Technical Data and Computer

Software clause at DFARS sec. 252.227-7013.

 

                Cisco Systems, Inc.

                170 West Tasman Drive

                San Jose, California 95134-1706

 

Reading from flash...

!!!.......

Cryptochecksum (unchanged): 9db145a0 ceddd2a5 4416d104 91137070

INFO: File /mnt/disk0/.private/dynamic-config.json not opened; errno 2

INFO: Network Service reload not performed.

 

INFO: Power-On Self-Test in process.

........................

INFO: Power-On Self-Test complete.

 

INFO: Starting HW-DRBG health test...

INFO: HW-DRBG health test passed.

 

INFO: Starting SW-DRBG health test...

INFO: SW-DRBG health test passed.

User enable_1 logged in to ciscoasa

Logins over the last 1 days: 1. 

Failed logins since the last login: 0. 

Type ' for a list of available commands.

 

 

ciscoasa> show version

 

Cisco Adaptive Security Appliance Software Version 9.18(3)56

SSP Operating System Version 2.12(0.519)

Device Manager Version 7.20(1)

 

Compiled on Tue 12-Sep-23 19:15 GMT by builders

System image file is "disk0:/installables/switch/fxos-k8-fp3k-lfbff.2.12.0.519.SPA"

Config file at boot was "startup-config"

 

ciscoasa up 1 min 57 secs

Start-up time 8 secs

 

Hardware:   FPR-3110, 52168 MB RAM, CPU Ryzen Zen 2 2900 MHz, 1 CPU (24 cores)

 

Encryption hardware device : Cisco ASA Crypto on-board accelerator (revision 0x1)

                             AE microcode        : CNN5x-MC-AE-MAIN-0007

                             SE SSL microcode    : CNN5x-MC-SE-SSL-0018

                             Number of accelerators: 1

 

 1: Int: Internal-Data0/1    : address is 0000.0041.0004, irq 152

 3: Int: Not licensed        : irq 0

 4: Ext: Management1/1       : address is c47e.e07e.1482, irq 0

 5: Int: Internal-Data1/1    : address is 0000.0100.0001, irq 0

 

License mode: Smart Licensing

             

Licensed features for this platform:

Maximum Physical Interfaces       : Unlimited     

Maximum VLANs                     : 1024          

Inside Hosts                      : Unlimited     

Failover                          : Active/Active 

Encryption-DES                    : Enabled       

Encryption-3DES-AES               : Disabled      

Security Contexts                 : 2             

Carrier                           : Disabled      

AnyConnect Premium Peers          : 3000          

AnyConnect Essentials             : Disabled      

Other VPN Peers                   : 3000          

Total VPN Peers                   : 3000          

AnyConnect for Mobile             : Enabled       

AnyConnect for Cisco VPN Phone    : Enabled       

Advanced Endpoint Assessment      : Enabled       

Shared License                    : Disabled      

Total TLS Proxy Sessions          : 4000          

Cluster                           : Enabled       

 

Serial Number: FJZ27231234

Configuration register is 0x1

Configuration has not been modified since last system restart.

 

 

It took around 30 mins for the ASA software upgrade to finish.