Troubleshoot FortiGuard Server Connectivity

Here's a Fortinet link in troubleshooting FortiGuard server connectivity over the Internet. I was configuring a new FortiGate firewall in Multiple VDOM mode but I can't ping or perform a license update to the FortiGuard server (a cloud service over the Internet).

FGT # config vdom

 

FGT (vdom) # edit root

current vf=root:0

 

FGT (root) # execute ping update.fortiguard.net

Unable to resolve hostname.

 

The FortiGate uses FortiGuard public DNS server IP: 96.45.45.45 and 96.45.46.46 by default. To change DNS server settings, go to Network > DNS > select: Specify > type the usable public DNS server IP (Google DNS 8.8.8.8 or your private DNS server) > enable/toggle: DNS (UDP/53) > click Apply.

FGT (root) # execute ping service.fortiguard.net

PING guard.fortinet.net (208.184.237.61): 56 data bytes

64 bytes from 208.184.237.61: icmp_seq=0 ttl=47 time=255.0 ms

64 bytes from 208.184.237.61: icmp_seq=1 ttl=47 time=254.7 ms

64 bytes from 208.184.237.61: icmp_seq=2 ttl=47 time=254.7 ms

64 bytes from 208.184.237.61: icmp_seq=3 ttl=47 time=254.7 ms

64 bytes from 208.184.237.61: icmp_seq=4 ttl=47 time=254.7 ms

 

--- guard.fortinet.net ping statistics ---

5 packets transmitted, 5 packets received, 0% packet loss

round-trip min/avg/max = 254.7/254.7/255.0 ms

 

FGT (root) # execute ping update.fortiguard.net

PING fds1.fortinet.com (12.34.97.16): 56 data bytes

64 bytes from 12.34.97.16: icmp_seq=0 ttl=46 time=332.9 ms

64 bytes from 12.34.97.16: icmp_seq=1 ttl=46 time=333.5 ms

64 bytes from 12.34.97.16: icmp_seq=2 ttl=46 time=333.4 ms

64 bytes from 12.34.97.16: icmp_seq=3 ttl=46 time=333.5 ms

64 bytes from 12.34.97.16: icmp_seq=4 ttl=46 time=337.0 ms

 

--- fds1.fortinet.com ping statistics ---

5 packets transmitted, 5 packets received, 0% packet loss

round-trip min/avg/max = 332.9/334.0/337.0 ms

 

FGT (root) # execute ping guard.fortinet.net

PING guard.fortinet.net (208.184.237.61): 56 data bytes

64 bytes from 208.184.237.61: icmp_seq=0 ttl=47 time=254.9 ms

64 bytes from 208.184.237.61: icmp_seq=1 ttl=47 time=254.6 ms

64 bytes from 208.184.237.61: icmp_seq=2 ttl=47 time=254.5 ms

64 bytes from 208.184.237.61: icmp_seq=3 ttl=47 time=254.5 ms

64 bytes from 208.184.237.61: icmp_seq=4 ttl=47 time=254.5 ms

 

--- guard.fortinet.net ping statistics ---

5 packets transmitted, 5 packets received, 0% packet loss

round-trip min/avg/max = 254.5/254.6/254.9 ms

 

 

The FortiCare (support and warranty), Next Generation Firewall licenses (Intrusion Prevention, Antivirus and Web filtering) were updated/enabled after a few minutes.

 

The Internet Service Database (ISDB) objects were updated as well. You can view these under Policy & Objects > Internet Service Database.

The ISDB is a comprehensive list of public IP addresses (Geolocation based), service/port numbers, reputation, popularity (Facebook, Amazon, Microsoft, etc.) which can be used in creating a firewall policy or security profile (Antivirus, Web filter, Application Control, etc.) in a FortiGate firewall.

Create a Fortinet Support Ticket

Here's a Fortinet link for device hardening and best practice in a FortiGate firewall.

To create a new Fortinet support ticket, go to this link > select Create a Ticket.

Select a Request Ticket Type (closest to your issue/inquiry). In this case, I selected Customer Service > Submit ticket.

Select a CS category. In this case I selected: Cloud Portal Query.

Put the device Serial Number > Contact Information > Ticket Information.

Add Comment to describe your issue or upload a screenshot of the error in the Attachments.

Click Finish and note the ticket number. The Fortinet ticket number and summary will be sent to your registered email.

Another way to create a Fortinet ticket is via the Asset Management portal. Click Support > FortiCare > Create a Ticket.

Click New Ticket.

Choose: Technical Support Ticket > Submit Ticket.

You can get the FortiGate serial number with the get system status CLI command:

FG# get system status

Version: FortiGate-xx v7x,buildxx

Security Level: 2

Firmware Signature: certified

Virus-DB: 1.00000(2018-04-09 18:07)

Extended DB: 1.00000(2018-04-09 18:07)

Extreme DB: 1.00000(2018-04-09 18:07)

AV AI/ML Model: 0.00000(2001-01-01 00:00)

IPS-DB: 6.00741(2015-12-01 02:30)

IPS-ETDB: 6.00741(2015-12-01 02:30)

APP-DB: 6.00741(2015-12-01 02:30)

INDUSTRIAL-DB: 6.00741(2015-12-01 02:30)

IPS Malicious URL Database: 1.00001(2015-01-01 01:01)

IoT-Detect: 0.00000(2022-08-17 17:31)

Serial-Number: FGxx

BIOS version: 06000008

System Part-Number: Pxx

Log hard disk: Available

Hostname: FG

Private Encryption: Disable

Operation Mode: NAT

Current virtual domain: root

Max number of virtual domains: 10

Virtual domains status: 2 in NAT mode, 0 in TP mode

Virtual domain configuration: multiple

FIPS-CC mode: disable

Current HA mode: a-p, primary

Cluster uptime: 241 days, 23 hours, 33 minutes, 20 seconds

Cluster state change time: 2024-03-06 07:04:47

Branch point: xx

Release Version Information: GA

FortiOS x86-64: Yes

System time: Fri Sep  6 03:20:07 2024

Last reboot reason: warm reboot

 

Or retrieve it via the web GUI under Dashboard > Status.

Under the Product Info > type the device SN > click Go

Fill up the required info > click Next.

Type the Comment (answer the pre-filled questionnaire) or click File Upload to upload a screenshot of the error.

It's also very useful to upload the Debug log which is similar to show tech-support in a Cisco device. Go to System > Settings > Debug logs > click Download.

It only took a few seconds to download the Debug log text file. Here's a snippet of the Debug log output:

----------------------------------------------------------------

  Serial Number: FG4Hxx   Diagnose output

----------------------------------------------------------------

 

### get system status

 

 

Version: FortiGate-xxv7x

Security Level: 2

Firmware Signature: certified

Virus-DB: 1.00000(2018-04-09 18:07)

Extended DB: 1.00000(2018-04-09 18:07)

Extreme DB: 1.00000(2018-04-09 18:07)

AV AI/ML Model: 0.00000(2001-01-01 00:00)

IPS-DB: 6.00741(2015-12-01 02:30)

IPS-ETDB: 6.00741(2015-12-01 02:30)

APP-DB: 6.00741(2015-12-01 02:30)

INDUSTRIAL-DB: 6.00741(2015-12-01 02:30)

IPS Malicious URL Database: 1.00001(2015-01-01 01:01)

IoT-Detect: 0.00000(2022-08-17 17:31)

Serial-Number: FG4xx

BIOS version: 06000008

System Part-Number: P27xx

Log hard disk: Available

Hostname: xx

Private Encryption: Disable

Operation Mode: NAT

Current virtual domain: root

Max number of virtual domains: 10

Virtual domains status: 2 in NAT mode, 0 in TP mode

Virtual domain configuration: multiple

FIPS-CC mode: disable

Current HA mode: a-p, primary

Cluster uptime: 241 days, 23 hours, 40 minutes, 25 seconds

Cluster state change time: 2024-03-06 07:04:47

Branch point: xx

Release Version Information: xx

FortiOS x86-64: Yes

System time: Fri Sep  6 03:27:12 2024

Last reboot reason: warm reboot

 

### get system performance status

 

 

CPU states: 1% user 0% system 0% nice 99% idle 0% iowait 0% irq 0% softirq

CPU0 states: 2% user 0% system 0% nice 98% idle 0% iowait 0% irq 0% softirq

CPU1 states: 0% user 0% system 0% nice 99% idle 0% iowait 0% irq 1% softirq

CPU2 states: 7% user 5% system 0% nice 87% idle 0% iowait 0% irq 1% softirq

CPU3 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq

CPU4 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq

CPU5 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq

CPU6 states: 0% user 0% system 0% nice 99% idle 0% iowait 0% irq 1% softirq

CPU7 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq

CPU8 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq

 

<OUTPUT TRUNCATED>

 

 

Review the ticket summary before submitting > click Confirm to proceed.

 

Cisco GRE Tunnel Keepalive

This Cisco link covers the GRE Tunnel and how a keepalive works. I got a GRE over IPSec VPN configured between Singapore and London. The GRE tunnel only goes up whenever I perform a ping. So I configured the GRE tunnel keepalive so it always stays up. The default keepalive interval is 10 seconds and 3 retries.

SIN#show run interface Tunnel40
Building configuration...

Current configuration : 314 bytes
!
interface Tunnel40
 ip address 10.16.2.194 255.255.255.252
 ip mtu 1400
 tunnel source 192.168.1.18
 tunnel destination 192.168.1.146
end

SIN#configure terminal
Enter configuration commands, one per line.  End with CNTL/Z.
SIN(config)#interface Tunnel40
SIN(config-if)#keepalive ?
  <0-32767>  Keepalive period (default 10 seconds)
  <cr>       <cr>

SIN(config-if)#keepalive
SIN(config-if)#end
SIN#write memory
Building configuration...
[OK]

SIN#show run interface Tunnel40
Building configuration...

Current configuration : 330 bytes
!
interface Tunnel40
 ip address 10.106.192.194 255.255.255.252
 ip mtu 1400
 keepalive 10 3
 tunnel source 192.168.1.18
 tunnel destination 192.168.1.146
end

LON#show run interface Tunnel40
Building configuration...

Current configuration : 322 bytes
!
interface Tunnel40
 ip address 10.16.2.193 255.255.255.252
 ip mtu 1400
 tunnel source 192.168.1.146
 tunnel destination 192.168.1.18
end

LON#configure terminal
Enter configuration commands, one per line.  End with CNTL/Z.
LON(config)#interface Tunnel40
LON(config-if)#keepalive
LON(config-if)#end
LON#write memory
Building configuration...
[OK]

LON#show run interface Tunnel40
Building configuration...

Current configuration : 338 bytes
!
interface Tunnel40
 ip address 10.16.2.193 255.255.255.252
 ip mtu 1400
 keepalive 10 3   // DEFAULT IS 10 SECOND INTERVAL AND 3 RETRIES
 tunnel source 192.168.1.146
 tunnel destination 192.168.1.18
end

LON#ping 10.16.2.194
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.16.2.194, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 181/181/181 ms

 

I've checked the IPSec VPN was already up in the FortiGate firewall even before I did a ping.

Cisco ASA Firewall SNMP OID

There's a "hidden" Cisco ASA CLI command in order to retrieve the SNMP MIB OID info without performing an SNMP walk from a remote server/NMS. The Management Information Base (MIB) is the hierarchical (tree) structure of the SNMP Object Identifier (OID). OID is the long code string or numbers separated by dots. It uniquely identifies an SNMP managed object such as a device interface, CPU, memory, bandwidth/traffic stats, etc.

I had a high CPU alarm in our NMS but there was no high CPU when checked. It was later found out the NMS didn't support the new Firepower ASA platform using multiple Core CPU, so it needed to update its MIB OID database.

FPR2100# show cpu core all

Core              5 sec  1 min  5 min

Core 0             1.4%   0.8%   0.7%

Core 1             0.2%   0.2%   0.2%

Core 2             0.2%   0.2%   0.2%

Core 3             0.2%   0.2%   0.2%

Core 4             0.2%   0.2%   0.2%

Core 5             0.2%   0.2%   0.2%

Core 6             0.2%   0.2%   0.2%

Core 7             0.2%   0.2%   0.2%

Core 8             0.2%   0.2%   0.2%

Core 9             0.2%   0.2%   0.2%

Core 10            0.2%   0.2%   0.2%

Core 11            1.0%   0.6%   0.5%

Core 12            0.2%   0.2%   0.2%

Core 13            0.2%   0.2%   0.2%

Core 14            0.2%   0.2%   0.2%

Core 15            0.2%   0.2%   0.2%

Core 16            0.2%   0.2%   0.2%

Core 17            0.2%   0.2%   0.2%

Core 18            0.2%   0.2%   0.2%

Core 19            0.2%   0.2%   0.2%

Core 20            0.2%   0.2%   0.2%

Core 21            0.2%   0.2%   0.2%

The output below came a Cisco ASA5515-X firewall. You'll need to run this command in the admin context if the ASA is in Multiple Context mode.

ciscoasa# show snmp-server ?      

  engineID    Show snmp engineID
  group       Show snmp groups
  host        Show snmp host's
  statistics  Show snmp-server statistics
  user        Show snmp users 

ciscoasa# show snmp-server oidlist ?   // IT'S A HIDDEN CLI COMMAND
ERROR: % Unrecognized command

ciscoasa# show snmp-server oidlist

-------------------------------------------------
[0]     1.3.6.1.2.1.1.1.        sysDescr
[1]     1.3.6.1.2.1.1.2.        sysObjectID
[2]     1.3.6.1.2.1.1.3.        sysUpTime
[3]     1.3.6.1.2.1.1.4.        sysContact
[4]     1.3.6.1.2.1.1.5.        sysName
[5]     1.3.6.1.2.1.1.6.        sysLocation
[6]     1.3.6.1.2.1.1.7.        sysServices
[7]     1.3.6.1.2.1.1.8.        sysORLastChange
[8]     1.3.6.1.2.1.1.9.1.2.    sysORID
[9]     1.3.6.1.2.1.1.9.1.3.    sysORDescr
[10]    1.3.6.1.2.1.1.9.1.4.    sysORUpTime
[11]    1.3.6.1.2.1.2.1.        ifNumber
[12]    1.3.6.1.2.1.2.2.1.1.    ifIndex
[13]    1.3.6.1.2.1.2.2.1.2.    ifDescr
[14]    1.3.6.1.2.1.2.2.1.3.    ifType
[15]    1.3.6.1.2.1.2.2.1.4.    ifMtu
[16]    1.3.6.1.2.1.2.2.1.5.    ifSpeed
[17]    1.3.6.1.2.1.2.2.1.6.    ifPhysAddress
[18]    1.3.6.1.2.1.2.2.1.7.    ifAdminStatus
[19]    1.3.6.1.2.1.2.2.1.8.    ifOperStatus
[20]    1.3.6.1.2.1.2.2.1.9.    ifLastChange
[21]    1.3.6.1.2.1.2.2.1.10.   ifInOctets
[22]    1.3.6.1.2.1.2.2.1.11.   ifInUcastPkts
<--- More --->

<OUTPUT TRUNCATED>

[1002]  1.3.6.1.6.3.15.1.2.2.1.3.       usmUserSecurityName
[1003]  1.3.6.1.6.3.15.1.2.2.1.4.       usmUserCloneFrom
[1004]  1.3.6.1.6.3.15.1.2.2.1.5.       usmUserAuthProtocol
[1005]  1.3.6.1.6.3.15.1.2.2.1.6.       usmUserAuthKeyChange
[1006]  1.3.6.1.6.3.15.1.2.2.1.7.       usmUserOwnAuthKeyChange
[1007]  1.3.6.1.6.3.15.1.2.2.1.8.       usmUserPrivProtocol
[1008]  1.3.6.1.6.3.15.1.2.2.1.9.       usmUserPrivKeyChange
[1009]  1.3.6.1.6.3.15.1.2.2.1.10.      usmUserOwnPrivKeyChange
[1010]  1.3.6.1.6.3.15.1.2.2.1.11.      usmUserPublic
[1011]  1.3.6.1.6.3.15.1.2.2.1.12.      usmUserStorageType
[1012]  1.3.6.1.6.3.15.1.2.2.1.13.      usmUserStatus
[1013]  1.3.6.1.6.3.16.1.2.1.3. vacmGroupName
[1014]  1.3.6.1.6.3.16.1.2.1.4. vacmSecurityToGroupStorageType
[1015]  1.3.6.1.6.3.16.1.2.1.5. vacmSecurityToGroupStatus
-------------------------------------------------

 

Cisco Firepower FX-OS show tech-support

We've encountered an outage in our Cisco FPR 2100 High Availability (HA) pair running ASA OS. The Primary ASA firewall crashed or auto reload and didn't failover properly to the Secondary ASA firewall. I also had to reload the Secondary ASA firewall in order for HA to synchronize.

Aside from the usual show tech-support in the ASA command, Cisco TAC will ask for the show tech-support fprm detail output (GZ archive file), which can be generated from the FX-OS CLI. This saves time in troubleshooting and would allow TAC to further investigate using their internal database.

Based on the FX-OS show tech, the auto reload was due to a memory bug CSCwk27830. TAC recommended to perform an ASA OS upgrade using the known fixed release.

Threadname: **lina**

| Rip: ****

| Version: **9.xx**

| Hardware: **FPR-21xx**

| 0x00000000019862b8 : ikev2_copy_ike_policy+216 at ikev2/granite/ikev2/core/policy/ikev2_policy.c:1677

| 0x00000000019c1144 : ikev2_initiate_sa+476 at ikev2/granite/ikev2/core/ikev2_sa_management.c:132

| 0x00000000018e300c : asa_connect_continue+136 at ikev2/ikev2_asa_connect.c:663

| 0x000000000193f214 : asa_spi_mgt_callback+1060 at ikev2/ikev2_spi_mgt.c:666

| 0x000000000193dcc0 : ikev2_pitcher+328 at ikev2/ikev2_pitcher.c:880

| 0x000000000193a768 : IKEv2ProcessMsg+140 at ikev2/ikev2_daemon.c:548

| 0x000000000193c9c4 : Ikev2Daemon+1452 at ikev2/ikev2_daemon.c:343

 

ciscoasa/pri/act/admin# connect fxos admin
Configuring session.
.
Connecting to FXOS.
...
Connected to FXOS. Escape character sequence is 'CTRL-^X'.

NOTICE: You have connected to the FXOS CLI with admin privileges.
Config commands and commit-buffer are not supported in appliance mode.


Certain components of this software are licensed under the "GNU General Public
License, version 3" provided with ABSOLUTELY NO WARRANTY under the terms of
"GNU General Public License, Version 3", available here:
http://www.gnu.org/licenses/gpl.html. See User Manual (''Licensing'') for
details.

<OUTPUT TRUNCATED>


firepower-2100# connect
  asa         Connect to ASA Application CLI
  local-mgmt  Connect to Local Management CLI


firepower-2120# connect local-mgmt
Warning: network service is not available when entering 'connect local-mgmt'
firepower-2120(local-mgmt)# show  
  active-connections     Show active TCP/IP connections
  cli                    CLI Information
  clock                  Clock
  consent-token          consent token
  debug                  Debugging functions
  env                    Show environmental monitoring data
  failsafe-params        Show the failsafe mode configuration
  file                   File Commands
  fxos-mode              Fxos-mode
  lacp                   LACP command
  mgmt-ip-debug          IP Debug Info
  npu-accel              Show NPU accelerator data
  ntp                    NTP Status
  open-network-ports     Show open network ports
  pktmgr                 pktmgr command
  platform-sw-processes  Show the state of platform software processes
  pmon                   Pmon
  portchannel            portchannel command
  portmanager            portmanager command
  processes              Processes
  running-config         Running-config
  software               Software
  sshkey                 Sshkey
  tech-support           Tech Support
  version                System version


firepower-2100(local-mgmt)# show tech-support
  fprm  FPRM


firepower-2100(local-mgmt)# show tech-support fprm
  <CR>      
  >       Redirect it to a file
  >>      Redirect it to a file in append mode
  brief   Brief
  detail  Detail
  |       Pipe command output to filter


firepower-2120(local-mgmt)# show tech-support fprm detail

The show tech output is savedin the ASA flash (disk0:) and can be simply transferred to your PC via ASDM and then upload it to the Cisco Support Case portal.

ciscoasa/pri/act/admin# changeto system
ciscoasa/pri/act# show flash
--#--  --length--  -----date/time------  path
44053  98          Apr 06 2023 07:50:39  log
134673345  4096        Jun 26 2023 05:19:00  log/from_tmp
134673346  145713      Jul 31 2024 17:26:02  log/from_tmp/asa-appagent.log
134673347  0           Jul 22 2024 16:46:23  log/from_tmp/asa-fxos_xml.log

<OUTPUT TRUNCATED>

134217933  17421854    Jul 30 2024 05:41:58  fxos/20240730054152_firepower-2100_FPRM.tar.gz

<OUTPUT TRUNCATED>

21475885056 bytes total (20623392768 bytes free)

Changing a Cisco Switchport Mode From Access to Trunk

I had to reconfigure a Cisco switchport mode from access to a trunk in order to run multiple VLANs in a Cisco ASA firewall interface. I configured a new sub-interface on the ASA using VLAN 10.

ciscoasa# show interface ip brief
Interface                  IP-Address      OK? Method Status                Protocol
GigabitEthernet0/0         46.4.4.66    YES CONFIG up                    up  
GigabitEthernet0/1         172.30.3.4    YES CONFIG up                    up  
GigabitEthernet0/1.10     172.20.1.7     YES manual up                    up  
GigabitEthernet0/2         unassigned      YES unset  administratively down down
GigabitEthernet0/3         unassigned      YES unset  administratively down down
GigabitEthernet0/4         unassigned      YES unset  administratively down down
GigabitEthernet0/5         unassigned      YES unset  administratively down down
GigabitEthernet0/6         unassigned      YES unset  administratively down down
GigabitEthernet0/7         172.30.3.254  YES unset  up                    up  
Internal-Control0/0        127.0.1.1       YES unset  up                    up  
Internal-Data0/0           unassigned      YES unset  up                    up  
Internal-Data0/1           unassigned      YES unset  down                  down
Internal-Data0/2           unassigned      YES unset  up                    up  
Internal-Data0/3           169.254.1.1     YES unset  up                    up  
Management0/0              10.10.6.9   YES CONFIG up                    up  

 

ciscoasa# show run interface GigabitEthernet0/1
!
interface GigabitEthernet0/1    <<< VLAN 30 ON SWITCH
 description | SW G1/0/3 : INSIDE |
 nameif inside
 security-level 100
 ip address 172.30.3.4 255.255.255.0

 

ciscoasa# show run interface GigabitEthernet0/1.10
!
interface GigabitEthernet0/1.10
 description | DMZ |
 vlan 10
 nameif dmz
 security-level 100
 ip address 172.20.1.7 255.255.255.248

Since it's a remote site with an IPSec VPN over the Internet via the native "inside" interface, I had to use the reload command to avoid being locked out. I reconfigured the switch port from an access port (single VLAN 30) to a trunk. I also used the switchport trunk native vlan command in order for the original "inside" interface to work (untagged). Once everything resumed and working, I canceled the reload command.

Switch#reload in 10
Reload scheduled for 10:24:35 UTC Thu Feb 8 2024 (in 10 minutes) by john on vty0 (172.30.3.1)Reload command is being issued on Active unit, this will reload the whole stack
Proceed with reload? [confirm]
Switch#configure terminal
Enter configuration commands, one per line.  End with CNTL/Z.
Switch(config)#interface GigabitEthernet1/0/3
Switch(config-if)#switchport trunk native vlan 30
Switch(config-if)#switchport mode trunk

There was about a 5 second outage, then my SSH session resumed.

Switch(config-if)#no switchport access vlan 30
Switch(config-if)#no spanning-tree portfast
Switch(config-if)#end

Switch#show run interface g1/0/3
Building configuration...

Current configuration : 140 bytes
!
interface GigabitEthernet1/0/3
 description | FW G0/1 : INSIDE |
 switchport trunk native vlan 30
 switchport mode trunk
end

Switch#reload cancel


***
*** --- SHUTDOWN ABORTED ---
***
 

The proper design in the ASA should a sub-interface and VLAN configured away from the G0/1 main interface. Then the switchport is plainly configured as a trunk (no native vlan).

interface GigabitEthernet0/1
 no nameif
 no security-level
 no ip address

interface GigabitEthernet0/1.30
 vlan 30
 nameif inside
 security-level 100
 ip address 172.30.3.4 255.255.255.0 

 

Troubleshoot Cisco IPSec Anti-Replay Error

Here's a Cisco link for troubleshooting an IPSec anti-replay errors. 

I was troubleshooting and observed a lot %CRYPTO-4-PKT_REPLAY_ERR: decrypt: replay check failed logs in a Cisco router configured for IPSec VPN:

*Sep 18 00:16:48 UTC: %CRYPTO-4-PKT_REPLAY_ERR: decrypt: replay check failed

        connection id=349, sequence number=4838

 

*Sep 18 00:26:35 UTC: %CRYPTO-4-PKT_REPLAY_ERR: decrypt: replay check failed

        connection id=349, sequence number=499055

 

*Sep 18 00:27:50 UTC: %CRYPTO-4-PKT_REPLAY_ERR: decrypt: replay check failed

        connection id=349, sequence number=591422

 

*Sep 18 00:30:58 UTC: %CRYPTO-4-PKT_REPLAY_ERR: decrypt: replay check failed

        connection id=349, sequence number=884929

 

*Sep 18 00:31:58 UTC: %CRYPTO-4-PKT_REPLAY_ERR: decrypt: replay check failed

        connection id=349, sequence number=969290

 

You can verify if the IPSec Security Association (SA) is from a legitimate peer using the show crypto ipsec sa | in peer|conn id command

2900#show crypto ipsec sa | in peer|conn id

   current_peer 110.17.9.8 port 500

   current_peer 175.23.30.6 port 19302

        conn id: 2349, flow_id: Onboard VPN:349, sibling_flags 80000040, crypto map: VPN-PROFILE-head-1

        conn id: 2350, flow_id: Onboard VPN:350, sibling_flags 80000040, crypto map: VPN-PROFILE-head-1

 

 

You can adjust the replay window size once the SA peer is confirmed and it's not doing a man-in-the-middle (MITM) attack. The default window size is 64 packets.

2900#show run all | inc crypto ipsec

crypto ipsec optional retry 300

crypto ipsec security-association lifetime kilobytes 4608000

crypto ipsec security-association lifetime seconds 3600

no crypto ipsec security-association replay disable

crypto ipsec security-association replay window-size 64

crypto ipsec security-association dummy

crypto ipsec transform-set default esp-aes esp-sha-hmac

crypto ipsec transform-set VPN-TSET esp-aes 256 esp-sha-hmac

crypto ipsec nat-transparency udp-encapsulation

crypto ipsec profile VPN-PROFILE

crypto ipsec profile default

Use the crypto ipsec security-association replay window-size <WINDOW SIZE> command. Start with 512 first, clear and observe the logs again.

2900#configure terminal

Enter configuration commands, one per line.  End with CNTL/Z.

2900(config)#crypto ipsec security-association replay window-size ? 

  1024  Window size of 1024

  128   Window size of 128

  256   Window size of 256

  512   Window size of 512

  64    Window size of 64 (default)

 

2900(config)#crypto ipsec security-association replay window-size 512

2900(config)#end

2900#write memory

Building configuration...

[OK]

 

2900#clear log
Clear logging buffer [confirm] 

 

 

I didn't observe the said logs afterwards (for almost more than 30 mins).

 

*Sep 18 01:08:37 UTC: %CLEAR-5-COUNTERS: Clear counter on all interfaces by admin on vty0 (202.7.6.8)
2900#
2900#show clock
*01:46:12.854 UTC Mon Sep 18 2023