Networking-Blog

My WordPress Blog

How to resolve high CPU utilization on routers

These are common symptoms of high CPU utilization:

  • High percentages in the output of the show processes cpu commandIf you have the show processes cpu command output or a show tech-support command from enable mode, display potential issues and fixes from your Cisco device by referring to the Output Interpreter tool.
    .
  • Input queue drops
  • Slow performance

    .
    .

  • Services on the router fail to respond, for instance: 
    • Slow response in Telnet or unable to Telnet to the router.
    • Slow or no response to the ping command.
    • The router does not send routing updates.
  • If the router is being overloaded with traffic or the traffic not taking the optimal switching path through the router, the issue can also be caused by the Blaster and Nachi worms.
  •    You can see what are the processes shooting your CPU utilization using following command:
  • You can also check your CPU history in graphical pattern using “show processes cpu history”  command to see High CPU spike.

    .
    .

    Resolution

     

    To resolve this issue, determine the accessibility of the router by performing these steps:

    1. Determine if you are able to issue the show commands on the router. If so, start collecting more information immediately using these show commands.
    2. Determine if the router is inaccessible and if this problem is reproducible. If so, power-cycle the router. Before reproducing the problem, configure the scheduler interval 500 command; this schedules low priority processes to run every 500 ms. This provides time for you to run some commands, even if CPU usage is at 100 percent. On Cisco 7200 and Cisco 7500 series routers, issue the scheduler allocate 3000 1000 command.
    3. Determine if the router shows symptoms of high CPU utilization at brief and unpredictable intervals. If so, periodically collect the output of the show processes cpu command. This shows if the high CPU utilization is caused by interrupts or by a certain process. Use this UNIX script. Based on the first findings, modify the script to collect data needed for further investigation of the issue.

DMVPN – IPSEC TRAFFIC – ACCESS-LIST

interface GigabitEthernet0/0
 description Internet Facing Interface
 ip address 194.#.133.12 255.255.255.0
 ip access-group secure_internet_connection in
!
ip access-list extended secure_internet_connection
 permit gre any host 194.#.133.12
 permit esp any host 194.#.133.12
 permit ahp any host 194.#.133.12
 permit udp any host 194.#.133.12 eq isakmp
 

ICMP TYPE NUMBERS

ICMP TYPE NUMBERS

The Internet Control Message Protocol (ICMP) has many messages that
are identified by a "type" field.

Type	Name					Reference
----	-------------------------		---------
  0	Echo Reply				 [RFC792]
  1	Unassigned				    [JBP]
  2	Unassigned				    [JBP]
  3	Destination Unreachable			 [RFC792]
  4	Source Quench			 	 [RFC792]
  5	Redirect				 [RFC792]
  6	Alternate Host Address			    [JBP]
  7	Unassigned				    [JBP]
  8	Echo Request			        [RFC792]
  9	Router Advertisement			[RFC1256]
 10	Router Selection			[RFC1256]
 11	Time Exceeded				 [RFC792]
 12	Parameter Problem			 [RFC792]
 13	Timestamp				 [RFC792]
 14	Timestamp Reply				 [RFC792]
 15	Information Request			 [RFC792]
 16	Information Reply			 [RFC792]
 17	Address Mask Request                     [RFC950]
 18	Address Mask Reply			 [RFC950]
 19	Reserved (for Security)			   [Solo]
 20-29	Reserved (for Robustness Experiment)	    [ZSu]
 30	Traceroute				[RFC1393]
 31	Datagram Conversion Error		[RFC1475]
 32     Mobile Host Redirect              [David Johnson]
 33     IPv6 Where-Are-You                 [Bill Simpson]
 34     IPv6 I-Am-Here                     [Bill Simpson]
 35     Mobile Registration Request        [Bill Simpson]
 36     Mobile Registration Reply          [Bill Simpson]
 37     Domain Name Request                     [Simpson]
 38     Domain Name Reply                       [Simpson]
 39     SKIP                                    [Markson]
 40     Photuris                                [Simpson]
 41-255 Reserved				    [JBP]

What’s the difference between a Layer 2 & Layer 3 switch

A L2 switch does switching only. This means that it uses MAC addresses to switch the packets from a port to the destination port (and only the destination port). It therefore maintains a MAC address table so that it can remember which ports have which MAC address associated.

A L3 switch also does switching exactly like a L2 switch. The L3 means that it has an identity from the L3 layer. Practically this means that a L3 switch is capable of having IP addresses and doing routing. For intra-VLAN communication, it uses the MAC address table. For extra-VLAN communication, it uses the IP routing table.

This is simple but you could say “Hey but my Cisco 2960 is a L2 switch and it has a VLAN interface with an IP !”. You are perfectly right but that VLAN interface cannot be used for IP routing since the switch does not maintain an IP routing table.

There two modes L2 – Transparent and  L3 – Routed.

L2 is transparent to the network and not seen as a router hop to connected devices.
L3 is a Layer-3 switch (routing switch) device that simply do routing only.

 

CISCO – REMOTE VPN CLIENT ESTABLISHED CONNECTION PORTS

Provide Support for the Cisco VPN Client

On a stateless firewall we need to add a rule-set facing the WAN connection to source port incoming
ipsec ports as no traffic will be inspected.

To provide support for this configuration, create the following protocol definitions:

Note The client computer must be configured as a SecureNat client.

Port number: 500 – IKE
Protocol type: UDP
Direction:  Receive

Port number: 4500 – NAT-T
Protocol type: UDP
Direction:  Receive

ip access-list extended INTERNET
remark REMOTE_VPN_CLIENT
permit udp any eq 500 any
remark NAT-T
permit udp any eq 4500 any
remark ESTABLISHED_TRAFFIC
permit tcp any any gt 1023 established
remark DENY_ALL
deny ip any any log

 

 

 

Difference between CIDR and VLSM?

Classless Inter-Domain Routing is based on variable-length subnet masking.

CIDR and VLSM both allow a portion of the IP address space to be recursively divided into subsequently smaller pieces. The difference is that with VLSM, the recursion is performed on the address space previously assigned to an organization and is invisible to the global Internet. CIDR, on the other hand, permits the recursive allocation of an address block by an Internet Registry to a high-level ISP, a mid-level ISP, a low-level ISP, and a private organization’s network. 

In CIDR , an IP network is represented by a prefix, which is an IP address and some indication of the length of the mask. Length means the number of left-most contiguous mask bits that are set to one.

So network 172.16.0.0 255.255.0.0 can be represented as 172.16.0.0/16. CIDR also depicts a more hierarchical Internet architecture, where each domain takes its IP addresses from a higher level.

This allows for the summarization of the domains to be done at the higher level. For example, if an ISP owns network 172.16.0.0/16, then the ISP can offer 172.16.1.0/24, 172.16.2.0/24, and so on to customers. Yet, when advertising to other providers, the ISP only needs to advertise 172.16.0.0/16.

Variable Length Subnet Masks (VLSM) allows you to use different masks for each subnet, thereby using address space efficiently.
Given the same network and requirements as in Sample Exercise 2 develop a subnetting scheme with the use of VLSM, given:

netA: must support 14 hosts
netB: must support 28 hosts
netC: must support 2 hosts
netD: must support 7 hosts
netE: must support 28 host

so the difference is CIDR is supernetting while VLSM is explained better with the following:
204.15.5.0 can be broken down into 5 different networks given the example requirement above into:

netB: 204.15.5.0/27 host address range 1 to 30
netE: 204.15.5.32/27 host address range 33 to 62
netA: 204.15.5.64/28 host address range 65 to 78
netD: 204.15.5.80/28 host address range 81 to 94
netC: 204.15.5.96/30 host address range 97 to 98

CISCO IOS SPOKE – IPSEC ISAKMP vs IPSEC ISAKMP PROFILE

IPSEC ISAKMP : CISCO REMOTE SITE :


crypto isakmp policy 1

encr aes
hash md5
authentication pre-share
group 2
lifetime 3600
!
crypto isakmp key t35t1n5!!! address 85.234.92.94
crypto isakmp keepalive 10
crypto isakmp nat keepalive 10
!
!
crypto ipsec transform-set 3G_IPSEC esp-3des esp-sha-hmac
!
!
!
!
crypto map mapping 1 ipsec-isakmp
set peer 85.234.92.94
set security-association lifetime seconds 86400
set security-association idle-time 86400
set transform-set 3G_IPSEC
set pfs group2
match address VPN
!
interface Dialer0
crypto map mapping
!
!
ip access-list extended VPN
permit ip 192.168.0.0 0.0.0.255 any
!
!

IPSEC ISAKMP PROFILE :

crypto isakmp policy 10
encr aes
hash md5
authentication pre-share
group 2
lifetime 3600
!
crypto keyring TEST_VPN
local-address dialer0
pre-shared-key address 85.234.92.94 key t35t1n5!!!
!
crypto isakmp profile TEST_VPN
keyring TEST_VPN
match identity address 85.234.92.94
local-address dialer0
!
crypto ipsec transform-set 3G_IPSEC esp-3des esp-sha-hmac
mode tunnel
!
crypto map 3G-1 10 ipsec-isakmp
set security-association lifetime seconds 900
set peer 85.234.92.94
set pfs group2
set transform-set 3G_IPSEC
set isakmp-profile TEST_VPN
match address TEST_VPN
!
!
ip access-list extended TEST_VPN
permit ip 192.168.0.0 0.0.0.255 any
!
interface Dialer0
crypto map 3G-1

DHCP Snooping on a Cisco Catalyst switch

The 3750 is configured with ip routing and a layer 3 interface on the subnet where the DHCP servers are located (10.0.10.0/24). VLAN 20 has been created on the 3750 with an interface ip address of 10.0.20.254/24.

All the DHCP server configuration and helper addresses were tested and working prior to implementing DHCP snooping to eliminate any doubt as to whether the DHCP snooping configuration is working or not.

So, let’s get started.

For DHCP snooping to work, you have to enable it globally. That is done with the following global configuration command:

Switch(config)#ip dhcp snooping

You also have to tell the switch which VLANs to monitor. In a production environment, this would be the client VLANs, not a transit VLAN that leads to the rest of the network.

This is done with the following command:

Switch(config)#ip dhcp snooping vlan 20

At this point DHCP snooping is configured and enabled. There are several default settings that can be modified later, but that can be dealt with after we verify things are working. Here is the basic show command to verify DHCP snooping is working (specifically the top few lines):

Switch#show ip dhcp snooping

Switch DHCP snooping is enabled
DHCP snooping is configured on following VLANs:

20
DHCP snooping is operational on following VLANs:

Insertion of option 82 is enabled
circuit-id format: vlan-mod-port

remote-id format: MAC
Option 82 on untrusted port is not allowed
Verification of hwaddr field is enabled
Verification of giaddr field is enabled
DHCP snooping trust/rate is configured on the following Interfaces:
Interface Trusted Rate limit (pps)

Once you verify DHCP snooping is working, you can verify DHCP lease information starts to populate the DHCP snooping binding table on the switch with the following command:

Switch#show ip dhcp snooping binding

AA:2C:DD:09:D1:CD 10.0.20.28 28781 dhcp-snooping 20 FastEthernet0/13
Total number of bindings: 1

If you have a DHCP server plugged into a switch with DHCP snooping enabled, or if you have a layer 2 LAN port connected to an upstream switch where the DHCP server resides, you’ll have to trust that port. To do this, enter the following command in interface configuration mode:

Switch(config-if)#ip dhcp snooping trust

In summary :
ip dhcp snooping
ip dhcp snooping vlan 20
ip dhcp snooping trust
!
show ip dhcp snooping
show ip dhcp snooping binding

Cisco Event Manager

Reconnect to ADSL without rebooting the router

Add the following configuration:

event manager applet atmDownUp
event none
action 1.0 cli command “enable”
action 1.1 cli command “conf t”
action 1.2 cli command “int atm0”
action 1.3 cli command “shut”
action 1.4 cli command “no shut”

You can trigger this applet with the “event manager run atmDownUp“

 

Schedule a recurring automatic reload

You can schedule it to run on a recurring schedule like this:
event manager applet routerReload
event timer cron cron-entry “05 0 * * *”
action 1 reload

F-VRF IPsec

VRF’s essentially provide path isolation functions. They provide separate routing tables,
forwarding tables, associated policies and in some cases management. This is network
virtualisation “lite”. Also, VRF’s do not rely on MPLS (which is a massive misconception).
It’s just a container. Not a protocol . MPLS can be used to provide transport in and out of said
container, but the two are not interdependent.

 Did you know you can configure a Cisco IOS based router to perform highly available
(active/standby) IPSec terminating within a VRF.
 IPSec terminating within a VRF.
It can save your company money and can reduce device
count. The concept is called
“FVRF IPsec”, or in English, Front door VRF IPSec.

 

Figure 1.0 shows a typical scenario (below)

 

 

This scenario is common amongst service-providers and geographic separation is normally
sold as part of the solution. Under the geographic separation scenario, EoMPLS would
normally be used to link the two IP networks together which would be now be apart.
L2 connectivity is a requirement for HSRP to function and EoMPLS satisfies this.
Specific routes can be added on to each ‘edge x’ device within the VRF to reach the
remote IPSec  endpoint .I feel I need to clarify what the IP transit situation would be
under geographic separation and my answer to that is it depends. As long as the
L2 pseudolink or LAN connection linked the two edge routers together,  it doesn’t matter
which ingress path is taken via packets.  This is termed ‘hot potato’ routing for those
who are uninitiated.

Presuming each node has IP transit to and from the internet, once HSRP has converged
and each node now is aware of its responsibilities, IKE and IPSec can now go through
their phases and agree terms.  Presuming again that all other configuration is correct,
which includes filters for identifying traffic to be encrypted etc, at this point we should
have a working topology.

It’s wise to lock down each FVRF ‘edge x’ interface using access-lists. In some environments
I’ve worked on, internet breakout ingresses and egresses a different part of the topology
and the ‘edge x’ devices have been shared between multiple customers. In these scenarios,
you can lock the FVRF interfaces down to the remote peer for IKE , IPSec and ICMP.

It will take time for IPSec to renegotiate and depending on the exact
failure scenario, routing may take time to converge.

Please find a config snippet below for one of the ‘edge x’ devices. 

Please note the interface on the 10.10.10.0/24 LAN is the interface I refer to below
called <FVRF_INTERFACE>.

 

ip sla 1
icmp-echo <IP Address of IPSec Endpoint> source-interface <FVRF Interface>
vrf <CustomerX>
timeout 15000
frequency 15
ip sla schedule 1 life forever start-time now
!
track 1 ip sla 1
!
crypto keyring CUST_X vrf custx
pre-shared-key address <IP Address of IPSec Endpoint> key <KEYSTRING>
!
crypto isakmp policy 1
encr aes
hash md5
authentication pre-share
group 2
lifetime 3600
!
crypto isakmp policy 10
encr 3des
hash md5
authentication pre-share
!
crypto isakmp profile CUST_X
vrf custx
keyring CUST_X
match identity address <IP Address and Mask for IPSec Endpoint> custx
!
!
crypto ipsec transform-set CUST_X esp-aes esp-md5-hmac
mode transport
!
crypto map CUST_X 10 ipsec-isakmp
set peer <IP Address for IPSec Endpoint>
set transform-set CUST_X
set isakmp-profile CUST_X
match address CUST_X
!
interface <FVRF_INTERFACE>
description **<INSERT_DESCRIPTION_HERE>**
encapsulation dot1Q xxx
ip vrf forwarding custx
ip address <FVRF_ADDRESS> <FVRF_MASK>
ip access-group CUST_X_FILTER_IN in
ip access-group CUST_X_FILTER_OUT out
no ip redirects
no ip unreachables
ip mtu 1500
standby 1 ip <HSRP_ADDRESS>
standby 1 priority 120
standby 1 preempt
standby 1 name HSRPCUSTX
standby 1 track 1 decrement 30
crypto map CUST_X redundancy HSRPCUSTX
!
!
ip route vrf custx <REMOTE_DEST> <MASK> 10.10.10.254 track 1
!
!
ip access-list extended CUST_X_FILTER_IN
permit ip <FVRF_ADDRESS><FVRF_MASK> host 224.0.0.2 !(HSRP)
permit esp host <REMOTE_IPSEC><FVRF_ADDRESS>
permit udp host <REMOTE_IPSEC><FVRF_ADDRESS> eq 500
deny   ip any any log
!
ip access-list extended CUST_X_FILTER_OUT
permit ip <FVRF_ADDRESS> host 224.0.0.2
permit esp <FVRF_ADDRESS> host <REMOTE_IPSEC>
permit udp <FVRF_ADDRESS> host <REMOTE_IPSEC> eq 500
deny   ip any any log