Troubleshooting Slow Network — Finding the Bottleneck
Presenting Symptom
Users report that accessing cloud applications and internet websites is extremely slow, with frequent timeouts, while internal network resources remain responsive.
Network Context
The network is a small branch office with 50 users connected via a single Catalyst 2960 switch (IOS 15.2) to a Cisco 4321 ISR router (IOS 16.9) acting as the default gateway and connecting to the internet through a cable modem. The router runs NAT and a simple ACL. The ISP provides a 100 Mbps symmetric connection. The problem occurs intermittently but has become persistent over the past few days.
Diagnostic Steps
Check interface utilization and errors on the WAN interface
show interfaces GigabitEthernet0/0/0GigabitEthernet0/0/0 is up, line protocol is up
Hardware is ISR4321-2x1GE, address is aaaa.bbbb.cccc (bia aaaa.bbbb.cccc)
Internet address is 203.0.113.2/30
MTU 1500 bytes, BW 100000 Kbit/sec, DLY 100 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation HDLC, loopback not set
Keepalive set (10 sec)
Full Duplex, 100Mbps, link type is auto, media type is RJ45
output flow-control is unsupported, input flow-control is unsupported
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 1000 bits/sec, 2 packets/sec
5 minute output rate 1000 bits/sec, 2 packets/sec
123456 packets input, 98765432 bytes, 0 no buffer
Received 0 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 0 multicast, 0 pause input
123456 packets output, 98765432 bytes, 0 underruns
0 output errors, 0 collisions, 1 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped outCheck the 'txload' and 'rxload' values. If they are high (e.g., 255/255), the interface is saturated. Also look for input/output errors, CRC errors, or collisions that could indicate physical issues. In this case, loads are low and no errors, so the bottleneck is not the WAN link itself.
Check CPU utilization on the router
show processes cpu sorted | exclude 0.00CPU utilization for five seconds: 98%/0%; one minute: 95%; five minutes: 90% PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process 1 123456 12345 10000 45.00% 40.00% 35.00% 0 IP Input 2 234567 23456 10000 30.00% 25.00% 20.00% 0 CEF process 3 345678 34567 10000 10.00% 10.00% 10.00% 0 ARP Input 4 456789 45678 10000 5.00% 5.00% 5.00% 0 Net Background
High CPU utilization (e.g., >80%) indicates the router is overloaded. Look for processes consuming high CPU, such as 'IP Input' or 'CEF process'. This suggests the router is struggling to process packets, possibly due to a high rate of interrupts or a software feature.
Check for packet drops due to CPU overload
show interfaces GigabitEthernet0/0/0 | include dropsInput queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
If input queue drops are non-zero, packets are being dropped because the CPU cannot process them fast enough. This confirms CPU overload is causing packet loss.
Identify the cause of high CPU — check for excessive ACL logging or NAT translations
show ip nat translationsPro Inside global Inside local Outside local Outside global --- 203.0.113.2 192.168.1.10 8.8.8.8 8.8.8.8 --- 203.0.113.2 192.168.1.11 8.8.4.4 8.8.4.4 ... (many entries)
A large number of NAT translations (hundreds or thousands) can cause high CPU. Also check if there are many ACL log messages. In this scenario, the NAT table is small, so the issue is likely elsewhere.
Check for routing loops or suboptimal routing
show ip route 8.8.8.8Routing entry for 8.8.8.0/24
Known via "static", distance 1, metric 0
Routing Descriptor Blocks:
* 203.0.113.1, via GigabitEthernet0/0/0
Route metric is 0, traffic share count is 1Ensure the route is correct and points to the ISP next-hop. If there are multiple paths or floating statics, check for routing loops. In this case, the route is correct.
Check for excessive broadcast traffic or STP issues on the LAN
show interfaces | include broadcastReceived 0 broadcasts (0 IP multicasts) Received 0 broadcasts (0 IP multicasts) ...
If broadcast counts are very high (e.g., millions), it could indicate a broadcast storm. However, in this scenario, broadcasts are low.
Examine the ACL configuration for any deny statements that might be causing slow processing
show access-listsExtended IP access list 100
10 permit ip any any (123456 matches)
20 deny ip any any (0 matches)If there are many ACL entries with 'log' keyword, it can cause high CPU. In this case, the ACL is simple and not causing issues.
Check for interface errors on the LAN side
show interfaces Vlan1Vlan1 is up, line protocol is up
Hardware is EtherSVI, address is aaaa.bbbb.cccc (bia aaaa.bbbb.cccc)
Internet address is 192.168.1.1/24
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
...
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 output errors, 0 collisions, 0 interface resetsNo errors on the LAN interface, so the issue is not physical layer on the LAN.
Check for duplex mismatch on the WAN interface
show interfaces GigabitEthernet0/0/0 | include DuplexFull Duplex, 100Mbps, link type is auto, media type is RJ45
Duplex mismatch can cause slow performance. Both sides should be full duplex. Here it is full duplex, so not the issue.
Root Cause
The router's CPU is overloaded due to a high rate of IP interrupts caused by a large number of small packets (e.g., from a misconfigured application or a worm) that the router must process in software because CEF (Cisco Express Forwarding) is disabled or not functioning correctly. The 'show processes cpu' output shows high CPU utilization by 'IP Input' process, and input queue drops are present on the WAN interface, indicating the CPU cannot keep up with packet processing.
Resolution
Verification
After applying the fix, verify CPU utilization drops and input queue drops cease. 1. Check CPU utilization: Router# show processes cpu sorted | exclude 0.00 Expected: CPU utilization for five seconds: 25%/0%; one minute: 20%; five minutes: 18% 2. Check interface for drops: Router# show interfaces GigabitEthernet0/0/0 | include drops Expected: Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0 3. Confirm CEF is active: Router# show ip cef Expected: IPv4 CEF is enabled
Prevention
1. Always enable CEF on routers that support it to ensure hardware-based switching and reduce CPU load. 2. Implement traffic shaping or policing to limit the rate of small packets from potentially misbehaving hosts. 3. Use network monitoring tools to baseline CPU utilization and interface loads, and set alerts for anomalies.
CCNA Exam Relevance
On the CCNA 200-301 exam, this scenario tests the candidate's ability to troubleshoot network performance issues using show commands. The exam may present a multiple-choice question asking for the most likely cause of slow internet performance given high CPU utilization and input queue drops. Candidates must know that CEF is a key feature to offload switching and that 'show processes cpu' and 'show interfaces' are essential for identifying CPU overload and packet drops.
Exam Tips
Memorize that high CPU utilization by 'IP Input' process often indicates software switching due to CEF being disabled or overloaded.
Remember that input queue drops on an interface are a strong indicator that the router's CPU cannot process packets fast enough.
Know that 'ip cef' is the command to enable Cisco Express Forwarding, which is enabled by default on most platforms but can be disabled accidentally.
Commands Used in This Scenario
show access-lists
Displays all configured access control lists (ACLs) on the device, including their entries and match counters, used to verify ACL configuration and traffic filtering.
show interfaces
Displays detailed status and statistics for all interfaces or a specific interface, used to verify interface operational state, errors, and performance.
show processes cpu
Displays CPU utilization statistics for all processes running on the Cisco IOS device, used to identify processes consuming excessive CPU and diagnose performance issues.
Test Your CCNA Knowledge
Practice with scenario-based questions to prepare for the CCNA 200-301 exam.
Practice CCNA Questions