Monitoring and Troubleshooting Automation

Monitoring and troubleshooting are the two tasks that keep a network healthy. Monitoring means constantly watching the network’s vital signs—like CPU usage on a switch, temperature in a rack, or the number of packets dropped on a link. Troubleshooting means figuring out why something went wrong and fixing it. Automating these tasks means training the network equipment to do the watching and the fixing by itself, using scripts and built-in rules.

Before automation, a network engineer had to log into every device manually, run ‘show’ commands, and compare output by eye. That process was slow, error-prone, and impossible at scale. A data centre might have hundreds of switches and thousands of virtual servers. A human cannot check every interface on every device every second—but a script can.

The exam objective covers three main approaches: Embedded Event Manager (EEM), scripts, and telemetry.

EEM is a feature built into Cisco’s operating system (IOS-XE and NX-OS). It works like a ‘if this, then that’ rule set directly on the device. You define an event—for example, ‘the interface goes down’—and an action—‘send a syslog message and reload the module.’ EEM runs inside the device’s own software, so it acts immediately without relying on an external server. Common events for EEM include timer-based (run a check every 5 minutes), syslog-pattern-based (when a specific log message appears), and counter-based (when the CPU hits 90% for 30 seconds).

Scripts are more flexible. You write a program—often in Python—that runs on an external server (or directly on the device via ‘guest shell’ in NX-OS). The script can connect to multiple devices, parse their output with tools like regex, and make decisions based on the data. For example, a Python script could log into every switch in a pod once a minute, check the temperature sensor, and if any reading is over 50 degrees Celsius, send an email to the operations team and a Webex message to the on-call engineer. Scripts are powerful because they can combine data from many sources and do complex logic.

Telemetry is the most modern approach. Instead of the engineer polling devices every few minutes (which wastes CPU and bandwidth), the device pushes data to a collector whenever something changes or on a precise schedule. This is called ‘model-driven telemetry’ (MDT). The device sends structured data—usually in JSON or GPB format—to a receiver like Cisco’s ‘Data Center Network Manager’ (DCNM) or an open-source tool like InfluxDB. Telemetry gives you real-time, granular visibility. For instance, an interface buffer utilisation metric can be streamed every second, allowing you to spot micro-bursts that would never show up in a 5-minute poll.

Why does this matter for the exam? Cisco 350-601 expects you to know when to use each tool. For a simple, device-local action like ‘if this interface flaps, shut it down,’ the answer is EEM. For a multi-device check across the whole fabric, the answer is a script. For continuous, low-latency monitoring, the answer is telemetry. The exam will present scenarios—‘you need to watch 40 leaf switches for buffer drops and alert within 2 seconds’—and you must pick the right technology.

Another key concept is the ‘network controller’ role. In a data centre, you might use Cisco ACI (Application Centric Infrastructure) or NDO (Nexus Dashboard Orchestrator). These controllers have built-in monitoring and troubleshooting automation. For example, ACI’s ‘fault’ system automatically raises alerts for misconfigurations and can even suggest remediation scripts. The exam tests whether you understand how controller-based automation differs from device-by-device EEM or scripts—controllers centralise the logic and enforce policies from one place.

Finally, the exam covers ‘remediation automation’—using scripts or EEM to not just detect a problem, but fix it automatically. This might mean triggering a port bounce if a link has too many CRC errors, or rerouting traffic if a temperature sensor goes critical. You need to know that automatic remediation is powerful but risky, so Cisco recommends ‘confirm’ steps or approval workflows in production.

In summary, monitoring and troubleshooting automation replaces manual, slow, human-driven checks with fast, consistent, machine-driven actions. EEM handles simple triggers on a single device, scripts orchestrate complex logic across many devices, and telemetry streams live data for proactive detection. The 350-601 exam tests your ability to match the right automation method to the right problem scenario.

Consider a large financial services company running a Cisco ACI fabric with 60 leaf switches and 8 spine switches, connecting thousands of virtual machines running trading applications. The network operations team has two goals: never miss a performance anomaly, and fix any failure within 60 seconds.

First, they set up model-driven telemetry from every leaf switch. They configure the ‘interface-statistics’ sensor group to stream inbound and outbound packet counters, queue drops, and CRC errors every 2 seconds. The data flows to a central collector running the open-source Telegraf agent and into InfluxDB, a time-series database. A Grafana dashboard displays real-time charts. The team can see when a specific port starts dropping packets long before users complain.

Second, they write Python scripts that run on a management server. One script connects to the APIC (Application Policy Infrastructure Controller) via its REST API every 30 seconds. It fetches the ‘fault summary’ and checks for any new ‘critical’ faults. If a fault indicates ‘link down on interface eth1/1 on leaf-12’, the script automatically opens a ticket in ServiceNow and sends a Webex message to the on-call engineer with the exact interface name and the time since failure.

Third, they deploy an EEM policy on each leaf switch. The policy watches for the event ‘_syslog_msg .* LINK-UPDOWN.*INTERFACE.*eth1/11.*down’. When that pattern matches, EEM executes a ‘show tech-support’ command, saves the output to a file on the bootflash, and sends a simple SNMP trap to the network management system. This gives them a forensic snapshot without any manual login.

The real power comes from combining these tools. One morning, telemetry shows that leaf-05’s buffer utilisation is hitting 100% every 10 seconds for 200-millisecond bursts. The Python script detects the upward trend (comparing current values to a moving average), and triggers an EEM action that temporarily changes the queue-limit on the affected interface to absorb the burst. The trading application never sees latency, and the team is alerted only after the automated mitigation, not before.

Another common scenario is software upgrade troubleshooting. The team writes a Python script that, before an upgrade, takes a ‘snapshot’ of all interface statuses, OSPF neighbours, and BGP sessions. After the upgrade, it runs again and compares the two sets of data. Any mismatch—like an interface that stayed down after reboot—is highlighted in a report. This saves hours of manual comparison.

The key lesson for the exam: real-world automation is not one tool. It is a layered approach—telemetry for real-time visibility, scripts for orchestration and analysis, and EEM for immediate, local reaction. The scenario above demonstrates each layer and how they complement each other.