NetOps Bench
Can an AI agent fix a broken network without breaking something else? Each agent gets the same live, faulty networks as our human candidates: Cisco IOS and FortiGate consoles, user-style incident tickets and change requests. It is graded on what the network does at the end, on whether everything that worked still works, and on how it worked.
Associate: Branch office
10 public networks, 3 incidents and 1 change request each.
| Agent | Avg score | Pass rate | Tickets solved | Broke something | Avg commands | Runs |
|---|---|---|---|---|---|---|
| Reference solution (ceiling) | 100 | 100% | 100% | 0% | 45.9 | 10 |
| gemini-3.5-flash-lite | 55.7 | 17% | 21% | 17% | 99.5 | 6 |
| Does nothing (floor) | 17.5 | 0% | 0% | 0% | 0 | 10 |
Professional: Enterprise edge (FortiGate + Catalyst)
10 public networks, 4 incidents and 2 change requests each.
| Agent | Avg score | Pass rate | Tickets solved | Broke something | Avg commands | Runs |
|---|---|---|---|---|---|---|
| Reference solution (ceiling) | 100 | 100% | 100% | 0% | 78.1 | 10 |
| Does nothing (floor) | 28.2 | 0% | 0% | 0% | 0 | 10 |
Method
- Agents act through two tools: run a line in a device console, and submit. They see console output exactly as a person would.
- The networks are generated from public seeds, so every agent and every human taking a benchmark run faces the same faults.
- Score: 70% ticket requirements met, 15% no harm (everything that worked must still work), 15% method (checking before changing, verifying after).
- “Broke something” counts runs that left at least one previously working service broken: the risk that matters when an agent has write access.
- Runs use temperature 0 and up to 80 tool turns. The harness is the same simulator and grader as the human assessment.