Chaos Engineering and Resilience Testing

Chaos engineering is the discipline of experimenting on a system to build confidence in its capability to withstand turbulent conditions in production. It is not about breaking things randomly but about systematically injecting failures to uncover weaknesses before they cause user-facing outages. The practice originated at Netflix with the Chaos Monkey tool, which randomly terminated virtual machine instances to ensure the system could survive instance failures.

The AZ-305 exam tests your ability to design highly available and resilient solutions. Simply stating that you will deploy across availability zones or use geo-redundancy is not enough; you must also demonstrate how you will validate that those designs actually work. Chaos engineering provides a method to test recovery mechanisms, failover logic, and performance degradation behaviors. The exam expects you to know:

Fault Injection: The deliberate introduction of errors into a system. Examples include:

Steady State Hypothesis: Before running a chaos experiment, you define what 'normal' looks like (e.g., response time < 200ms, error rate < 0.1%). The experiment then verifies that the system remains in (or quickly returns to) that steady state despite the injected fault.

Blast Radius: The scope of the experiment. In production, you start with a small blast radius (e.g., one instance in a load-balanced set) and gradually expand as confidence grows. Never start with a full-region failure.

Azure Chaos Studio is a managed service that enables you to run chaos experiments on Azure resources. It provides: - Targets: Resources you want to test (VMs, AKS clusters, App Services, etc.). - Faults: Predefined failure actions (e.g., shutdown VM, network latency, CPU pressure). - Experiment: A JSON-based definition that specifies the sequence of faults, targets, and duration. - Agent-based vs. Agentless: Some faults require an agent installed on the VM (e.g., CPU pressure), while others (like VM shutdown) are agentless and use Azure Resource Manager.

Example Experiment Definition (JSON snippet):

{
  "properties": {
    "steps": [
      {
        "name": "Shutdown VM",
        "branches": [
          {
            "actions": [
              {
                "type": "continuous",
                "name": "urn:csci:microsoft:virtualMachine:shutdown/1.0",
                "parameters": [
                  {
                    "name": "duration",
                    "value": "PT2M"
                  }
                ]
              }
            ]
          }
        ]
      }
    ]
  }
}

Azure CLI to create a chaos experiment:

az chaos experiment create \
  --name myExperiment \
  --resource-group myRG \
  --location eastus \
  --experiment-file experiment.json

Start the experiment:

az chaos experiment start \
  --name myExperiment \
  --resource-group myRG

Check experiment status:

az chaos experiment show \
  --name myExperiment \
  --resource-group myRG \
  --query "properties.status"

A large e-commerce company runs its application on Azure VMs in an availability set behind a load balancer. During Black Friday, they cannot afford any downtime. They use Chaos Engineering to test their resilience ahead of time. They run experiments that simulate a VM crash, a network latency spike, and a database replica failure. They discover that when a VM fails, the load balancer health probe takes 60 seconds to mark it as unhealthy, causing a 60-second window of failed requests. They reduce the probe interval from 30 seconds to 5 seconds and the unhealthy threshold from 2 to 1, cutting the failover time to under 10 seconds. They also find that their database read replicas are not automatically promoted when the primary fails, so they implement auto-failover groups. These changes ensure the platform handles Black Friday traffic without interruption.

A SaaS provider uses Azure Traffic Manager to route traffic to multiple regions. They use Chaos Studio to simulate a full region outage by shutting down all VMs in one region. They discover that Traffic Manager's DNS TTL of 300 seconds causes clients to continue sending traffic to the failed region for up to 5 minutes, resulting in a poor user experience. They reduce the TTL to 60 seconds and implement client-side retry logic. They also find that their database geo-replication lag is often above 5 minutes, exceeding their RPO of 1 minute. They switch to active geo-replication with a commit lag of under 1 minute. The chaos experiment directly drives these architectural improvements.