Bulkhead and Circuit Breaker Patterns

Resilience patterns help applications handle failures gracefully. The Bulkhead pattern isolates components into separate pools so that a failure in one does not cascade. The Circuit Breaker pattern detects failures and prevents repeated attempts that are likely to fail, allowing the system to recover.

The term comes from shipbuilding: a ship's hull is divided into watertight compartments (bulkheads) so that if one compartment floods, the others keep the ship afloat. In software, the Bulkhead pattern partitions resources (thread pools, connections, memory) into isolated groups. Each group serves a specific workload, and if one group fails, the others remain unaffected.

Key Components: - Partitions: Logical or physical separation of resources. In Azure, partitions can be:

- Separate Azure App Service plans for different microservices - Separate Azure SQL databases for different tenants - Separate Service Bus namespaces for different message streams - Resource Pools: Each partition has its own thread pool, connection pool, or memory quota. - Failure Isolation: A failure in one partition does not consume resources from another.

How It Works: 1. The system identifies workloads that should be isolated (e.g., user-facing vs. batch processing). 2. Each workload is assigned its own resource pool. For example, an Azure Function App with a dedicated App Service plan for critical APIs, and a separate plan for background jobs. 3. If the batch processing pool experiences a memory leak, it crashes only that App Service plan. The critical APIs continue running on their own plan.

Configuration in Azure: - Azure App Service: Create separate App Service plans for different tiers. Each plan has its own set of VMs and resource limits. - Azure Service Bus: Use separate namespaces or topics for high-priority vs. low-priority messages. This prevents a flood of low-priority messages from starving high-priority ones. - Azure SQL Database: Use elastic pools or separate databases for different tenants (multi-tenant isolation).

Default Values and Timers: - No specific timers; it's a design pattern. However, you set resource limits per partition:

- App Service Plan: max instances, max burst instances - Service Bus: namespace throughput units, message size limits - SQL Database: DTU/vCore limits per database or elastic pool

Verification: - Monitor partition-specific metrics in Azure Monitor: CPU, memory, requests per partition. - Use Application Insights to trace failures per component.

A circuit breaker wraps calls to an external service or resource. It monitors for failures and, when the failure rate exceeds a threshold, it 'opens' the circuit, causing subsequent calls to fail immediately without attempting the actual call. After a timeout, it allows a limited number of test calls (half-open state) to see if the service has recovered. If successful, the circuit closes again.

States: - Closed: Normal operation. Calls pass through. Failure count is tracked. - Open: Calls fail immediately. After a timeout, transitions to half-open. - Half-Open: A limited number of calls are allowed to test recovery. If successful, close; if failure, back to open.

Key Components: - Failure Threshold: Number of consecutive failures or failure rate (e.g., 5 failures within 10 seconds). - Timeout (Reset Timeout): Duration the circuit stays open before transitioning to half-open (e.g., 30 seconds). - Half-Open Max Calls: Number of test calls allowed in half-open state (e.g., 1). - Success Threshold: Number of successful test calls to close the circuit (e.g., 2 consecutive successes).

How It Works: 1. In closed state, each call increments a failure counter on timeout or exception. 2. When failure counter reaches threshold (e.g., 5), circuit opens. 3. During open state, all calls immediately throw an exception (e.g., CircuitBreakerOpenException). 4. After reset timeout (e.g., 30 seconds), circuit transitions to half-open. 5. In half-open, a limited number of calls are permitted. If they succeed, circuit closes and counters reset. If any fails, circuit reopens and timer restarts.

Implementation in Azure: - Polly Library: A .NET resilience library that provides circuit breaker policies. Example:

var circuitBreakerPolicy = Policy
    .Handle<HttpRequestException>()
    .CircuitBreakerAsync(
        exceptionsAllowedBeforeBreaking: 5,
        durationOfBreak: TimeSpan.FromSeconds(30),
        onBreak: (ex, duration) => { /* log */ },
        onReset: () => { /* log */ },
        onHalfOpen: () => { /* log */ });

Default Values and Timers: - Common defaults: 5 failures before open, 30-second break, 1 call in half-open. - These are configurable; no Azure service enforces specific values.

Verification: - Log circuit state transitions using onBreak, onReset, onHalfOpen callbacks. - Monitor circuit breaker metrics: open count, half-open count, failure rate. - Use Application Insights to track exceptions and circuit state changes.

A SaaS provider hosts a multi-tenant application on Azure. Each tenant has its own database (Bulkhead pattern). The application uses Azure App Service with separate App Service plans for each tenant tier (free, standard, premium). This ensures that a noisy neighbor tenant on the free tier cannot consume CPU or memory from premium tenants. Additionally, the application calls a third-party email service. To protect against email service outages, the team implements a circuit breaker using Polly. They configure 5 failures before opening, 30-second break, and 1 half-open call. In production, the email service experienced intermittent failures. The circuit breaker prevented the application from hanging for 30 seconds on each request; instead, users saw a 'email temporarily unavailable' message. The team monitors circuit state using Application Insights and sets an alert if the circuit remains open for more than 5 minutes, triggering a manual switch to a backup email provider.

A financial trading platform uses Azure Service Bus to decouple order ingestion from processing. To ensure high-priority orders (e.g., large trades) are not delayed by low-priority market data, they use separate Service Bus topics (Bulkhead). Each topic has its own throughput units. The processing services (Azure Functions) have dedicated App Service plans per priority. For the order processing function, calls to a risk analysis service are wrapped in a circuit breaker. The risk service is critical but occasionally slow. The circuit breaker is configured with 3 failures (aggressive) and a 10-second break (short) because financial transactions need quick fallback. If the circuit opens, the function uses a cached risk score. This setup ensures that a risk service outage does not block order processing entirely.