# Durable Functions: Fan-Out, Chaining, Human Interaction

> Chapter 14 of the Courseiva AZ-204 curriculum — https://courseiva.com/learn/az-204/durable-functions-204

**Official objective:** 1.1 — Compute

## Introduction

This chapter covers Durable Functions, an extension of Azure Functions that enables stateful workflows in a serverless environment. You will learn three core patterns: function chaining (sequential execution), fan-out/fan-in (parallel execution), and human interaction (asynchronous approval workflows). These patterns are heavily tested on the AZ-204 exam, appearing in approximately 15–20% of questions related to compute and serverless. Understanding the internal mechanisms, default timeouts, and storage dependencies is critical for both the exam and real-world implementation.

## Assembly Line with Parallel Workstations and Manager

Imagine a car assembly line. The line has a main conveyor belt (the orchestrator function). When a car chassis arrives, the conveyor belt moves it to a workstation where the engine is installed (function chaining). But sometimes, you need to install all four wheels simultaneously — that's fan-out. The conveyor belt stops, and four robotic arms (activity functions) each install one wheel in parallel. The conveyor belt waits until all four arms report completion before moving on. Now consider human interaction: if a part is missing, the conveyor belt pauses and sends a notification to a human supervisor. The supervisor has a limited time (e.g., 30 minutes) to respond. If they respond with 'use substitute part', the line continues; if they don't respond, the line escalates to a manager. The conveyor belt itself never installs parts — it only orchestrates the workflow. The robotic arms are stateless and can be scaled independently. The conveyor belt maintains state (which chassis is where, which steps completed) in a durable storage — in Durable Functions, this is the Azure Storage queue and table. The belt's position is like the orchestration history — it can be replayed from any point if the belt crashes. This exactly mirrors Durable Functions: orchestrators coordinate, activities do work, fan-out/fan-in runs parallel tasks, and human interaction uses durable timers and external events.

## Core explanation

### What Are Durable Functions and Why Do They Exist?

Durable Functions is an extension of Azure Functions that allows you to write stateful functions in a serverless compute environment. The core problem it solves is that standard Azure Functions are stateless — they execute in response to an event and then terminate, with no built-in mechanism to maintain state across multiple invocations or to orchestrate complex workflows that involve waiting, retries, and parallel execution. Durable Functions addresses this by introducing a durable orchestration framework that automatically manages state, checkpoints, and replays.

### How It Works Internally — The Orchestrator and Activity Functions

Durable Functions introduces two main function types: **orchestrator functions** and **activity functions**. Orchestrator functions define the workflow using procedural code (e.g., C# async methods) and coordinate the execution of activity functions. Activity functions are the actual units of work — they perform one specific action, such as calling an external API, writing to a database, or processing a file.

The orchestrator function is deterministic: every time it runs, it must produce the same sequence of calls to activity functions given the same input. This determinism is enforced by the Durable Functions runtime through a mechanism called **replay**. The orchestrator function's execution history is persisted in Azure Storage (specifically in a table named `DurableFunctionsHistory`). If the orchestrator is interrupted (e.g., due to a host restart), it can be replayed from the beginning by reading the history and skipping already-completed steps. This ensures that the orchestrator is resilient to failures without requiring explicit checkpointing code.

### Key Components, Values, Defaults, and Timers

- **Task Hubs**: A logical container for Durable Functions resources within a storage account. Each task hub uses separate queues, tables, and containers. Default name is the function app name.
- **Storage Account**: Required. Durable Functions uses Azure Storage queues for message delivery, tables for orchestration history, and blobs for leases and large messages.
- **Control Queue**: A queue that holds messages for orchestrator functions. Each task hub has one or more control queues. Default number of partitions is 4, configurable via `maxConcurrentTaskOrchestrationWorkItems`.
- **Work-Item Queue**: A queue for activity functions. Each task hub has one work-item queue.
- **Orchestration Instance ID**: A unique identifier for each orchestration execution. You can provide it when starting an orchestration, or let the system auto-generate a GUID.
- **Durable Timer**: Created using `CreateTimer` in the orchestrator. Timers are durable — they survive host restarts. The minimum timer resolution is 1 second, but practical minimum is about 1 minute due to storage latency.
- **Retry Policies**: Activity functions can be configured with automatic retry policies. Default: no retries. You can specify `MaxNumberOfAttempts`, `FirstRetryInterval` (default 1 second), `BackoffCoefficient` (default 2), and `MaxRetryInterval`.
- **Human Interaction Timeout**: Typically implemented using a durable timer that waits for an external event. The external event is sent via `RaiseEventAsync`. Common timeout values: 24 hours, 7 days. The maximum timer duration is limited by the storage account's queue message TTL (7 days by default, configurable up to infinite).

### Configuration and Verification Commands

To enable Durable Functions in a function app, you must add the `DurableTask` extension. In the `host.json` file, you can configure the storage connection string and task hub name:

```json
{
  "version": "2.0",
  "extensions": {
    "durableTask": {
      "hubName": "MyTaskHub",
      "storageProvider": {
        "connectionStringName": "DurableFunctionsStorage",
        "partitionCount": 4
      }
    }
  }
}
```

To verify correct configuration, you can monitor the storage account: check that queues (e.g., `MyTaskHub-control-00`, `MyTaskHub-workitems`) and tables (e.g., `DurableFunctionsHistory`, `DurableFunctionsInstances`) are created. Use Azure Storage Explorer or CLI:

```bash
az storage table list --connection-string "<connection-string>" --query "[?contains(name, 'DurableFunctions')]"
```

### How Durable Functions Interacts with Related Technologies

- **Azure Functions**: Durable Functions is built on top of Azure Functions. All triggers and bindings available in Azure Functions are also available in activity functions, but orchestrator functions have restrictions: they cannot use I/O directly (e.g., HTTP calls, database queries) — they must use activity functions for any I/O.
- **Azure Storage**: The default storage provider. For production, consider using the **Netherite** provider (which uses Event Hubs) or the **Microsoft SQL Server** provider for better performance and scalability.
- **Azure Logic Apps**: Both can orchestrate workflows, but Durable Functions is code-first, more flexible, and cheaper for high-volume scenarios. Logic Apps is designer-driven and better for simple integrations.
- **Azure Event Grid**: Can be used to trigger orchestrations or receive notifications (e.g., when a human interaction step completes).

### Pattern: Function Chaining

Function chaining executes a sequence of activities in a specific order. The orchestrator calls each activity one after another, and the output of one can be passed as input to the next. This is implemented using `await` in C#:

```csharp
[FunctionName("Chaining")]
public static async Task<List<string>> RunOrchestrator(
    [OrchestrationTrigger] IDurableOrchestrationContext context)
{
    var input = context.GetInput<string>();
    var result1 = await context.CallActivityAsync<string>("Activity1", input);
    var result2 = await context.CallActivityAsync<string>("Activity2", result1);
    var result3 = await context.CallActivityAsync<string>("Activity3", result2);
    return new List<string> { result1, result2, result3 };
}
```

The orchestrator replays after each activity completes. If the host crashes after Activity1 but before Activity2, the orchestrator replays from the start, sees that Activity1 has already completed (via history), and skips directly to calling Activity2.

### Pattern: Fan-Out/Fan-In

Fan-out/fan-in executes multiple activities in parallel and then aggregates their results. This is done using `Task.WhenAll`:

```csharp
[FunctionName("FanOutFanIn")]
public static async Task<int> RunOrchestrator(
    [OrchestrationTrigger] IDurableOrchestrationContext context)
{
    var workBatch = await context.CallActivityAsync<string[]>("GetWorkBatch", null);
    var tasks = new Task<int>[workBatch.Length];
    for (int i = 0; i < workBatch.Length; i++)
    {
        tasks[i] = context.CallActivityAsync<int>("ProcessItem", workBatch[i]);
    }
    var results = await Task.WhenAll(tasks);
    var total = results.Sum();
    return total;
}
```

Key points: The orchestrator issues all activity calls immediately (fan-out) and then awaits all of them (fan-in). The number of concurrent activities is limited by the `maxConcurrentTaskActivityWorkItems` setting (default 10 per host). For large fan-outs, consider batching or using a sub-orchestrator pattern.

### Pattern: Human Interaction

Human interaction involves waiting for an external event (e.g., approval) with a timeout. The orchestrator creates a durable timer and waits for either the timer to expire or an external event to be raised:

```csharp
[FunctionName("ApprovalWorkflow")]
public static async Task RunOrchestrator(
    [OrchestrationTrigger] IDurableOrchestrationContext context)
{
    await context.CallActivityAsync("RequestApproval", null);
    using (var timeoutCts = new CancellationTokenSource())
    {
        DateTime deadline = context.CurrentUtcDateTime.AddHours(24);
        Task timeoutTask = context.CreateTimer(deadline, timeoutCts.Token);
        Task approvalTask = context.WaitForExternalEvent("ApprovalResult");
        var winner = await Task.WhenAny(timeoutTask, approvalTask);
        if (winner == approvalTask)
        {
            // Approval received
        }
        else
        {
            // Timeout occurred
            await context.CallActivityAsync("Escalate", null);
        }
    }
}
```

The external event is raised from another function (e.g., an HTTP-triggered function) using:

```csharp
await client.RaiseEventAsync(instanceId, "ApprovalResult", approvalValue);
```

Important: The `WaitForExternalEvent` call is durable — if the host restarts while waiting, it will resume waiting after replay. The timer is also durable.

### Error Handling and Retries

Activity functions can throw exceptions. The orchestrator can catch them and implement retry logic using `CallActivityWithRetryAsync`:

```csharp
try
{
    await context.CallActivityWithRetryAsync("FlakyActivity", 
        new RetryOptions(TimeSpan.FromSeconds(5), 3)
        {
            BackoffCoefficient = 2.0,
            MaxRetryInterval = TimeSpan.FromMinutes(1)
        }, input);
}
catch (Exception ex)
{
    // Handle after retries exhausted
}
```

### Performance and Scaling

- **Throughput**: Durable Functions can handle thousands of orchestrations per second, but performance depends on the storage provider. The default Azure Storage provider is suitable for most workloads; Netherite can achieve 10x higher throughput.
- **Scale**: The function app scales out based on the number of messages in the control queues. Each host processes one message at a time per control queue partition. Increase partition count for higher concurrency.
- **Latency**: Orchestrator replay adds overhead. For very low-latency requirements, consider minimizing the number of activity calls or using the Netherite provider.

### Common Pitfalls

- **Non-deterministic code**: Using `DateTime.Now`, `Guid.NewGuid()`, or random numbers directly in orchestrator functions will cause replay errors. Use `context.CurrentUtcDateTime` and `context.NewGuid()` instead.
- **Infinite loops**: Orchestrators must terminate. Ensure that loops have proper exit conditions.
- **Large payloads**: Passing large inputs/outputs between activities can cause storage pressure. Use external storage references (e.g., blob URLs) for large data.
- **Timer accuracy**: Durable timers are not real-time; they are checked periodically. The default polling interval is 1 second, but actual resolution may be coarser under load.

## Real-world context

### Enterprise Scenario 1: Order Processing Pipeline

A large e-commerce company uses Durable Functions to process orders from submission to fulfillment. The orchestration chains activities: validate payment, check inventory, reserve stock, ship, and send confirmation. Fan-out is used to check inventory across multiple warehouses in parallel. Human interaction is required when a payment fails — an email is sent to the customer with a link to update payment, and if not completed within 48 hours, the order is cancelled and inventory is released. In production, this system handles 10,000 orders per hour. The key configuration is setting the `maxConcurrentTaskActivityWorkItems` to 50 per host to balance throughput and resource usage. A common misconfiguration is using too few control queue partitions (default 4), which limits parallelism — increasing to 16 improved throughput by 3x.

### Enterprise Scenario 2: Document Approval Workflow

A financial services firm uses Durable Functions for document approval. When a document is uploaded, an orchestrator starts: first, it runs an activity to classify the document, then fans out to multiple reviewers (each reviewer is an activity that sends an email with an approval link). The orchestrator waits for all reviewers to approve or reject. If any reviewer rejects, the entire workflow is terminated. A durable timer enforces a 7-day deadline; after that, the document is escalated to a manager. The system processes 500 documents per day. The main challenge is managing the large number of concurrent timers — each timer creates a storage queue message. The team had to increase the storage account's queue capacity and use the Netherite provider for better timer performance.

### Enterprise Scenario 3: Data Pipeline with Fan-Out

A media company processes video uploads: transcode to multiple formats (fan-out), generate thumbnails, and extract metadata. Each activity runs in parallel. After all complete, the orchestrator aggregates results and updates a database. The fan-out can involve 10–20 parallel activities. The orchestrator uses `CallActivityWithRetryAsync` for each transcode activity with 3 retries and exponential backoff. In production, they noticed that if one activity fails and exhausts retries, the entire orchestration fails. They implemented a custom error-handling pattern: catch exceptions and store partial results, then continue. This required careful design of the orchestrator's state management.

## Exam focus

### Exactly What AZ-204 Tests on This Topic

AZ-204 objective 1.1 covers "Develop Azure compute solutions" and includes Durable Functions patterns. Specifically, you must know:
- **Function chaining**: How to sequence activities and pass data between them.
- **Fan-out/fan-in**: How to execute parallel tasks and aggregate results.
- **Human interaction**: How to implement durable timers and external events.
- **Orchestrator constraints**: Determinism, no I/O, use of `CurrentUtcDateTime` and `NewGuid`.
- **Storage dependencies**: Azure Storage requirements and configuration.

### Common Wrong Answers and Why Candidates Choose Them

1. **Using `DateTime.Now` in orchestrator**: Candidates think it's fine because it's C#. Wrong — the orchestrator replays, so `DateTime.Now` would return different values each replay, causing non-determinism. Use `context.CurrentUtcDateTime`.
2. **Using `HttpClient` directly in orchestrator**: Candidates think they can make HTTP calls. Wrong — orchestrators must not perform I/O. Use an activity function to make the HTTP call.
3. **Believing fan-out/fan-in requires `Task.WhenAll` with a list of tasks**: While correct, many think they must use `Task.WhenAll` only. Actually, you can use any pattern that awaits multiple tasks. The key is that the orchestrator issues all calls before awaiting.
4. **Thinking human interaction requires polling**: Candidates propose a polling loop in the orchestrator. Wrong — use `WaitForExternalEvent` with a durable timer for efficiency and reliability.

### Specific Numbers and Terms That Appear on the Exam

- **Default partition count for control queues**: 4
- **Default retry policy**: `FirstRetryInterval` = 1 second, `BackoffCoefficient` = 2, `MaxRetryInterval` = unlimited, `MaxNumberOfAttempts` = no limit (but you must specify).
- **Maximum timer duration**: Limited by storage queue message TTL (default 7 days, configurable).
- **Task hub name**: Must be unique within a storage account.
- **Instance ID**: Can be provided or auto-generated (GUID).

### Edge Cases and Exceptions the Exam Loves to Test

- **Orchestrator replay and non-deterministic code**: The exam will present code using `Random` or `DateTime.Now` and ask if it's valid.
- **Large fan-out**: The exam may ask about limits — there's no hard limit, but performance degrades. Use batching.
- **Timer accuracy**: Durable timers are not real-time; they have a resolution of about 1 second but can be delayed under load.
- **Storage provider**: The default is Azure Storage; Netherite and MSSQL are alternatives. The exam may ask which provider to use for higher throughput.

### How to Eliminate Wrong Answers Using the Underlying Mechanism

- If an answer suggests using `Thread.Sleep` in an orchestrator, eliminate it — orchestrators must not block threads.
- If an answer suggests using `context.CallActivity` inside a `Parallel.ForEach`, consider that the orchestrator must await all tasks — `Task.WhenAll` is preferred.
- If an answer suggests using `context.CallActivity` for a long-running operation, remember that activity functions have a default timeout of 5 minutes (configurable). For longer operations, use a sub-orchestrator.

## Step by step

1. **Define the Orchestrator Function** — Create an orchestrator function using the `[OrchestrationTrigger]` binding. This function defines the workflow using procedural code. It must be deterministic — no direct I/O, no random values, no DateTime.Now. Use `context.CurrentUtcDateTime` for time and `context.NewGuid()` for unique IDs. The orchestrator is replayed on every execution, so any side effects must be idempotent. The function should be a static method returning `Task` (C#) or equivalent in other languages.
2. **Implement Activity Functions** — Activity functions are stateless functions triggered by the `[ActivityTrigger]` binding. They perform the actual work — calling APIs, processing data, etc. Each activity function should do one thing and do it well. They can be scaled independently. Activity functions can use any Azure Functions binding (HTTP, Blob, etc.). They receive input from the orchestrator and return output. They can throw exceptions, which the orchestrator can catch and retry.
3. **Configure Storage and Task Hub** — Durable Functions requires an Azure Storage account. In `host.json`, configure the `durableTask` section with a `hubName` and `storageProvider` connection string. The task hub name must be unique within a storage account. The storage account must have queues, tables, and blobs enabled. For production, use a separate storage account for Durable Functions to avoid throttling from other workloads. Default partition count is 4; increase for higher concurrency.
4. **Start the Orchestration** — Use a client function (e.g., HTTP-triggered) to start the orchestration. Call `starter.StartNewAsync("OrchestratorFunction", input)` where `starter` is an `IDurableOrchestrationClient`. This returns an instance ID. The client function can return the instance ID to the caller for status checks. Orchestrations can be started from various triggers: HTTP, Queue, Timer, etc. The start message is placed in the control queue.
5. **Monitor and Manage Instances** — Use the `IDurableOrchestrationClient` to query status, raise events, terminate, or purge instances. The `GetStatusAsync` method returns orchestration status (Running, Completed, Failed, etc.). Use `RaiseEventAsync` to send external events. Use `TerminateAsync` to stop a running orchestration. Use `PurgeInstanceHistoryAsync` to clean up completed instances (important for cost and performance). The status is stored in the `DurableFunctionsInstances` table.

## Comparisons

### Durable Functions vs Azure Logic Apps

**Durable Functions:**
- Code-first workflow definition (C#, JavaScript, Python, etc.)
- Supports fan-out/fan-in, chaining, human interaction natively
- Lower cost per execution for high-volume scenarios
- Tight integration with Azure Functions ecosystem
- Requires storage account for state management

**Azure Logic Apps:**
- Designer-driven, no-code/low-code workflow
- Built-in connectors for 200+ services (SAP, Salesforce, etc.)
- Higher per-execution cost, but easier to set up for simple workflows
- Limited custom code (can use Azure Functions as a connector)
- State managed internally by Logic Apps runtime

## Common misconceptions

- **Misconception:** Durable Functions requires a premium plan. **Reality:** Durable Functions works on Consumption, Premium, and Dedicated plans. Consumption plan has a 10-minute timeout for all functions, but orchestrator replay resets the clock. Activity functions have a 5-minute default timeout (configurable up to 10 minutes on Consumption).
- **Misconception:** Orchestrator functions can use any C# code. **Reality:** Orchestrator functions must be deterministic. They cannot use `DateTime.Now`, `Guid.NewGuid()`, `Random`, or perform I/O (HTTP, database). They must use `context.CurrentUtcDateTime` and `context.NewGuid()`. Any non-deterministic code will cause replay errors.
- **Misconception:** Fan-out/fan-in runs activities in parallel within the same process. **Reality:** Activities are dispatched as separate function invocations. They can run on different hosts, even in different regions. The orchestrator only coordinates via storage messages.
- **Misconception:** Human interaction requires a separate Azure Logic App. **Reality:** Human interaction can be implemented entirely within Durable Functions using `WaitForExternalEvent` and `CreateTimer`. No external service is required, though you may use Logic Apps for complex approvals.
- **Misconception:** Durable Functions state is stored in memory. **Reality:** State is persisted in Azure Storage (or another provider). Orchestrator history is stored in tables, messages in queues, and large state in blobs. This ensures durability across restarts.

## Key takeaways

- Durable Functions enables stateful workflows in serverless Azure Functions using orchestrator and activity functions.
- Orchestrator functions must be deterministic; use context.CurrentUtcDateTime and context.NewGuid() instead of DateTime.Now and Guid.NewGuid().
- Function chaining sequences activities; fan-out/fan-in runs activities in parallel and aggregates results.
- Human interaction uses durable timers and WaitForExternalEvent for approval workflows with timeouts.
- Default storage provider is Azure Storage; configure task hub name and partition count in host.json.
- Activity functions have a default timeout of 5 minutes (configurable up to 10 minutes on Consumption plan).
- Retry policies can be set per activity call with FirstRetryInterval (default 1s), BackoffCoefficient (default 2), and MaxNumberOfAttempts.
- Orchestrator replay reads history from storage; this ensures fault tolerance but adds latency.
- Fan-out concurrency is limited by maxConcurrentTaskActivityWorkItems (default 10 per host) and control queue partition count (default 4).
- Durable timers are persisted; their maximum duration is limited by storage queue message TTL (default 7 days).
- External events are raised via RaiseEventAsync from a client function; the orchestrator waits using WaitForExternalEvent.
- Common exam trap: using non-deterministic code in orchestrator, or using direct I/O instead of activity functions.

## FAQ

**What is the difference between an orchestrator function and an activity function?**

An orchestrator function defines the workflow and coordinates activities. It must be deterministic and cannot perform I/O. An activity function performs the actual work (e.g., calling an API, processing data) and can use any Azure Functions binding. The orchestrator calls activity functions via `context.CallActivityAsync`. Activity functions are stateless and can be scaled independently.

**How do I implement a timeout for human interaction in Durable Functions?**

Use `context.CreateTimer` to create a durable timer that fires at the deadline, and `context.WaitForExternalEvent` to wait for an external event. Use `Task.WhenAny` to race the timer and the event. If the timer completes first, handle the timeout (e.g., escalate). The timer and event are both durable — they survive host restarts.

**Can I use Durable Functions on the Consumption plan?**

Yes. Durable Functions works on Consumption, Premium, and Dedicated plans. On Consumption, the default function timeout is 10 minutes, but orchestrator functions are not subject to this timeout because they replay. Activity functions have a default timeout of 5 minutes (configurable up to 10 minutes on Consumption).

**How do I pass large amounts of data between activities?**

Avoid passing large payloads directly as inputs/outputs because they are stored in Azure Tables (max 64 KB per entity). Instead, store large data in Azure Blob Storage and pass the blob URL. The activity can then read the blob. This is more efficient and avoids storage limits.

**What happens if an activity function throws an exception?**

The exception is caught by the orchestrator. You can use `CallActivityWithRetryAsync` to automatically retry based on a retry policy. If retries are exhausted or you don't use retry, the exception propagates to the orchestrator, which can catch it and handle it (e.g., call a compensation activity). If unhandled, the orchestration fails.

**How do I monitor the progress of an orchestration?**

Use the `IDurableOrchestrationClient` to query status by instance ID. The `GetStatusAsync` method returns the orchestration status (Running, Completed, Failed, etc.), custom status, and output. You can also enable Application Insights for detailed telemetry. The status is stored in the `DurableFunctionsInstances` table.

**What is the role of the task hub in Durable Functions?**

A task hub is a logical container that groups related Durable Functions resources within a storage account. Each task hub has its own queues, tables, and blobs. Multiple function apps can share the same storage account if they use different task hub names. The task hub name is configured in host.json under `durableTask.hubName`.

---

Interactive version with quiz and diagrams: https://courseiva.com/learn/az-204/durable-functions-204