# Step Functions Error Handling and Retries

> Chapter 114 of the Courseiva SAA-C03 curriculum — https://courseiva.com/learn/saa-c03/step-functions-error-handling

**Official objective:** 2.1 — Resilient Architectures

## Introduction

This chapter covers AWS Step Functions error handling and retries, a critical skill for the SAA-C03 exam under Resilient Architectures (Objective 2.1). You will learn how to design state machines that gracefully handle failures using retry policies, catch clauses, and fallback states. Approximately 10-15% of exam questions touch on Step Functions or related serverless workflow concepts, making this a high-yield topic. Mastering error handling ensures your architectures are resilient and maintainable.

## The Parcel Courier with a Smart Routing System

A busy courier depot, parcels piled high on sorting racks. Each parcel has a delivery plan: go from office A to B to C, with specific tasks at each stop (e.g., sign, stamp, weigh). Sometimes a delivery fails — maybe the recipient is out, the address is wrong, or the parcel is damaged. The courier doesn't just give up; they follow a retry policy: wait 10 seconds, try again up to 3 times, each time with a longer wait (exponential backoff). If still failing, they escalate to a fallback action: redirect to a different office (a catch-all state). If the fallback also fails, they send the parcel to a dead letter queue (DLQ) for manual inspection. The courier also logs every attempt with timestamps and reasons, so the manager can audit. This is exactly how AWS Step Functions handles errors: it retries with configurable intervals, uses fallback states (Catch) to route to alternative logic, and can send failed executions to a DLQ for analysis. The state machine definition is the delivery plan, and each state can have its own error handling rules.

## Core explanation

### What is AWS Step Functions and Why Error Handling Matters

AWS Step Functions is a serverless orchestration service that lets you coordinate multiple AWS services into a workflow. You define a state machine using Amazon States Language (ASL) — a JSON-based language. Each state performs a task (e.g., invoke a Lambda function, publish to SNS, run a batch job) or makes a choice, waits, etc. In a distributed system, failures are inevitable: a Lambda may timeout, an API may throttle, or a service may be unavailable. Without proper error handling, a single failure can cause the entire workflow to abort. Step Functions provides two powerful mechanisms: **Retry** and **Catch**. Retry automatically re-attempts a failed state after a delay, optionally with exponential backoff. Catch catches errors and routes the execution to a different state (fallback) or ends the workflow gracefully. You can also send failed executions to an Amazon SQS dead-letter queue (DLQ) for later analysis.

### How Error Handling Works Internally

When a state execution fails (e.g., Lambda returns an error, an activity task times out), Step Functions evaluates the **Retry** block first. If a Retry block exists and matches the error (by error name), it applies the retry policy: interval, max attempts, backoff rate. The default retry interval is 1 second, with max attempts of 3, and a backoff rate of 2.0 (exponential). After the interval, Step Functions re-executes the state. If all retries are exhausted, or if no Retry matches, Step Functions evaluates the **Catch** block. If a Catch block matches, the execution transitions to the state specified in the `Next` field of the Catch rule. If no Catch matches, the execution fails (ends with `FAILED` status).

### Key Components, Values, and Defaults

- **Retry**: Array of retrier objects. Each retrier has:
  - `ErrorEquals`: List of error strings to match (e.g., `States.ALL`, `Lambda.ServiceException`, `States.Timeout`).
  - `IntervalSeconds`: Initial wait time before first retry (default 1).
  - `MaxAttempts`: Maximum number of retry attempts (default 3).
  - `BackoffRate`: Multiplier for each subsequent interval (default 2.0).
- **Catch**: Array of catcher objects. Each catcher has:
  - `ErrorEquals`: Like Retry.
  - `Next`: The name of the next state to transition to if this catch matches.
  - `ResultPath`: Optional JSON path that stores the error information (e.g., `$.error`).
- **Dead Letter Queue (DLQ)**: Configured at the state machine level (not per state). When enabled, any execution that reaches a terminal state (FAILED or TIMED_OUT) can be sent to an SQS queue for later inspection.

### Configuration and Verification

You define error handling in the state machine definition JSON. Example:

```json
{
  "StartAt": "ProcessOrder",
  "States": {
    "ProcessOrder": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:us-east-1:123456789012:function:processOrder",
      "Retry": [
        {
          "ErrorEquals": ["Lambda.ServiceException", "Lambda.AWSLambdaException", "Lambda.Unknown"],
          "IntervalSeconds": 2,
          "MaxAttempts": 3,
          "BackoffRate": 2.0
        },
        {
          "ErrorEquals": ["States.Timeout"],
          "IntervalSeconds": 5,
          "MaxAttempts": 2,
          "BackoffRate": 1.5
        }
      ],
      "Catch": [
        {
          "ErrorEquals": ["States.ALL"],
          "Next": "FallbackState",
          "ResultPath": "$.error-info"
        }
      ],
      "End": true
    },
    "FallbackState": {
      "Type": "Pass",
      "Result": "Order processing failed after retries.",
      "End": true
    }
  }
}
```

To verify, you can use the AWS Management Console to view execution history, which shows each retry attempt and the final error. You can also use the AWS CLI:

```bash
aws stepfunctions describe-execution --execution-arn <arn>
aws stepfunctions get-execution-history --execution-arn <arn>
```

### Interaction with Related Technologies

Step Functions integrates deeply with Lambda, ECS, Batch, DynamoDB, SNS, SQS, etc. Error handling is especially important when invoking Lambda: Lambda can throw `ServiceException` (service-side), `ResourceConflictException` (if function is being updated), or `TooManyRequestsException` (throttling). You should retry on throttling and service errors, but not on client errors like invalid input. For ECS tasks, you may get `ECS.TaskFailed` or `ECS.Timeout`. For API Gateway, `States.Timeout` occurs if the HTTP request times out. Always use `States.ALL` as a catch-all for unexpected errors.

### Edge Cases and Exam Traps

- **Retry vs. Catch**: Retry only applies to the same state; Catch routes to a different state. If you want to attempt recovery in a different state, use Catch.
- **ErrorEquals**: `States.ALL` matches any error. It must be listed alone in its own retrier/catcher because it is mutually exclusive with other error names.
- **ResultPath**: In Catch, you can capture error details (e.g., `$.error-info`) to use in the fallback state. In Retry, you cannot capture error info; the state is re-executed fresh.
- **DLQ**: Only works for state machines with `LoggingConfiguration` set to `ALL` or `ERROR`. The DLQ must be an SQS queue with the correct IAM permissions for Step Functions to send messages.
- **Timeout**: Each state can have a `TimeoutSeconds` parameter. If the state exceeds this, it triggers a `States.Timeout` error, which can be caught or retried.
- **Heartbeat**: For Activity tasks, you can set a `HeartbeatSeconds`. If the activity doesn't send a heartbeat within that time, it fails with `States.HeartbeatTimeout`.

### Best Practices

- Use exponential backoff with jitter (not supported natively, but you can simulate via wait states).
- Limit max attempts to 3-5 to avoid infinite loops.
- Use `States.ALL` in Catch to handle unexpected errors gracefully.
- For critical workflows, enable DLQ to capture failures for manual review.
- Monitor execution history and set CloudWatch alarms on `ExecutionFailed` metrics.

## Real-world context

### Enterprise Scenario 1: E-commerce Order Processing

A large e-commerce platform uses Step Functions to orchestrate order fulfillment: charge credit card (via Lambda), update inventory (DynamoDB), send confirmation email (SES), and notify warehouse (SQS). If the payment Lambda fails due to a temporary service outage, retry with exponential backoff (3 attempts, 2s initial) gives the service time to recover. If all retries fail, a Catch block routes to a `ManualReview` state that sends a notification to the operations team and stores the order in a backup DynamoDB table. The DLQ captures all failed executions for auditing. In production, with 10,000 orders per hour, retries succeed in 95% of transient failures, reducing manual intervention. Misconfiguration, like setting `MaxAttempts` too high (e.g., 10), can cause long delays and cost overruns.

### Enterprise Scenario 2: Data Processing Pipeline

A financial services company processes daily transaction files: download from S3, validate format (Lambda), transform (Glue), and load into Redshift. If the Glue job fails due to resource contention, retry with a longer interval (30s) and only 2 attempts to avoid excessive wait. A Catch block routes to a `SendAlert` Lambda that emails the data team. The DLQ stores the file name and error for later reprocessing. Scaling: the pipeline runs once per day, so retry delays are acceptable. A common mistake is not catching `States.Timeout` errors when the Glue job takes longer than expected, causing the entire workflow to fail silently.

### What Goes Wrong When Misconfigured

- **Infinite retries**: Setting `MaxAttempts` to 0 or omitting it (default 3) is fine, but if the error is non-transient (e.g., invalid input), retries waste resources. Always use Catch for non-retryable errors.
- **Missing catch-all**: If you only catch specific errors and an unexpected error occurs, the execution fails without any fallback. Always include `States.ALL` in Catch.
- **DLQ not working**: Forgetting to set `LoggingConfiguration` to `ALL` or `ERROR` means the DLQ never receives messages. Also, the SQS queue policy must grant `sqs:SendMessage` to Step Functions.
- **ResultPath conflicts**: If you use `ResultPath` in Catch, ensure the path doesn't overwrite existing data needed later.

## Exam focus

### What SAA-C03 Tests

SAA-C03 Objective 2.1: 'Design resilient architectures' includes designing for fault tolerance. Step Functions error handling appears in questions about orchestrating microservices, handling failures in serverless workflows, and designing retry logic. Expect 1-2 questions on this exact topic.

### Common Wrong Answers and Why Candidates Choose Them

1. **'Use a Lambda function to retry the failed step.'** – Candidates think they can write custom retry logic inside Lambda. But Step Functions built-in retry is simpler, more reliable, and cost-effective. The exam wants you to use native features.
2. **'Set MaxAttempts to 0 to disable retries.'** – The default is 3, not 0. Setting to 0 actually means infinite retries (because it's the maximum number of *attempts* including the first? No, MaxAttempts is the total number of executions including the first. Setting to 0 would mean zero attempts, which is invalid. The correct way to disable retries is to omit the Retry block entirely.
3. **'Use Catch to retry the same state.'** – Catch routes to a *different* state; it does not retry the original state. Retry is for retrying the same state. Candidates confuse the two.
4. **'The DLQ catches all errors automatically.'** – DLQ only works if logging is enabled and the execution fails. It does not catch errors during execution; it captures the final failed execution record.

### Specific Numbers and Terms

- Default `IntervalSeconds`: 1
- Default `MaxAttempts`: 3
- Default `BackoffRate`: 2.0
- `States.ALL` matches any error.
- `States.Timeout` is a built-in error for state timeouts.
- `States.HeartbeatTimeout` for activity tasks.
- `ResultPath` in Catch stores error info (e.g., `$.error`).
- DLQ requires `LoggingConfiguration` with level `ALL` or `ERROR`.

### Edge Cases

- Retry on `States.ALL` is possible but rarely advisable because it will retry on non-transient errors.
- If both Retry and Catch are present, Retry is evaluated first.
- If a Catch block transitions to a state that also fails, that state's error handling applies (recursive).
- You cannot have multiple Catch blocks with the same `ErrorEquals`; the first match wins.

### How to Eliminate Wrong Answers

- If the question asks about retrying the same operation, look for `Retry` in the options.
- If it asks about doing something different after failure, look for `Catch`.
- If it asks about capturing failed executions for later analysis, look for DLQ.
- If the answer mentions writing custom retry logic in Lambda, it's almost certainly wrong.

## Step by step

1. **Define the State Machine** — Create a state machine using Amazon States Language (ASL) JSON. Specify the `StartAt` state and define each state with its type (Task, Choice, Wait, etc.). For each Task state, you set the `Resource` ARN (e.g., Lambda function). This is where you will add error handling blocks. The definition is the blueprint of the workflow. Use the AWS Management Console, AWS CLI, or Infrastructure as Code (e.g., CloudFormation) to create the state machine. Ensure IAM roles grant Step Functions permission to invoke the target services.
2. **Configure Retry Policy** — Inside each state, add a `Retry` array. Each retry object specifies which errors to retry (`ErrorEquals`), the initial wait time (`IntervalSeconds`), maximum attempts (`MaxAttempts`), and backoff multiplier (`BackoffRate`). For example, to retry on Lambda throttling errors up to 3 times with a 2-second initial delay and doubling each time, set `IntervalSeconds: 2`, `MaxAttempts: 3`, `BackoffRate: 2.0`. The actual wait times would be 2s, 4s, 8s. If the state fails after all retries, Step Functions moves to the Catch block.
3. **Configure Catch Block** — Add a `Catch` array after the `Retry` block (or standalone). Each catch object matches errors and transitions to a fallback state via `Next`. You can optionally set `ResultPath` to store error details (e.g., `$.error`). If multiple catch rules exist, they are evaluated in order; the first match wins. Use `States.ALL` as a catch-all at the end. If no Catch matches, the execution fails. Example: catch any error and go to a `NotifyFailure` state that sends an SNS alert.
4. **Define Fallback States** — Create one or more fallback states that handle errors gracefully. These can be `Pass` states (to record a result), `Task` states (to run cleanup logic), or `Choice` states (to branch based on error type). Use the `ResultPath` from the Catch to access error details. For example, a fallback state might update a DynamoDB table with the error message and then transition to an `End` state. Alternatively, you can use a `Succeed` or `Fail` state to terminate with a custom cause.
5. **Enable Dead Letter Queue** — At the state machine level, under `LoggingConfiguration`, set `Level` to `ALL` or `ERROR` and specify a `Destinations` list with an SQS queue ARN. This is the DLQ. When an execution fails (reaches a terminal FAILED state), Step Functions sends the execution input and error details to the SQS queue. You can then process these messages (e.g., via Lambda) for manual retry or analysis. Ensure the SQS queue has a policy allowing Step Functions to send messages.

## Comparisons

### Retry vs Catch

**Retry:**
- Re-attempts the same state after failure.
- Uses exponential backoff with configurable interval, max attempts, and backoff rate.
- Cannot transition to a different state.
- Does not capture error details unless using ResultPath? No, Retry does not support ResultPath.
- Best for transient errors (e.g., throttling, service timeouts).

**Catch:**
- Transitions to a different state on failure.
- No retry logic; you must implement retry in the fallback if needed.
- Can capture error details via ResultPath.
- Evaluated after Retry is exhausted or if no Retry matches.
- Best for non-transient errors or when you need fallback logic.

## Common misconceptions

- **Misconception:** Retry and Catch are interchangeable; you can use either to handle errors. **Reality:** Retry re-attempts the same state; Catch routes to a different state. They serve different purposes. Retry is for transient failures; Catch is for fallback logic.
- **Misconception:** Setting MaxAttempts to 0 disables retries. **Reality:** MaxAttempts is the total number of executions including the first. Setting it to 0 is invalid and causes an error. To disable retries, omit the Retry block entirely.
- **Misconception:** The Dead Letter Queue (DLQ) captures all errors during execution. **Reality:** DLQ only captures the final failed execution record (input and error) when the execution ends in a terminal FAILED state. It does not capture intermediate retries.
- **Misconception:** You can use Retry to catch errors and transition to a different state. **Reality:** Retry only re-executes the same state. To transition to a different state on failure, you must use Catch.
- **Misconception:** States.ALL matches all errors including custom errors. **Reality:** States.ALL matches any error, including custom ones thrown by Lambda. It is a wildcard.

## Key takeaways

- Retry re-executes the same state; Catch routes to a different state.
- Default Retry values: IntervalSeconds=1, MaxAttempts=3, BackoffRate=2.0.
- Use States.ALL in Catch to handle unexpected errors.
- DLQ requires LoggingConfiguration with level ALL or ERROR.
- Retry is evaluated before Catch.
- MaxAttempts includes the original attempt; setting to 0 is invalid.
- ResultPath in Catch stores error information (e.g., $.error).

## FAQ

**What is the difference between Retry and Catch in Step Functions?**

Retry automatically re-attempts the same state after a failure, with configurable intervals and backoff. Catch, on the other hand, catches errors and transitions the execution to a different state. Retry is evaluated first; if all retries are exhausted, Catch is then evaluated. Use Retry for transient errors that may succeed on retry, and use Catch for fallback logic when retries fail or for non-retryable errors.

**How do I set up a Dead Letter Queue for Step Functions?**

To enable a DLQ, you must configure logging on the state machine. In the state machine definition, set `LoggingConfiguration` with `Level` set to `ALL` or `ERROR` and specify a `Destinations` list containing an SQS queue ARN. Ensure the SQS queue has a resource-based policy allowing Step Functions to send messages. The DLQ will capture the input and error details of any execution that reaches a terminal FAILED state.

**Can I use Retry to handle custom errors thrown by Lambda?**

Yes. Lambda can throw custom errors (e.g., using `context.fail()` in Node.js or raising an exception in Python). These errors appear with the error name you specify. You can match them in the `ErrorEquals` field of a Retry or Catch block. For example, if your Lambda throws 'InvalidInput', you can add `"ErrorEquals": ["InvalidInput"]` to a Retry block, but it's better to use Catch for custom errors because they are usually non-transient.

**What happens if I don't specify any Retry or Catch?**

If a state fails and no Retry or Catch is defined, the entire state machine execution fails immediately with a `FAILED` status. The execution history will show the error. This is often undesirable; you should at least add a Catch with `States.ALL` to handle failures gracefully.

**How does exponential backoff work in Step Functions Retry?**

The retry interval after the nth attempt is calculated as: `IntervalSeconds * (BackoffRate ^ (n-1))`. For example, with `IntervalSeconds=2`, `BackoffRate=2.0`, after the first failure (attempt 2), wait 2 seconds; after second failure (attempt 3), wait 4 seconds; after third failure (attempt 4), wait 8 seconds. Note that the first attempt is not delayed.

**Can I use ResultPath in Retry?**

No. The `ResultPath` field is only available in Catch blocks. In Retry, the state is re-executed with the original input; you cannot capture error details. If you need to capture error information, use Catch.

**What is the maximum number of retries I can set?**

There is no hard limit, but you should keep it reasonable (e.g., 3-5) to avoid excessive execution time and cost. The state machine has a maximum execution time of 1 year, but practical limits apply.

---

Interactive version with quiz and diagrams: https://courseiva.com/learn/saa-c03/step-functions-error-handling