CloudFormation Custom Resources with Lambda

CloudFormation Custom Resources allow you to write custom provisioning logic for resources that are not natively supported by CloudFormation. They act as a bridge between CloudFormation and any AWS service or external system. The custom resource is defined in a CloudFormation template using the AWS::CloudFormation::CustomResource or Custom::* type. When CloudFormation creates, updates, or deletes the stack, it sends a request to a specified endpoint — typically an AWS Lambda function or an SNS topic — which performs the desired action and returns a response to CloudFormation.

AWS CloudFormation supports hundreds of native resources, but you may need to:

Custom Resources fill this gap by allowing you to define arbitrary provisioning logic.

The interaction between CloudFormation and the custom resource provider (Lambda) follows a strict protocol:

1. CloudFormation sends a request — When creating, updating, or deleting a stack, CloudFormation generates a JSON payload and sends it to the specified ServiceToken. The payload includes: - RequestType: Create, Update, or Delete. - ResourceProperties: The properties defined in the template. - OldResourceProperties: (only on Update) the previous properties. - StackId, RequestId, ResourceType, LogicalResourceId, PhysicalResourceId (for Update/Delete).

2. Lambda function processes the request — The Lambda function must perform the desired action (e.g., create an ACM certificate) and then send a response back to CloudFormation via a pre-signed S3 URL provided in the request (ResponseURL). The response must be a JSON object with: - Status: SUCCESS or FAILED. - PhysicalResourceId: A unique identifier for the resource (used in subsequent Update/Delete requests). - StackId, RequestId, LogicalResourceId (must match the request). - Data: Optional key-value pairs that can be referenced using Fn::GetAtt in the template.

To create a custom resource, define it in the template:

Resources:
  MyCustomResource:
    Type: Custom::MyCustomType
    Properties:
      ServiceToken: !GetAtt MyLambdaFunction.Arn
      Property1: value1
      Property2: value2

The Lambda function must include code to parse the request, perform the action, and send the response. A common pattern is to use the cfn-response module (Node.js) or cfnresponse (Python) to simplify sending responses.

Example Python handler:

import cfnresponse
import boto3

def handler(event, context):
    physical_resource_id = event.get('PhysicalResourceId', None)
    try:
        if event['RequestType'] == 'Create':
            # Create resource
            physical_resource_id = 'unique-id'
        elif event['RequestType'] == 'Update':
            # Update resource
        elif event['RequestType'] == 'Delete':
            # Delete resource
        cfnresponse.send(event, context, cfnresponse.SUCCESS, {}, physical_resource_id)
    except Exception as e:
        cfnresponse.send(event, context, cfnresponse.FAILED, {}, physical_resource_id, reason=str(e))

Important: The function must send a response for every request type, including Delete. Failing to do so will cause the stack to hang and eventually fail.

If the Lambda function returns FAILED or times out, CloudFormation will:

Common mistake: Not handling the Delete request during rollback. If the function does not respond to the Delete request, the stack will hang and eventually fail rollback.

A company uses CloudFormation to deploy a multi-region application with HTTPS endpoints. ACM certificates are not natively supported by CloudFormation for DNS validation (only email validation was native at the time). They create a custom resource backed by a Lambda function that:

Scale: Deployed across 5 regions, each with 10 certificates. The Lambda function must handle concurrent requests and respect AWS API rate limits. Pitfall: If the Lambda function times out during DNS validation (which can take minutes), the stack hangs. They set ServiceTimeout to 600 seconds and Lambda timeout to 900 seconds.

A financial services company uses CloudFormation to provision EC2 instances that must be registered in an on-premises CMDB. They use a custom resource with a Lambda function that makes HTTP calls to the on-premises API via a VPN. The Lambda function:

Considerations: The Lambda function must have VPC access to reach the on-premises API. Network latency and API failures must be handled with retries. Misconfiguration: If the Lambda function fails to respond due to a network timeout, CloudFormation will wait up to 1 hour before failing, causing a long deployment cycle.

A gaming company uses CloudFormation to build AMIs using a custom resource. The Lambda function triggers an EC2 Image Builder pipeline. The pipeline takes up to 30 minutes. The custom resource:

Challenge: The Lambda function must handle the long-running process. Since Lambda has a 15-minute max timeout, they cannot wait 30 minutes. Solution: Use an S3 bucket as a coordination point. The function starts the pipeline, sends a response immediately with a placeholder PhysicalResourceId, and then a separate process (e.g., EventBridge rule) updates the stack after completion. This is an advanced pattern called 'async custom resource'.