CloudFormation Drift Detection

CloudFormation Drift Detection is a feature that allows you to determine whether a stack's actual resource configuration has deviated from the expected configuration defined in the stack template and any associated parameter values. Drift occurs when resources are modified outside of CloudFormation management, such as through the AWS Management Console, CLI, or SDK. Drift detection helps you identify these changes so you can take corrective action, ensuring your infrastructure remains consistent with your infrastructure-as-code templates.

In a well-managed environment, all resource changes should be made through CloudFormation stack updates. However, in practice, team members or automated processes may inadvertently modify resources directly. This creates a divergence between the template and the actual state, leading to:

When you initiate drift detection on a stack, CloudFormation performs the following steps: 1. Resource-level comparison: For each resource in the stack, CloudFormation retrieves the current configuration from the AWS service (e.g., EC2, S3, IAM). It then compares this configuration against the expected properties defined in the stack template, taking into account any resolved parameter values and stack policies. 2. Supported resources: Not all resources support drift detection. AWS maintains a list of resource types that can be checked. For example, AWS::EC2::Instance supports drift detection, but AWS::CloudFormation::Stack (nested stacks) does not. The list is updated as AWS adds support for new resource types. 3. Drift status categories: Each resource is assigned a drift status: - IN_SYNC: The resource's current configuration matches the template. - MODIFIED: The resource's current configuration differs from the template. - DELETED: The resource has been deleted from the stack but was expected to exist. - NOT_CHECKED: Drift detection was not performed on the resource (e.g., unsupported resource type). 4. Stack-level drift status: The overall stack drift status is determined by the status of its resources: - DRIFTED: At least one resource has a status of MODIFIED or DELETED. - IN_SYNC: All resources are IN_SYNC. - NOT_CHECKED: No drift detection has been performed on the stack. - UNKNOWN: The stack is in an unknown state (e.g., during drift detection).

- Drift detection operation: You can start drift detection on a stack via the AWS Management Console, AWS CLI, or API. There is no charge for drift detection. - Detection frequency: Drift detection is not automatic; you must manually initiate it. There is no built-in scheduler. However, you can use Amazon EventBridge (formerly CloudWatch Events) to trigger a Lambda function that calls the DetectStackDrift API on a schedule. - Time to complete: The time depends on the number of resources and the APIs called. Typically, it takes a few seconds to a few minutes for stacks with dozens of resources. - Output: The result includes a list of drifted resources with details of the differences. For each property that drifted, the output shows the expected value (from the template) and the actual value (from the resource). - API operations: - DetectStackDrift: Initiates drift detection on a stack. Returns a StackDriftDetectionId. - DescribeStackDriftDetectionStatus: Returns the status of a drift detection operation. - DescribeStackResourceDrifts: Returns drift information for the resources in a stack. - IAM permissions: To perform drift detection, the IAM user or role needs cloudformation:DetectStackDrift, cloudformation:DescribeStackDriftDetectionStatus, and cloudformation:DescribeStackResourceDrifts. Additionally, read permissions for the resources being checked (e.g., ec2:DescribeInstances, s3:GetBucketPolicy) are required.

To detect drift using the AWS CLI:

aws cloudformation detect-stack-drift --stack-name my-stack

This returns a StackDriftDetectionId, which you use to check the status:

aws cloudformation describe-stack-drift-detection-status --stack-drift-detection-id <id>

To view the drift details for each resource:

aws cloudformation describe-stack-resource-drifts --stack-name my-stack

Example output snippet:

{
    "StackResourceDrifts": [
        {
            "StackId": "arn:aws:cloudformation:us-east-1:123456789012:stack/my-stack/...",
            "LogicalResourceId": "MySecurityGroup",
            "PhysicalResourceId": "sg-12345678",
            "ResourceType": "AWS::EC2::SecurityGroup",
            "ExpectedProperties": "{\"GroupDescription\":\"My security group\",\"SecurityGroupIngress\":[{\"IpProtocol\":\"tcp\",\"FromPort\":22,\"ToPort\":22,\"CidrIp\":\"10.0.0.0/8\"}]}",
            "ActualProperties": "{\"GroupDescription\":\"My security group\",\"SecurityGroupIngress\":[{\"IpProtocol\":\"tcp\",\"FromPort\":22,\"ToPort\":22,\"CidrIp\":\"0.0.0.0/0\"}]}",
            "PropertyDifferences": [
                {
                    "PropertyPath": "/SecurityGroupIngress/0/CidrIp",
                    "ExpectedValue": "10.0.0.0/8",
                    "ActualValue": "0.0.0.0/0",
                    "DifferenceType": "NOT_EQUAL"
                }
            ],
            "DriftStatus": "MODIFIED"
        }
    ]
}

When drift is detected, you have several options: - Update the stack: Modify the stack template to match the actual resource configuration, then perform a stack update. This reconciles the template with reality. - Revert the manual change: Manually change the resource back to the expected configuration using the appropriate AWS service console or CLI. - Ignore the drift: If the drift is intentional and you want to keep the manual change, you can update the template to reflect the new configuration. - Delete and recreate the resource: If the resource is fully drifted and cannot be corrected, you may need to delete it and let CloudFormation recreate it.

A financial services company uses CloudFormation to deploy and manage its PCI DSS-compliant infrastructure. The security team mandates that all security group rules must be reviewed and approved. Despite strict change management, an engineer accidentally opens a security group port to 0.0.0.0/0 via the console. Drift detection, scheduled weekly via EventBridge and Lambda, catches this change. The security team receives an SNS notification and reverts the rule. Without drift detection, the unauthorized change could go unnoticed for weeks, leading to a compliance violation.

A large organization uses AWS Organizations and CloudFormation StackSets to deploy a standard VPC configuration across hundreds of accounts. After a regional outage, a network engineer in one account manually modifies a route table to add a temporary route. The central cloud team runs drift detection on all stack instances using a script that iterates through accounts. They discover the drifted route table in one account. They then update the StackSet template to include the temporary route permanently (if needed) or remove the manual route. Drift detection here is critical for maintaining consistency across accounts.

For stacks with thousands of resources, drift detection can take several minutes. The API calls to describe resources can be throttled if too many are made in a short time. To avoid throttling, you can implement exponential backoff in your automation scripts. Additionally, some resources, like AWS::Lambda::Function, have many properties; drift detection compares all supported properties, which can be time-consuming. It's best to run drift detection during off-peak hours.