Troubleshooting Scenario Questions

Check the stack outputs.

Why it fails: Stack outputs are values defined in the Outputs section of the template, and CloudFormation only populates them after the stack has been successfully created or updated. If the stack creation failed, the stack never reaches a complete state, so no Outputs are available in the console or via DescribeStacks. Therefore, checking outputs offers zero diagnostic value for a failed creation.

Delete the stack and recreate it with the same parameters.

Why it fails: Deleting the failed stack triggers CloudFormation to clean up any resources that were created, which also discards the stack events and the rollback history that contained the failure reasons. Recreating with identical parameters will almost certainly reproduce the same failure, and you'll be no closer to identifying the root cause. The correct pattern is to inspect stack events and template logic before any destructive action.

Check AWS CloudTrail logs for the stack creation attempt.

Why it fails: AWS CloudTrail records the underlying API calls that CloudFormation makes on your behalf (for example, ec2:RunInstances or iam:CreateRole), but it does not include CloudFormation-specific resource status reasons or stack-level error messages. To understand why a particular resource failed, you need the statusReason found in the stack events, not the raw AWS API logs. CloudTrail is useful for auditing who triggered the stack operation, not for diagnosing mostly internal CloudFormation resource failures.

The user needs s3:PutObjectAcl permission

Why it fails: The `s3:PutObjectAcl` permission is specifically required only when an S3 upload operation explicitly attempts to set an Access Control List (ACL) on the new object, such as specifying a canned ACL like `bucket-owner-full-control`. For standard uploads where object ownership is enforced by the bucket policy or the object is owned by the uploading account, this permission is not necessary. Therefore, its absence does not inherently prevent a basic object upload.

The policy does not allow s3:ListBucket

Why it fails: The `s3:ListBucket` permission is used for listing the contents of an S3 bucket, such as retrieving a list of objects or common prefixes. This permission is entirely distinct from the `s3:PutObject` action, which is required for uploading new objects into a bucket. Therefore, the absence or presence of `s3:ListBucket` has no direct bearing on a user's ability to successfully upload an object.

The user does not have s3:GetObject permission

Why it fails: The `s3:GetObject` permission grants the ability to retrieve or download an object from an S3 bucket. While it is a common S3 action, it is completely separate from the `s3:PutObject` permission, which is necessary for uploading an object. The question specifically concerns an upload operation, making the user's `s3:GetObject` permissions irrelevant to the success or failure of that particular action.

Check the IAM role attached to the Lambda function for insufficient permissions.

Why it fails: Insufficient IAM permissions for the Lambda function's execution role would typically manifest as an AccessDeniedException or similar error within the function's CloudWatch Logs, often leading to a FunctionError status. If the function executed successfully without a FunctionError, it implies that the necessary permissions for its own execution and any AWS service calls it attempted were likely present.

Check the payload format against the function's expected input.

Why it fails: Although an incorrect payload format could certainly lead to unexpected behavior, if the Lambda function executed successfully without a FunctionError, it means the runtime successfully invoked the handler. Any issues with parsing or processing the event payload within the function's code would typically result in an error logged to CloudWatch, or the function would return an unexpected output based on its internal logic, rather than preventing the function from running at all.

Check the Lambda function's memory and timeout configuration.

Why it fails: Issues related to the Lambda function's memory allocation or timeout configuration would typically cause the function to terminate prematurely with a FunctionError or Task timed out message, respectively. Since the question implies the function executed successfully, these configuration parameters are unlikely to be the root cause of an unexpected output rather than a complete failure.

The IAM role does not have a trust policy for EC2.

Why it fails: A trust policy defines which AWS entities are permitted to assume an IAM role. If the EC2 instance were unable to assume the role due to a missing or incorrect trust policy for the `ec2.amazonaws.com` service principal, then *all* attempts to access AWS services using that role, including the successful S3 access, would fail with an `AccessDenied` error. The fact that S3 access works confirms the trust policy is correctly configured for EC2.

The DynamoDB table is in a different region than the EC2 instance.

Why it fails: AWS services like DynamoDB are designed to support cross-region access, provided the application explicitly configures the AWS SDK to target the correct region. If the DynamoDB table were in a different region and the application failed to specify it, the error would typically manifest as a `ResourceNotFoundException` or similar, indicating the table could not be located in the default or specified region, rather than an `AccessDeniedException` related to IAM permissions.

The application is using the wrong AWS SDK.

Why it fails: Using a different or outdated AWS SDK version might lead to issues such as incompatible API calls, serialization errors, or missing features, but it does not fundamentally alter the underlying IAM permissions granted by a role. The SDK's role is to facilitate interaction with AWS services and automatically retrieve credentials from the instance metadata service. Permission failures stem from the IAM policy itself or the role's attachment mechanism, not the SDK version used.

The Lambda function's concurrency is set too high, causing race conditions.

Why it fails: While a high concurrency setting for a Lambda function can indeed increase the likelihood of race conditions, this issue is distinct from the problem of duplicate event processing. Race conditions occur when multiple concurrent executions attempt to modify shared resources simultaneously, leading to unpredictable or inconsistent states. Duplicate event processing, however, stems from the source (S3) sending the same event multiple times, causing the Lambda function to perform the same logical operation redundantly, irrespective of how many instances run concurrently.

The DynamoDB table does not have a primary key that prevents duplicates.

Why it fails: A DynamoDB table must have a primary key, and it inherently prevents items with identical primary key values from being written. However, if the Lambda function generates a new, unique identifier (e.g., a UUID) for the primary key for each invocation, then even duplicate S3 events will result in separate, unique primary keys being generated and new items being inserted. The problem isn't the absence of a primary key, but rather the lack of a robust idempotency strategy that uses a consistent, event-derived identifier to prevent duplicate writes for the same logical operation.

The S3 bucket is configured with versioning, causing multiple object creation events.

Why it fails: S3 versioning tracks multiple versions of an object and can generate events for actions like creating new versions or deleting objects. While versioning might lead to multiple distinct events for different object states (e.g., s3:ObjectCreated:Put for a new version, then another for an update), it does not inherently cause the same event notification to be delivered multiple times. The fundamental cause of duplicate processing for a single logical S3 action is the "at least once" delivery guarantee of S3 event notifications, which applies regardless of whether versioning is enabled.

Increase the function's timeout

Why it fails: While increasing the function's timeout would prevent the specific timeout error, it merely extends the maximum allowed execution duration rather than addressing the root cause of the function's slowness. This approach can lead to higher costs due to longer billing durations and doesn't improve the actual efficiency or speed of the function's processing. It is generally recommended to optimize the function's performance first, for instance by increasing memory, before resorting to extending the timeout.

Increase the function's reserved concurrency

Why it fails: Increasing a Lambda function's reserved concurrency allocates a dedicated pool of execution environments, preventing other functions from consuming them and ensuring the function can scale to a specific number of simultaneous invocations. However, reserved concurrency does not influence the execution time or performance of a single invocation. It manages the maximum number of parallel executions, not the speed at which each individual execution completes, making it irrelevant for a function that is timing out due to slow processing.

Increase the S3 event notification batch size

Why it fails: S3 event notifications operate on a one-to-one basis, meaning each individual object creation, deletion, or modification event triggers a separate Lambda function invocation. Unlike some other event sources like SQS or Kinesis, S3 event notifications do not support or have a configurable batch size parameter for grouping multiple events into a single Lambda invocation. Therefore, attempting to increase a non-existent batch size would have no effect on the function's behavior or performance.

Modify the CloudFormation template to increase instance size.

Why it fails: Resizing the instance type addresses a resource-capacity problem, but a failed health check is almost always caused by an application error, misconfigured listener path, or crashed process — none of which are resolved by giving the instance more CPU or memory.

Add an Application Load Balancer to the environment.

Why it fails: A single-instance Elastic Beanstalk environment intentionally omits a load balancer to reduce cost; adding an ALB changes the environment type and traffic routing but does nothing to diagnose why the application itself is failing its health check.

SSH into the EC2 instance and restart the web server.

Why it fails: SSH access requires a pre-configured key pair and open port 22 in the security group, which may not be set up for this environment, and even if available, manually restarting the process only masks the symptom without revealing the underlying cause shown in logs.

The Auto Scaling group does not specify a VPC subnet.

Why it fails: An Auto Scaling group (ASG) must be configured with at least one VPC subnet through its `VpcZoneIdentifier` property to successfully launch instances. If this property were missing or invalid, the CloudFormation error message would specifically indicate a problem with the subnet configuration, such as "No default VPC for this user" or "Invalid Subnet ID," rather than an issue related to a null `SecurityGroupId`.

The Auto Scaling group's user data script contains a syntax error.

Why it fails: Syntax errors within an Auto Scaling group's user data script typically manifest much later in the instance launch process. CloudFormation successfully provisions the EC2 instance, but the instance itself fails to execute the script correctly during boot. This would lead to instances entering a `running` state but failing health checks or not joining the application, without generating a `SecurityGroupId` validation error during the initial CloudFormation stack creation.

The launch template specifies an invalid key pair name.

Why it fails: If a launch template specifies an invalid or non-existent `KeyPairName`, the EC2 API would return a specific error indicating that the "Key pair 'your-key-name' does not exist." This error is distinct from a `SecurityGroupId` issue, as the validation failure would directly reference the key pair property, not a missing or null security group identifier.

The task execution role does not have permission to retrieve the secret.

Why it fails: The ECS Task Execution Role is primarily responsible for actions the ECS agent performs on behalf of the task, such as pulling container images from ECR, publishing container logs to CloudWatch, and managing network interfaces. It is distinct from the Task Role, which is assumed by the application inside the container to make AWS API calls, including retrieving secrets from AWS Secrets Manager at runtime. Therefore, a lack of secret retrieval permissions on the Task Execution Role would not prevent the application itself from accessing the secret.

The task is in a private subnet without a VPC endpoint to Secrets Manager.

Why it fails: While an ECS task running in a private subnet requires a network path to reach AWS Secrets Manager, the absence of a VPC endpoint for Secrets Manager does not automatically mean no connectivity. Tasks in a private subnet can still access public AWS service endpoints if a NAT Gateway is configured in a public subnet and the private subnet's route table is correctly configured to route outbound internet traffic through it. Therefore, simply lacking a VPC endpoint is not a definitive cause for failure if a NAT Gateway provides an alternative path.

The secret name does not match the pattern in the policy.

Why it fails: AWS IAM policies and resource-based policies often utilize wildcards in resource ARNs to grant permissions to a group of resources. If a policy's resource ARN pattern, such as arn:aws:secretsmanager:region:account:secret:my-app-secret-*, is designed to match secrets starting with "my-app-secret-", and the actual secret name adheres to this pattern, then the policy would apply. Therefore, the issue is not a mismatch in the secret name against the policy's pattern if the wildcard correctly encompasses the secret.

The bucket is in a different region than the user's default region.

Why it fails: While S3 buckets are provisioned within a specific AWS region, IAM permissions are evaluated globally across all regions where a principal might attempt to access resources. The region where a user's default settings are configured, or the region where the S3 bucket resides, does not inherently restrict or deny IAM permissions for S3 actions like s3:ListBucket. Access is determined solely by the applicable IAM and resource policies, irrespective of regional differences.

The bucket is encrypted with AWS KMS.

Why it fails: AWS KMS encryption primarily protects the data at rest within S3 objects, encrypting the object content itself. However, the s3:ListBucket action is a bucket-level permission that allows an IAM principal to view the metadata of objects within a bucket, such as their names and sizes, but not their actual encrypted content. Therefore, KMS encryption on the objects does not inherently prevent or deny the ability to list the contents of the bucket.

The IAM policy does not include the s3:GetObjectVersion action.

Why it fails: The s3:GetObjectVersion action is specifically used to retrieve a particular version of an object from a versioned S3 bucket. However, the question implies a general failure to retrieve an object, for which the s3:GetObject action is sufficient to retrieve the current version by default. Without explicit mention of needing to access a non-current object version, requiring s3:GetObjectVersion is an unnecessary and incorrect assumption for troubleshooting a basic access issue.

The Lambda function is in a different AWS account than the S3 bucket.

Why it fails: While cross-account access requires specific configuration, it is a fully supported and common use case in AWS. A Lambda function in one AWS account can successfully access an S3 bucket in another account, provided the S3 bucket policy grants the necessary permissions to the Lambda function's execution role ARN from the other account. Therefore, simply being in a different account does not inherently cause access failure if policies are correctly configured.

The IAM policy uses an incorrect resource ARN format.

Why it fails: The question stem provides no information to suggest that the IAM policy's resource ARN format is incorrect. AWS ARNs follow a well-defined structure, and developers typically use correct formats. An incorrect ARN format would usually lead to errors indicating an invalid resource or syntax, rather than a policy-based access denial, implying the resource is identifiable but access is forbidden.

The IAM policy does not allow the s3:GetObject action on the bucket.

Why it fails: The s3:GetObject action is used to retrieve the actual content of an object from an S3 bucket. However, when the goal is merely to list the objects within a bucket, only the metadata of those objects is required, which is provided by the s3:ListBucket action. Therefore, s3:GetObject is not a prerequisite for successfully listing the contents of an S3 bucket.

The IAM policy resource for s3:ListBucket should include the bucket and objects.

Why it fails: The s3:ListBucket action specifically grants permission to list the objects and their prefixes within a given S3 bucket. Its Resource element in an IAM policy must target the bucket ARN itself, such as arn:aws:s3:::your-bucket-name, not a wildcard for objects like arn:aws:s3:::your-bucket-name/*. Listing objects is an operation performed *on* the bucket, not on individual objects.

The IAM policy is missing the s3:ListAllMyBuckets action.

Why it fails: The s3:ListAllMyBuckets action provides permission to list all S3 buckets owned by the AWS account, which is a global operation. This is distinct from listing the *contents* of a specific bucket. To list objects within a particular bucket, the s3:ListBucket action is required, making s3:ListAllMyBuckets irrelevant for the task of listing objects inside a specified bucket.

Increase the read capacity units (RCU) to 2000.

Why it fails: Increasing Read Capacity Units (RCU) primarily addresses throttling issues, where requests are rejected because the provisioned throughput is exceeded. If the table is merely 'slow' due to high latency per request rather than throttling, simply increasing RCU will not inherently reduce the individual request latency. It would only increase cost without improving the response time for each read operation.

Add a Global Secondary Index (GSI) on the frequently read attribute.

Why it fails: Adding a Global Secondary Index (GSI) allows for querying data using an alternate key, enabling new access patterns. However, it does not inherently speed up existing read operations on the base table's primary key. Furthermore, maintaining a GSI adds write overhead to the main table and can introduce eventual consistency considerations for queries against the index, potentially increasing overall complexity and not directly solving 'slow' reads on the primary key.

Decrease the write capacity units (WCU) to 250.

Why it fails: Decreasing Write Capacity Units (WCU) only impacts the rate at which data can be written to the DynamoDB table. This action has no direct or indirect effect on the performance, latency, or throughput of read operations. Read and write capacities are provisioned and consumed independently, meaning a change to WCU will not alleviate slow read performance.

Change the deployment configuration to AllAtOnce.

Why it fails: Changing the deployment configuration to `AllAtOnce` merely alters the strategy by which CodeDeploy attempts to deploy to all instances simultaneously, rather than in batches. This modification does not address underlying issues such as a misconfigured agent, incorrect application revision, or permission problems that prevent *any* successful deployment. If a fundamental issue exists, deploying to all instances at once will simply result in all instances failing simultaneously, offering no diagnostic benefit.

Verify that the EC2 instance has an IAM role that allows CodeDeploy to access it.

Why it fails: The IAM role attached to an EC2 instance (instance profile) grants permissions for the CodeDeploy agent running *on* that instance to communicate with the AWS CodeDeploy service and other AWS services, such as S3 for fetching revisions. CodeDeploy itself, as a service, uses a separate service role to interact with EC2 instances (e.g., to tag them or manage Auto Scaling groups). This option incorrectly implies CodeDeploy needs an instance role to 'access' the instance, rather than the agent on the instance needing permissions to interact with the CodeDeploy service.

Increase the health check grace period on the Auto Scaling group.

Why it fails: The health check grace period in an Auto Scaling group is designed to provide newly launched instances sufficient time to initialize and pass their Elastic Load Balancer (ELB) or EC2 status checks before the Auto Scaling group considers them unhealthy. This setting is unrelated to CodeDeploy's internal deployment process or its ability to successfully deploy an application revision to an instance. CodeDeploy manages its own health checks and lifecycle hooks during a deployment, independent of the ASG's health check grace period.

Ensure the CodeDeploy agent is installed and running on the instance.

Why it fails: If the `BeforeInstall` script executed, even if it failed with an exit code 1, it confirms that the CodeDeploy agent was successfully installed, running, and actively managing the deployment on the instance. A non-running agent would prevent the deployment from even reaching the script execution phase, instead failing much earlier with an agent-related error or a timeout during the initial deployment setup. Therefore, the agent's operational status is not the root cause of a script execution failure.

Verify that the scripts location in the AppSpec file is correct.

Why it fails: The fact that the `BeforeInstall` script *ran* and produced an exit code implies that CodeDeploy successfully located and invoked it according to the AppSpec file. If the script's location in the AppSpec file were incorrect, CodeDeploy would report an error indicating it could not find or execute the specified script, rather than reporting an exit code from the script's execution itself. Thus, the AppSpec file's script path is validated by the script's attempt to run.

Check that the instance's IAM role has permissions to download the revision.

Why it fails: The `BeforeInstall` lifecycle hook executes *after* the application revision has been successfully downloaded from the S3 bucket or GitHub repository to the instance. If the deployment reached the `BeforeInstall` phase, it confirms that the instance's IAM role possessed the necessary `s3:GetObject` or equivalent permissions to retrieve the revision. A permissions issue for downloading would cause the deployment to fail during the 'Download Bundle' stage, long before any install scripts are invoked.