Troubleshooting Scenario Questions

Increase the stack creation timeout to allow more time for the database to be created.

Why it fails: Increasing the stack creation timeout only adjusts how long CloudFormation waits before declaring the overall stack creation failed; it does not address resource-level validation errors that occur immediately during the DBInstance provisioning. If the database resource returns a CREATE_FAILED status quickly, the problem is almost always an invalid configuration parameter (e.g., unsupported DB instance class, wrong engine, or misconfigured DB subnet group), not a lack of time. Timeout changes would leave the root cause unresolved and would merely delay the same failure.

Verify that the VPC has at least two public subnets in different Availability Zones.

Why it fails: Verifying that the VPC has at least two public subnets is incorrect because an Amazon RDS database does not require public subnets; it can be deployed entirely within private subnets. The actual networking requirement is that the DB subnet group contains at least one subnet in two or more Availability Zones, and those subnets can be private. Public subnets with an Internet Gateway are only necessary if you explicitly enable Public Access on the RDS instance, which is neither required by CloudFormation templates nor a common cause of stack creation failure.

Use the Amazon RDS console to check if a DB instance with the same identifier already exists.

Why it fails: Using the RDS console to check for an existing DB instance with the same identifier is a possible but indirect and incomplete approach. A duplicate identifier would indeed cause a failure, but that is only one of many potential causes, and CloudFormation stack events already capture the specific error message from the RDS CreateDBInstance call. Furthermore, DB instance identifiers are scoped per AWS account and region, so collisions are less common; even if one exists, you would still need to examine stack events to confirm the failure reason and decide whether to change the identifier or delete the old instance.

The launch template has insufficient IAM permissions to create instances.

Why it fails: Insufficient IAM permissions in the launch template would cause the RunInstances API call to be rejected with an 'UnauthorizedOperation' or 'AccessDenied' error, not a quota-based message. The instance would never be created because the call fails at the authorization layer, and the Auto Scaling activity history would show an IAM failure reason with a CloudTrail entry. Since the error described in the scenario is explicitly a quota for running instances being exceeded, permissions issues are ruled out.

The VPC subnet does not have enough available IP addresses.

Why it fails: If the VPC subnet had no available IP addresses, the error would be 'Insufficient IP addresses available in subnet' with an InsufficientFreeAddressesInSubnet code, which occurs during the network interface allocation phase. This is a different failure mode than a quota exceeded error; the instance launch fails because the subnet's CIDR is exhausted, not because of account-wide EC2 limits. The scenario explicitly mentions a quota for running instances, so this option is incorrect.

There is an instance that is not passing health checks, preventing new instances.

Why it fails: An instance failing health checks would trigger Auto Scaling to terminate the unhealthy instance and replace it, not block the launch of new instances. The health check failure would cause a lifecycle action and replacement, and any failure to launch a replacement would be due to other constraints like instance limits or network issues. The error message would reflect the underlying launch failure (e.g., quota), not the health status itself, so this option does not explain the quota error.

The EC2 instance type is not supported in the region

Why it fails: An unsupported instance type would be rejected by the EC2 API during the CreateInstance call, so CloudFormation would record the resource as CREATE_FAILED with an error such as InvalidInstanceType or 'The requested configuration is currently not supported'. That is a validation failure, not the 'Resource creation cancelled' state, which indicates the stack operation itself was interrupted by a timeout or user cancellation. Thus this option cannot explain the observed CloudFormation event.

The IAM role attached to the instance lacks permissions

Why it fails: Permissions problems with the attached IAM instance profile manifest as AccessDeniedException errors on API calls made by the instance, such as uploading logs or invoking cfn-signal; these appear as a CREATE_FAILED status on the relevant resource or a rollback trigger, not as a cancelled resource event. CloudFormation does not cancel the creation of an EC2 instance merely because the instance's IAM role lacks permissions—that would either fail at a dependent resource or leave the instance running with an unhealthy configuration. Therefore this is not the cause of a 'Resource creation cancelled' message.

The UserData script failed due to a syntax error

Why it fails: A syntax error in UserData occurs after the EC2 instance has launched, and CloudFormation only cares about the successful creation of the instance itself, so the EC2 resource would still transition to CREATE_COMPLETE. Unless you have a WaitCondition or a CreationPolicy with cfn-signal, CloudFormation never inspects the output of the UserData script, and even a failing script merely logs an error inside the instance. The 'Resource creation cancelled' status is a CloudFormation resource-lifecycle event triggered by stack timeout or cancellation, not by a script failure taking place in the guest OS.

Review the Auto Scaling group scaling policies.

Why it fails: Auto Scaling group scaling policies govern capacity changes and instance lifecycle actions; they do not participate in the ALB's per-request idle timeout mechanism. A 504 Gateway Timeout is emitted by the ALB when it has successfully forwarded a request to a healthy target but the target fails to return a response before the timeout expires, which can happen even when the group's desired capacity is perfectly sized. Troubleshooting scaling policies would only be relevant if the symptom were insufficient capacity causing connection failures or increased latency, not a timeout already being generated by the ALB.

Verify the target group health checks are passing.

Why it fails: Target group health checks are pass/fail probes that control whether a target is placed in service; a target that fails health checks is deregistered and typically causes a 503 Service Unavailable response from the ALB, not a 504. A 504 specifically requires the target to have accepted the TCP connection and be considered healthy enough to receive traffic, yet fail to complete the HTTP response within the ALB's idle timeout. Therefore, verifying health checks confirms availability but does not diagnose slow application responses, which is the actual cause of 504s.

Check security group rules for the ALB.

Why it fails: Security group rules govern which hosts can initiate connections to the ALB or to the target instances; a misconfiguration would result in connection timeouts, refused connections, or health-check failures, usually surfacing as 502 or 503 errors. A 504 is not a connection-level rejection — it means the ALB successfully established a connection and forwarded the request, but the target did not send response headers before the ALB's idle timeout expired. Checking security groups only validates network reachability, not the backend application's time-to-response, so it would not reveal the hanging request or slow target.

The Auto Scaling group has too many instances, causing increased network overhead

Why it fails: This misidentifies the scaling dynamic: an Application Load Balancer distributes each request to the least loaded healthy target, so adding instances generally reduces per-instance queueing and lowers response latency, not increases it. Only if the application performs heavy inter-instance communication or relies on a shared stateful store would more nodes add meaningful overhead, but that is not inherent to Auto Scaling. Network overhead from additional instances is negligible compared to the latency caused by a slow downstream dependency.

The ALB is routing requests to instances in different Availability Zones, increasing latency

Why it fails: For an Application Load Balancer, cross-zone load balancing is enabled by default, meaning each ALB node actively distributes requests to healthy targets across all Availability Zones in the region. Within a region, AZ-to-AZ traffic travels over AWS's redundant, high-bandwidth backbone—typically adding less than a millisecond—so it cannot be the source of a perceptible slowdown. A request would have to cross regions to experience real network latency impact, which is not the case here.

The ALB idle timeout is set too low, causing connections to be dropped

Why it fails: ALB idle timeout controls how long a connection may remain inactive before the load balancer closes it; setting it too low would terminate idle keep-alive connections, causing clients to see connection resets or 502/504 errors on the next request—not high latency during an active request. The timeout does not affect how quickly an in-flight request is processed, since it is based on inactivity, not request duration. Therefore, a low idle timeout cannot explain a slow web application; it would manifest as sporadic connectivity failures instead.

Increase the instance size of the primary database to handle the load.

Why it fails: Scaling up the primary RDS instance size only increases the compute and memory capacity of the single writer node. It does not offload read traffic because all read requests still hit the same primary instance, and the Multi-AZ standby is not used for reads. This approach addresses raw capacity limits but does not distribute the read workload, so the primary can remain the bottleneck and the cost scales linearly without any architectural benefit.

Modify the application to use separate endpoints for read and write operations.

Why it fails: Modifying the application to use separate endpoints for read and write operations would require code changes and manual management of read replicas. Even then, you would have to provision and maintain replica instances, handle replica lag, and implement custom load balancing. RDS Proxy is specifically designed to avoid this complexity by automatically routing statements to the correct endpoint based on query type, so a manual split is an unnecessary development effort that does not leverage managed AWS capabilities.

Create a new Multi-AZ cluster with a read-only endpoint.

Why it fails: A Multi-AZ DB instance in RDS for MySQL provides an automatic failover standby that is not accessible for read traffic, and it does not have a read-only endpoint. Creating another Multi-AZ deployment with a read-only endpoint is not possible for classic Multi-AZ because the standby is a synchronous replica used only for durability and failover. Even if you created a Multi-AZ DB cluster, you would still need to redirect read queries appropriately, whereas RDS Proxy provides built-in read/write splitting without such infrastructure changes.

Check the CloudWatch agent status using the CloudWatch agent status command.

Why it fails: The `amazon-cloudwatch-agent-status` command only reports whether the agent process is running and its basic operational state (e.g., version, config path). It does not surface detailed transport or delivery errors, IAM permission failures, or malformed log config entries. In a pure troubleshooting scenario where logs are not appearing, this command provides no diagnostic path to identify why a log stream is missing or why the agent cannot reach CloudWatch Logs.

Configure a Lambda function to poll the CloudWatch agent for logs.

Why it fails: A Lambda function cannot 'poll' the CloudWatch agent itself because the agent is a local service and has no API endpoint for remote queries. The agent actively pushes log events to CloudWatch Logs via the PutLogEvents API; it does not expose a pull mechanism. While Lambda could be used to inspect CloudWatch Logs API metrics or query already-delivered log streams, that would only confirm the absence of logs after the fact and does not diagnose an agent-side problem such as a bad config or connectivity issue. This approach adds complexity without addressing the agent's failure mode.

Verify that the EC2 instances have an SQS queue configured for log delivery.

Why it fails: The CloudWatch agent does not use Amazon SQS at any point in its log delivery flow. Logs are gathered from local files/streams and sent directly to the CloudWatch Logs service using the AWS SDK (PutLogEvents). Requiring an SQS queue on the EC2 instances implies a queue-based pipeline that does not exist in this architecture. Misdiagnosing the problem as a missing SQS queue would lead you away from the agent's actual log file and error messages, which are the correct sources of truth for delivery failures.

The script is not included in the 'files' section of the appspec.yml

Why it fails: The script's location relative to the 'files' section is irrelevant to execution. In appspec.yml, the 'files' section lists source files to be copied onto the instance, while lifecycle scripts must be declared under the 'hooks' section, which maps each event (e.g., AfterInstall, ApplicationStart) to a run script. If a script is omitted from the 'hooks' section, it would not be executed at all, but that would not generate a non-zero exit error; it would simply not run.

The script requires dependencies that are not installed on the instance

Why it fails: Missing dependencies can cause a script to fail, but the deployment failure is caused by the script returning a non-zero exit code, not by the absence of the dependency itself. CodeDeploy does not inspect why a script failed; it only observes the exit status. Moreover, missing dependencies typically produce errors like 'command not found' or 'module not found' at runtime—these still result in a non-zero exit, but the immediate signal CodeDeploy reacts to is the exit code, not the dependency issue.

The script is not owned by the root user

Why it fails: File ownership does not impact whether a script can be executed; only the file's permission bits and the executing user's access rights matter. In CodeDeploy, the 'runas' attribute in the hooks section specifies the user under which the script runs, and as long as that user has execute permission (or the file is world-executable), the script runs regardless of whether the file is owned by root. An ownership mismatch would only matter if the 'runas' user lacked read/execute permission on the script, which is a permissions issue, not an ownership issue.

Add annotations to the traces for better filtering.

Why it fails: Annotations in X-Ray are user-defined key-value pairs attached to segments or subsegments for the purpose of indexing, searching, and filtering traces; they do not capture or display timing data for the network hops between services. Adding annotations would only help you retrieve traces that meet certain criteria after the fact, but it provides no visibility into where latency is introduced along the service boundary — making it an auxiliary tool for trace organization, not a diagnostic aid for slow inter-service communication.

View the raw segments of the upstream service.

Why it fails: Viewing the raw segments of the upstream service shows the internal work performed within that service (e.g., database queries, local processing, and subsegment timing) but stops at the point where the service sends its response. It does not include the time spent on the wire or the network round-trip between the upstream and downstream services, so it cannot reveal connection-level latency such as packet loss, DNS resolution issues, or load balancer delays that occur outside the service's own execution context.

Adjust the sampling rules to capture more traces.

Why it fails: Adjusting sampling rules changes the proportion of requests that are recorded by X-Ray, which controls trace volume and cost but has no bearing on the analysis of a specific latency problem in an already-captured trace. Even if you increase the sampling rate, you are simply recording more of the same type of trace data; the information needed to diagnose the slow connection (the per-http-edge latency metrics) is still only available through the trace map, and changing sampling rules will not add any dimensions of diagnostic visibility to the existing traces.

The build environment ran out of disk space.

Why it fails: An out-of-disk-space condition would not appear as a Gradle exit status 1 from `./gradlew build`; it would instead generate filesystem-level errors such as 'No space left on device' or 'Failed to write...' when Gradle tries to download dependencies or write its cache. Such an environment failure would likely interrupt the process earlier, for example during the `install` or `pre_build` phase, not as a clean non-zero exit from the build command itself.

The artifact packaging step failed.

Why it fails: The artifact packaging step, such as creating a ZIP/tarball or copying outputs to a designated directory, is defined in the `post_build` phase, which executes only after the `build` phase has exited successfully. Because `./gradlew build` stopped with status 1, the pipeline aborted during the `build` phase, meaning `post_build` never ran and therefore cannot be the cause of this failure.

The linting step failed.

Why it fails: In this buildspec, the `build` phase runs `./gradlew build`, a Gradle task that compiles code and runs tests; a separate linting task would be invoked explicitly (e.g., `./gradlew lint`) in `pre_build` or as its own command. A lint failure would produce lint-specific output and its own non-zero exit from that distinct command, not the generic status 1 from `./gradlew build`. Thus the linting step is not what caused the failure shown.

Configure the build to use a VPC with a NAT gateway

Why it fails: Adding the build to a VPC and routing traffic through a NAT gateway only changes the network path for outbound requests; it neither stores nor reuses Maven artifacts between executions. Each build still performs a cold download from the external repository, and the VPC/NAT overhead (NAT processing, subnet routing, IAM roles) can even increase build latency. The bottleneck here is dependency re-downloading, not network reachability, so a NAT gateway does not address the root cause.

Use AWS CodeArtifact as a proxy for Maven dependencies

Why it fails: CodeArtifact as an upstream proxy does cache packages on its own infrastructure, but that cache is remote: every CodeBuild run must still pull the dependencies over the network into the ephemeral build container, so it does not eliminate the download step or the associated network time. For Maven specifically, the local build client likely must resolve and fetch coordinates through CodeArtifact endpoints each time, and if the cache hasn't already been populated by a prior run, it still contacts the upstream repository. While CodeArtifact can reduce repeated public repository traffic, enabling local disk caching is the more direct way to avoid re-fetching dependencies on every PR build.

Change the compute type to general1.2xlarge

Why it fails: Changing the CodeBuild compute type to general1.2xlarge increases the vCPU and memory allocation, which helps CPU- or memory-bound build phases such as compilation and test execution. However, when the reported slowness comes from downloading dependencies, the limiting factor is network bandwidth and the round-trip latency of remote repositories — both of which remain unchanged on a larger instance. You will pay for more compute capacity without addressing the actual bottleneck of repeated dependency downloads.

The Deny statement condition incorrectly uses StringNotEquals, which denies all instances except those with the Production tag.

Why it fails: The Deny statement uses StringNotEquals to deny ec2:RunInstances for non-Production requests, but its action is limited to RunInstances and its condition evaluates only at launch time. It never applies to ec2:StopInstances, so it cannot 'deny all instances' for stop operations. Further, StringNotEquals is a legitimate condition operator; the issue is that the stop action has no Allow, not that the operator is malformed.

The Deny statement includes ec2:StopInstances implicitly because stop is a termination action.

Why it fails: IAM action coverage is explicit; ec2:StopInstances is a distinct API action from ec2:RunInstances and ec2:TerminateInstances, so it is not included implicitly by a Deny that mentions only RunInstances. Stopping an instance is not a termination action in IAM terms, and resource/condition elements in that Deny apply only to instance launches. Thus this rationale misreads the statement's scope and cannot explain the stop failure.

The policy does not prevent stopping instances with the Development tag; the failure must be caused by another policy or service control policy.

Why it fails: In IAM, actions not explicitly allowed are implicitly denied, so the absence of ec2:StopInstances in the Allow statement alone prevents stopping instances regardless of their Development tag. There is no need to invoke another identity-based policy or an SCP; while such policies can broaden or further restrict permissions, they are not required to produce the failure. Therefore the tag-based reasoning does not identify the actual cause.

Confirm that the load balancer's health check target is pointing to the correct port and path on the instances.

Why it fails: An incorrect health check target on the load balancer would cause the ELB to mark the instances as unhealthy and trigger Auto Scaling to terminate and replace them, not leave them running. The observed symptom of persistent running instances would not result from a mismatched port or path; instead you would see instance churn and replacement activity in the scaling group.

Ensure the instances are sending health check requests to the load balancer.

Why it fails: Health checks in an Application Load Balancer or Classic Load Balancer are sent by the load balancer to the instances, not by the instances to the load balancer. Instances simply listen on the health check port and return a response; they do not initiate these probes. Therefore 'instances sending health check requests' is not a mechanism that could affect termination, making this option irrelevant to the problem.

Verify that the security group for the instances allows inbound traffic from the load balancer on the health check port.

Why it fails: Security groups control traffic flow but the health check is initiated by the ELB; if the security group blocks the ELB, the health check would fail and the instance would be marked unhealthy, leading to termination.

Modify the Auto Scaling group to use a larger instance type.

Why it fails: Modifying the Auto Scaling group to use a larger instance type is a reactive capacity change rather than a diagnostic action; it assumes the problem is insufficient vCPUs without verifying whether the instance is actually CPU-constrained. It also involves updating the launch template and potentially recreating instances, which is disruptive and costly, and it may mask a temporary spike that a credit balance review would have identified. This should be a later remediation step, not the first response to an alarm.

Check the ALB's HTTP 5xx error rate metric for the target group.

Why it fails: Checking the ALB's HTTP 5xx error rate for the target group is a valid health indicator but it measures application layer failures, not host CPU utilization. A 5xx can result from application bugs, timeouts, or connection limits and may occur even when CPU utilization is normal, so it cannot explain why the CPU alarm fired. The alarm is specifically about resource consumption on the EC2 instance, making instance-level metrics the relevant diagnostic source.

Immediately increase the desired capacity of the Auto Scaling group.

Why it fails: Immediately raising the desired capacity of the Auto Scaling group is a blind reaction that increases cost and can create a false sense of resolution without identifying the underlying cause. If the problem is CPU credit exhaustion on a single unexpectedly busy instance, adding more instances will not fix the existing instance's throttling unless scaling policies and health checks force a replacement. Proper troubleshooting requires reading the instance's CloudWatch metrics first, because an alarm that goes off while utilization is low may indicate an alarm threshold problem rather than a true capacity shortage.

The canary schedule and frequency.

Why it fails: The schedule determines how often the canary runs, but each run is an independent invocation with its own timeout budget. A more frequent schedule does not alter the per-run timeout threshold or exhaust resources; the canary's execution duration is constrained by the configured timeout value, not by the schedule frequency. Therefore, schedule and frequency are not indicators of why a specific canary run is timing out.

CloudWatch Logs retention policy for the canary logs.

Why it fails: The retention policy dictates how long log events are kept in CloudWatch Logs before they are automatically deleted, which is purely a data lifecycle setting. It does not influence the runtime behavior of the canary, its execution timeout, or the success or failure of individual runs. While deleting logs could hinder post-hoc investigation, it cannot cause the timeout error itself or change the canary's operational status.