Troubleshooting and Optimization practice questions

Decrease the reserved concurrency of the Lambda function to 10

Why wrong: Reducing concurrency limits the number of concurrent executions but does not increase the capacity of the DynamoDB table, so throttling may still occur with fewer but still load-intensive requests.

Enable DynamoDB Accelerator (DAX) for caching reads

Why wrong: DAX only caches reads, not writes, and does not address write throttling. Also, it adds complexity.

Enable auto scaling on the DynamoDB table

Why wrong: Auto scaling adjusts capacity over time but requires a starting point; with only 5 units, it may not react fast enough to sudden spikes. Manual increase is more immediate.

Increase the RCUs and WCUs of the table to 50 each

Why wrong: This may reduce throttling temporarily, but if traffic spikes beyond 50, throttling will recur. It also requires ongoing management of capacity, whereas on-demand is simpler.

Implement a DynamoDB Accelerator (DAX) cluster for caching reads

Why wrong: DAX helps reduce read latency and offload read capacity from the table, but it does not solve write throttling. The error includes writes, and DAX does not help with writes.

Increase Lambda function memory to 1024 MB

Why wrong: Increasing memory improves CPU and network performance but does not affect DynamoDB capacity limits. It will not reduce throttling errors caused by DynamoDB.

The ALB's idle timeout is set too low, causing dropped connections

Why wrong: Idle timeout affects persistent connections but does not cause 503 errors due to insufficient healthy hosts.

The Auto Scaling group's maximum capacity of 10 is insufficient

Why wrong: With 10 instances, the group should be able to handle high traffic; the issue is more about the speed of scaling, not the maximum.

The health check grace period is preventing instances from being marked healthy

Why wrong: Grace period gives new instances time to start; it would cause a delay in recognizing health, but not the intermittent loss of all healthy hosts after a scaling event.

Increase the function timeout to the maximum allowed (900 seconds is already the max)

Why wrong: The Lambda function timeout maximum is 900 seconds (15 minutes). The function already reaches this limit; increasing timeout is not possible.

Use S3 Select to filter columns and rows before invoking the Lambda function

Why wrong: This would reduce the data size but requires code changes to integrate S3 Select, which violates the 'without code changes' constraint.

Increase the batch size of the S3 event notification to invoke the function with multiple files

Why wrong: The function processes one file per invocation; batch size affects how many file events are sent per invocation, but the function still handles one file at a time.

Increase the batch size to 500.

Why wrong: If the function is already close to timeout with a batch size of 100, increasing the batch size will likely cause it to timeout more often, worsening throughput.

Increase the parallelization factor to 10.

Why wrong: While this allows up to 100 concurrent invocations, each invocation still takes near 5 minutes. The total records processed per second may not increase significantly if each batch takes the same time.

Split the stream into more shards.

Why wrong: Adding more shards increases the number of concurrent Lambda invocations, but each invocation still processes slowly. The throughput per shard remains low, so the overall throughput may not improve enough.

The Lambda function's concurrency is set too high, causing race conditions.

Why wrong: High concurrency can cause race conditions but not duplicate event delivery. Duplicate events originate from S3, not from concurrency.

The DynamoDB table does not have a primary key that prevents duplicates.

Why wrong: Even with a primary key, the UUID generated per invocation differs, so each duplicate event creates a new item. The real issue is lack of idempotency, not table schema.

The S3 bucket is configured with versioning, causing multiple object creation events.

Why wrong: Versioning can generate multiple events (e.g., PutObject and CopyObject) for the same object, but the most common cause of duplicate processing is the 'at least once' delivery of notifications.

Increase the batch size to 1000

Why wrong: A larger batch size may increase execution time further, potentially causing the function to exceed the timeout. It does not address the per-record processing bottleneck.

Increase the memory of the Lambda function

Why wrong: Increasing memory may improve performance if the function is CPU-bound, but it is not guaranteed and increases cost. The function is already near its timeout, indicating a potential bottleneck in record processing logic.

Split the stream into more shards

Why wrong: Splitting the stream increases the number of shards, leading to more Lambda invocations but also higher Kinesis costs. This is less cost-effective than increasing the parallelization factor.

The health check interval is set too long, causing the ALB to consider unhealthy instances as healthy.

Why wrong: An overly long health check interval may delay the detection of unhealthy instances, but the issue described involves instances in 'Draining' state, which are known to be deregistering. Health checks are not the primary cause.

Cross-zone load balancing is disabled, so the ALB is routing requests to instances that are already draining.

Why wrong: Cross-zone load balancing affects traffic distribution across availability zones but does not cause 502 errors during connection draining. The target group still includes draining instances regardless of cross-zone setting.

The Auto Scaling group's minimum size is too small, causing the ALB to have no healthy targets.

Why wrong: If the minimum size is too small, there might be no healthy targets at all, leading to 503 errors, not 502. The problem is specific to connections being terminated during draining.

Increase reserved concurrency to 10.

Why wrong: Increasing reserved concurrency would reduce throttling but directly contradicts the requirement to not increase reserved concurrency.

Change the function timeout to 15 minutes.

Why wrong: Increasing timeout does not reduce the number of invocations or throttling. The function may still be throttled due to concurrency limits.

Enable SQS short polling to reduce latency.

Why wrong: Short polling returns results immediately if messages are available, but it does not affect the number of invocations or throttling. It may even increase the number of empty responses.

The ALB's idle timeout is too short, causing connections to be dropped before the application responds.

Why wrong: An idle timeout that is too short would cause 504 errors, not 502. Also, the issue is specifically during rolling updates, not during steady state.

The ALB's connection draining timeout is set to 0 seconds, causing connections to be dropped immediately when deregistering targets.

Why wrong: A connection draining timeout of 0 would immediately stop new connections to draining targets, but existing connections would still be allowed to complete. It would not cause 502 errors; it might cause 504 if connections are not completed in time. More importantly, the ALB's connection draining is separate from ECS task draining.

The ECS deployment circuit breaker is incorrectly configured to roll back on health check failures.

Why wrong: The circuit breaker rolls back a failed deployment, but it does not cause 502 errors during the update. The errors occur before the rollback is triggered.

The RDS Proxy target group is not configured with the correct DB instance

Why wrong: If misconfigured, connections would fail immediately, not cause gradual exhaustion.

The Lambda function's execution role is missing the rds-db:connect permission

Why wrong: Missing permissions would result in consistent connection failures, not gradual latency increase.

The VPC does not have a NAT Gateway for outbound traffic

Why wrong: NAT Gateway is needed for internet access, not for connecting to RDS Proxy within the same VPC.

Increase the Lambda function's timeout to 10 minutes.

Why wrong: This would allow the function to run longer but does not address the DynamoDB throttling. The function will likely still time out due to repeated retries.

Increase the Lambda function's memory allocation to 3008 MB.

Why wrong: More memory can improve CPU performance but does not resolve DynamoDB throttling. The bottleneck is the write capacity, not compute power.

Use an Amazon SQS FIFO queue instead of a standard queue for the Lambda trigger.

Why wrong: Changing the queue type does not address the DynamoDB throttling. It may affect ordering but not throughput to DynamoDB.

Increase the read capacity units (RCU) of the base table.

Why wrong: The base table's RCU does not affect the GSI's read capacity. The GSI has its own provisioned capacity; throttling on the GSI is not resolved by changing the base table.

Change the GSI projection to ALL.

Why wrong: Changing projection to ALL increases write costs and may reduce the number of reads needed, but it does not increase the GSI's read capacity. Throttling would persist.

Create a Local Secondary Index instead.

Why wrong: A Local Secondary Index shares the read capacity of the base table, which is already sufficient. However, it does not support queries across multiple partition keys as a GSI does, and may not solve the throttling issue.

Increase the Lambda function's memory allocation.

Why wrong: Increasing memory may provide more CPU, but it does not solve the DynamoDB throttling issue. The function is still likely to be throttled when calling the table, leading to retries and increased execution time.

Increase the Lambda function's reserved concurrency.

Why wrong: More concurrent invocations would only increase the number of requests to DynamoDB, potentially worsening throttling. It does not reduce execution time for each invocation.

Increase the Lambda function's timeout.

Why wrong: Increasing the timeout only allows the function to run longer, but it does not address the cause of increased execution time (DynamoDB throttling). The function may still time out if throttling persists.

Increase reserved concurrency to 100

Why wrong: This increases the number of concurrent invocations, which can lead to more throttling if the function cannot handle the rate. Without increasing batch size, each invocation processes only 10 messages, so throughput increase is limited.

Increase batch size to 100

Why wrong: This reduces the number of invocations, but with only 5 reserved concurrency, the total throughput may still be limited. The function can process 100 messages per invocation, but the concurrency cap restricts how many invocations can run simultaneously.

Decrease batch size to 1 and increase reserved concurrency to 50

Why wrong: Decreasing batch size increases the number of invocations, which would increase throttling risk under the same concurrency limits. This approach is counterproductive.

Increase the reserved concurrency of the Lambda function

Why wrong: This raises the ceiling, but the function is already being throttled at 100. Increasing reserved concurrency might help if the account limit allows, but it does not address the root cause of high invocation volume.

Decrease the batch window of the event source mapping

Why wrong: A smaller batch window causes Lambda to poll more frequently, increasing invocations and likely worsening throttling.

Add a dead-letter queue (DLQ) for the Lambda function

Why wrong: A DLQ captures messages that are not successfully processed, but does not reduce the number of invocations or throttling.

Increase the function's timeout to 60 seconds

Why wrong: Increasing timeout may prevent timeouts, but it does not improve efficiency and could increase costs. Also, it does not address the root cause of slow external calls.

Reduce the batch size to 5

Why wrong: Reducing batch size would lower the number of messages processed per invocation, decreasing throughput and potentially increasing cost due to more invocations.

Increase the SQS visibility timeout to 60 seconds

Why wrong: Visibility timeout controls how long a message is hidden from other consumers after being received. It does not affect the Lambda function's execution timeout.

The Lambda function is experiencing cold starts due to low invocation frequency.

Why wrong: Cold starts increase Lambda duration, not integration latency. Integration latency is the network time, not the function execution time.

The DynamoDB on-demand capacity is throttling requests.

Why wrong: Throttling would cause Lambda to retry, increasing its duration and possibly leading to timeouts, but it would not directly affect integration latency.

The external API call is adding network latency due to its geographic location.

Why wrong: Latency from the external API call is included in the Lambda function duration, not in integration latency. Integration latency specifically measures API Gateway to Lambda round-trip time.

Increase the function's timeout

Why wrong: Increasing timeout might prevent the error but does not address the underlying slowness; it is better to first try increasing memory.

Increase the function's reserved concurrency

Why wrong: Concurrency controls the number of simultaneous invocations, not the execution time of a single invocation.

Increase the S3 event notification batch size

Why wrong: S3 event notifications do not have a batch size; each event triggers a separate invocation.

Increase the number of replicas

Why wrong: Replicas improve read throughput and provide high availability, but they do not increase the total memory capacity of the cluster. Evictions will continue if the primary node is memory-constrained.

Enable cluster mode and add more shards

Why wrong: Enabling cluster mode with additional shards increases total memory, which can reduce evictions. However, this requires application support for Redis Cluster and may involve significant code changes. It is not the most straightforward solution.

Increase the TTL of cached items

Why wrong: Increasing TTL may cause items to stay in cache longer, but if memory is full, new items will still evict old ones. It might even increase evictions if the cache is over capacity.