Troubleshooting and Optimization practice questions

Increase the batch size to 500.

Why wrong: If the function is already close to timeout with a batch size of 100, increasing the batch size will likely cause it to timeout more often, worsening throughput.

Increase the parallelization factor to 10.

Why wrong: While this allows up to 100 concurrent invocations, each invocation still takes near 5 minutes. The total records processed per second may not increase significantly if each batch takes the same time.

Split the stream into more shards.

Why wrong: Adding more shards increases the number of concurrent Lambda invocations, but each invocation still processes slowly. The throughput per shard remains low, so the overall throughput may not improve enough.

Increase batch size to 100

Why wrong: This option is incorrect because the maximum batch size configurable for an AWS Lambda function triggered by an Amazon SQS queue is 10 messages. Attempting to set the batch size to 100 would result in a configuration error for the event source mapping, preventing the Lambda function from processing messages with this invalid setting. Therefore, this change cannot be implemented to increase throughput.

Increase both batch size to 100 and reserved concurrency to a higher value

Why wrong: This option is incorrect primarily because the batch size for an SQS-triggered Lambda function cannot exceed 10 messages per invocation. While increasing reserved concurrency can improve overall throughput by allowing more simultaneous function executions, the proposed batch size of 100 is an invalid configuration for the SQS event source mapping. Consequently, this combined approach is unfeasible and would fail to deploy or operate as intended.

Decrease batch size to 1 and increase reserved concurrency to 50

Why wrong: This option is incorrect because decreasing the batch size to 1 message per invocation significantly increases the overhead associated with processing each individual message. Although increasing reserved concurrency allows for more simultaneous Lambda invocations, the reduced efficiency of processing only one message at a time per function instance often results in lower overall message throughput and higher operational costs compared to processing messages in larger batches up to the maximum of 10.

Increase the batch size of the SQS event source mapping

Why wrong: When Lambda processes SQS messages, it fetches them in batches. Increasing the batch size allows each Lambda invocation to process more messages simultaneously. This reduces the total number of Lambda function invocations required to process a given volume of SQS messages, directly lowering the concurrent execution count and mitigating throttling issues.

Decrease the batch window of the event source mapping

Why wrong: The batch window defines the maximum time Lambda waits to gather messages before invoking the function, up to the batch size limit. Decreasing this window means Lambda will invoke the function more frequently, even with fewer messages, to avoid waiting. This increases the total number of Lambda invocations and concurrent executions, exacerbating the throttling problem rather than solving it.

Add a dead-letter queue (DLQ) for the Lambda function

Why wrong: A Dead-Letter Queue (DLQ) for a Lambda function is designed to capture messages that fail processing after all retry attempts. While crucial for robust error handling and preventing message loss, implementing a DLQ does not alter the rate at which Lambda is invoked or the number of concurrent executions. Therefore, it offers no solution for addressing or reducing Lambda throttling.

Use async HTTP calls with a callback

Why wrong: Employing asynchronous HTTP calls with a callback mechanism allows the Lambda function to initiate external requests without blocking its execution thread. This non-blocking I/O pattern enables the function to potentially handle other tasks or simply not consume billed compute time while awaiting the external service's response. Once the response is received, the callback function is invoked to process the results, significantly reducing the overall execution duration for I/O-bound operations and preventing timeouts.

Increase the function's timeout to 60 seconds

Why wrong: Increasing the function's timeout merely extends the maximum allowable execution duration, acting as a temporary fix rather than an optimization. While it might prevent premature termination, it does not accelerate the underlying slow external HTTP calls or improve the function's efficiency. This approach can lead to higher AWS costs, as the function is billed for the entire duration it is active, even if it's idly waiting for a slow external dependency.

Increase the SQS visibility timeout to 60 seconds

Why wrong: Increasing the SQS visibility timeout to 60 seconds controls how long a message remains hidden from other consumers after it has been received by one. This setting is crucial for preventing duplicate processing in case a consumer fails before deleting a message, but it has no direct impact on the Lambda function's execution performance or its ability to complete external HTTP calls within its own configured timeout. It is a queue-level setting unrelated to the compute duration of the function itself.

The DynamoDB on-demand capacity is throttling requests.

Why wrong: DynamoDB throttling happens when the request rate to a table exceeds its configured capacity, whether provisioned or on-demand. When a Lambda function encounters DynamoDB throttling, it typically retries the request or fails, which directly increases the Lambda function's execution duration or results in errors. This issue is internal to the Lambda function's workload and does not affect the network transfer time between API Gateway and the Lambda service, which is what `IntegrationLatency` specifically tracks.

The external API call is adding network latency due to its geographic location.

Why wrong: Latency introduced by an external API call is incurred during the execution of the Lambda function's code. This time contributes directly to the Lambda function's total `Duration` metric, as it is part of the work performed within the function's execution environment. API Gateway's `IntegrationLatency` metric, however, is precisely defined as the time taken for API Gateway to send the request to the integrated backend (Lambda) and receive the response, excluding any processing or external calls made by the Lambda function itself.

The API Gateway request/response payload size is large, increasing transfer time.

Why wrong: API Gateway's `IntegrationLatency` metric measures the time taken for API Gateway to establish a connection, send the request payload to the backend integration (like Lambda), and receive the response payload back. When the request or response payload size is large, the network serialization, transfer, and deserialization times increase proportionally. This directly extends the duration of data transmission over the network, thereby increasing the `IntegrationLatency` even if the Lambda function processes the data very quickly.

The function's reserved concurrency is too high, causing excessive write requests

Why wrong: Reserved concurrency limits the maximum number of concurrent executions for a Lambda function. If the function uses 10 WCUs per invocation and has a reserved concurrency of 10, the maximum theoretical write capacity consumed by the function would be 100 WCUs (10 invocations * 10 WCUs/invocation). This usage is well below the provisioned 500 WCUs for the DynamoDB table, indicating that the reserved concurrency itself is not the cause of excessive write requests or throttling.

The table's read capacity is insufficient, causing write throttling

Why wrong: DynamoDB manages read capacity units (RCUs) and write capacity units (WCUs) independently. Insufficient read capacity would lead to throttling for read operations, resulting in ProvisionedThroughputExceededException for reads, not writes. Write throttling, which is the observed issue, is exclusively governed by the available write capacity and distribution, making read capacity irrelevant to this specific problem.

The function is exceeding the DynamoDB item size limit of 400 KB

Why wrong: DynamoDB imposes a maximum item size limit of 400 KB. If a Lambda function attempts to write an item larger than this limit, DynamoDB would return a ValidationException error, specifically indicating an item size exceeding the maximum allowed. Since the items are approximately 1 KB, they are well within this limit, and exceeding it would not manifest as a ProvisionedThroughputExceededException, which is indicative of capacity issues.

Increase the number of shards

Why wrong: Each Kinesis shard is processed by a single, dedicated Lambda invocation at a time. Increasing the number of shards directly increases the maximum parallel processing capacity for the Kinesis stream, allowing more concurrent Lambda invocations. This directly reduces the IteratorAge by enabling faster consumption of the stream and mitigating throttling due to insufficient concurrency.

Decrease the maximum record age

Why wrong: The MaximumRecordAgeInSeconds setting for a Kinesis event source mapping dictates how long Lambda will attempt to process a record before discarding it. Decreasing this value does not accelerate the processing of current or new records; instead, it causes Lambda to skip records older than the new threshold. This action would lead to data loss for unacknowledged older records but would not improve the IteratorAge or overall processing throughput.

Increase the function's memory and CPU allocation

Why wrong: Increasing a Lambda function's memory also proportionally increases its allocated CPU, which can improve the execution speed of individual invocations if they are CPU or memory-bound. However, for a Kinesis-triggered Lambda, the primary bottleneck when IteratorAge is high is often the number of concurrent invocations, which is tied to the number of shards. Enhancing per-invocation performance does not increase the parallel processing capacity, thus it won't effectively address throttling caused by insufficient concurrency across the stream.

Increase the batch size for the Kinesis event source mapping.

Why wrong: Increasing the batch size for a Kinesis event source mapping means each Lambda invocation processes more records. While this can reduce the total number of invocations and potentially optimize costs, it does not mitigate cold start delays. A larger batch size might even exacerbate the impact of a cold start, as a single cold invocation would then be responsible for processing a larger chunk of data, potentially increasing the overall processing latency for that specific batch rather than reducing the cold start duration.

Enable reserved concurrency for the function.

Why wrong: Enabling reserved concurrency for a Lambda function guarantees a specific maximum number of concurrent executions, preventing other functions from consuming its allocated concurrency. This ensures the function can scale up to its limit without throttling. However, reserved concurrency does not proactively initialize execution environments; it only sets an upper limit. Therefore, a function with reserved concurrency can still experience cold starts if an invocation arrives when no warm execution environment is available, as it doesn't pre-warm instances.

Use provisioned concurrency for the function.

Why wrong: Using provisioned concurrency for a Lambda function pre-initializes a specified number of execution environments, keeping them warm and ready to process invocations. This mechanism effectively eliminates cold starts for requests that land on a provisioned concurrency instance. By ensuring execution environments are always available, provisioned concurrency significantly reduces invocation latency, providing consistent and predictable response times crucial for processing streaming data efficiently.

Increase the SQS queue visibility timeout

Why wrong: Increasing the SQS visibility timeout would make messages unavailable for longer after an initial receipt. While this might prevent immediate re-processing by *another* consumer, it does not reduce the number of concurrent Lambda invocations or directly mitigate throttling. If the Lambda function is still unable to process the message within the *increased* timeout, it will eventually become visible again and contribute to repeated processing attempts, potentially worsening the backlog rather than resolving throttles.

Increase the Lambda function's batch size

Why wrong: Increasing the Lambda function's batch size allows each invocation to process multiple messages simultaneously. This reduces the total number of Lambda invocations required to process a given volume of SQS messages. By decreasing the invocation rate, fewer concurrent executions are needed, which directly alleviates pressure on Lambda's concurrency limits and significantly reduces the likelihood of throttling, improving overall processing efficiency.

Decrease the SQS queue message retention period

Why wrong: Decreasing the SQS queue message retention period dictates how long messages remain in the queue before being automatically deleted, regardless of their processing status. While this might reduce the overall queue backlog by discarding older messages, it does not address the root cause of Lambda throttling. Furthermore, it introduces a significant risk of data loss for messages that haven't been successfully processed due to throttling or other issues within the shortened retention window.

Amazon CloudWatch Logs Insights

Why wrong: Amazon CloudWatch Logs Insights is a powerful query service specifically designed for analyzing log data, not distributed traces. It operates on log groups and log streams, allowing users to search and visualize patterns within textual log entries. It lacks the native capability to parse or filter AWS X-Ray trace segments by their structured annotations, which are distinct from log events and require a different indexing mechanism.

Amazon Athena queries on X-Ray data

Why wrong: While AWS X-Ray traces can be exported to Amazon S3, enabling subsequent querying via Amazon Athena, this approach requires significant upfront setup. It involves configuring an export process, defining an Athena table schema over the S3 data, and then executing SQL queries. This indirect method introduces latency and complexity compared to the native X-Ray console search, making it less suitable for immediate trace analysis by annotations.

Amazon Kinesis Data Analytics

Why wrong: Amazon Kinesis Data Analytics is a real-time analytics service primarily used for processing and analyzing streaming data as it arrives. It is designed for continuous queries on data streams from sources like Kinesis Data Streams or Firehose, not for querying historical, static X-Ray trace data stored for later analysis. Therefore, it cannot be directly used to search existing X-Ray traces by annotations, as it operates on live data streams rather than stored trace archives.

Enable DynamoDB Accelerator (DAX)

Why wrong: DAX acts as a fully managed, in-memory cache layer positioned in front of your DynamoDB table. It intercepts read requests, serving data from its cache with microsecond latency, which significantly reduces the load on the underlying DynamoDB table. This offloading mechanism is highly effective at absorbing sudden, intense read bursts, thereby preventing read throttling and improving application responsiveness.

Increase the write capacity

Why wrong: DynamoDB's capacity model distinctly separates read capacity units (RCUs) from write capacity units (WCUs). Increasing the provisioned write capacity only allocates more throughput for data modification operations, such as PutItem, UpdateItem, or DeleteItem. It has absolutely no bearing on the available capacity for GetItem or Query operations, and therefore will not alleviate read throttling.

Use a global table

Why wrong: DynamoDB Global Tables are primarily designed for multi-region active-active replication, enabling disaster recovery and providing low-latency data access for geographically dispersed users by serving reads from the nearest replica. While they replicate data across regions, they do not inherently increase the *provisioned read capacity* of an individual table within a single AWS region. Consequently, a global table will not directly prevent read throttling on a specific table experiencing a localized read burst.

Increase the number of cache nodes.

Why wrong: Adding nodes increases total memory but does not change how eviction selects keys; hot keys may still be evicted if not properly prioritized.

Disable the TTL on all cached keys.

Why wrong: Disabling TTL can cause the cache to fill with stale data and increase memory pressure, potentially worsening eviction of hot keys.

Increase the size of the cache cluster.

Why wrong: More memory can reduce eviction frequency, but if the eviction policy is not LRU-based, hot keys may still be evicted in favor of other keys.

Increase the Lambda function's timeout

Why wrong: The Lambda function's timeout setting dictates the maximum duration an invocation can run before AWS forcibly terminates it. Since the function is timing out after 30 seconds, it indicates the processing of larger files exceeds the currently configured execution limit. Increasing this timeout value directly addresses the problem by providing the function sufficient time to complete its operations on larger inputs, preventing premature termination and the associated timeout error. This is a direct configuration fix for an execution duration issue.

Increase the S3 event notification batch size

Why wrong: S3 event notifications, when configured to trigger a Lambda function, are typically sent on an individual object basis. There is no configurable "batch size" setting for S3 object creation or modification events that would cause multiple S3 events to be grouped into a single Lambda invocation. Therefore, attempting to increase a non-existent batch size is irrelevant to the problem and would not impact a Lambda function timing out while processing a single large file.

Enable Lambda function reserved concurrency

Why wrong: Enabling Lambda function reserved concurrency allocates a specific, dedicated pool of concurrency for that function, preventing other functions from consuming those execution slots. While useful for ensuring a function can always scale to a certain level or for preventing it from consuming too much shared concurrency, it has no impact on the maximum execution duration of a single invocation. Reserved concurrency manages how many instances can run simultaneously, not how long each instance runs before timing out.

The function's execution role lacks permissions to invoke the function.

Why wrong: The Lambda function's execution role defines the permissions that the function *itself* has when interacting with other AWS services (e.g., writing logs to CloudWatch, accessing S3 buckets). It does not govern who or what is allowed to *invoke* the Lambda function. If the execution role lacked necessary permissions, the function would fail during execution, not be throttled by the SQS event source attempting to invoke it.

The reserved concurrency is too low, causing SQS to throttle Lambda invocations.

Why wrong: Reserved concurrency explicitly caps the maximum number of concurrent executions for a specific Lambda function. When an SQS queue, configured as an event source, attempts to send messages to the Lambda function and the function has already reached its reserved concurrency limit, Lambda will reject further invocation requests. This rejection manifests as throttling, causing SQS to back off and retry, which delays message processing and can lead to messages accumulating in the queue.

The SQS queue visibility timeout is set too high.

Why wrong: The SQS queue visibility timeout dictates how long a message remains hidden from other consumers after it has been received by one. A high visibility timeout primarily affects the reprocessing logic for messages that fail to be processed or remain in-flight, potentially delaying their re-delivery. It does not, however, directly cause Lambda throttling, which is a mechanism that limits the rate of incoming invocation requests based on the function's available concurrency.

The Lambda function has a VPC configuration that causes cold starts.

Why wrong: A Lambda function configured within a Virtual Private Cloud (VPC) can indeed experience increased latency during cold starts due to the time required to provision and attach Elastic Network Interfaces (ENIs). While cold starts add initial overhead to the first invocation of a new execution environment, they do not directly cause throttling of *invocations* by SQS. Throttling occurs when the *rate* of incoming requests exceeds the function's available concurrency, not due to the startup time of individual instances.

Move all logs to S3 Glacier

Why wrong: Moving logs to S3 Glacier is a cost-saving measure for long-term archival storage of infrequently accessed data, typically after they've been processed or aged out of CloudWatch Logs. This action has no direct impact on the runtime performance or cold start duration of a Lambda function. Therefore, it will not address intermittent timeouts, which are related to the function's execution time or initialization overhead.

Disable all retries

Why wrong: Disabling retries for a Lambda function or its upstream invoker (like SQS or EventBridge) does not prevent the function from timing out; rather, it changes how the system *handles* a timeout event. If a function still times out, disabling retries simply means the invocation will fail immediately without another attempt. This action might reduce overall system load from failed retries but does not address the root cause of the intermittent timeouts within the Lambda execution itself.

S3 lifecycle expiration rules

Why wrong: S3 lifecycle expiration rules are designed to automatically transition or delete objects in an S3 bucket based on predefined policies, primarily for cost optimization and data retention compliance. These rules operate exclusively within Amazon S3 storage and have no functional relationship with API Gateway's runtime behavior or its ability to process incoming HTTP requests. Therefore, S3 lifecycle rules cannot cause API Gateway to return 429 throttling errors.

CloudFormation stack drift only

Why wrong: CloudFormation stack drift detection identifies when the actual configuration of resources in a stack deviates from their definition in the CloudFormation template, usually due to out-of-band manual changes. While drift might indicate a misconfiguration, the drift detection mechanism itself is a reporting feature and does not actively impose runtime throttling or rate limits on an API Gateway endpoint. Consequently, stack drift alone cannot directly cause API Gateway to return 429 errors during load testing.

Disable CloudWatch Logs

Why wrong: Disabling CloudWatch Logs for a Lambda function would prevent any logging of execution attempts, errors, or function output, making it impossible to diagnose why a "poison pill" message is causing repeated failures. While it might reduce logging costs, it does not stop the Lambda from being invoked by the problematic message and actively hinders troubleshooting, thus failing to address the core problem of repeated processing.

Set message retention to zero

Why wrong: Setting the SQS message retention period to zero is not a valid configuration; the minimum retention period is 60 seconds (1 minute). Even if it were possible, such a setting would immediately delete messages upon receipt or after a very short time, potentially before a Lambda function could even process them, leading to data loss rather than preventing repeated processing of a specific problematic message. This would destroy all messages, not just the poison pill, and is not a viable solution.

Step 1: Check credentials, Step 2: Check security groups, Step 3: Check network connectivity, Step 4: Test direct connection, Step 5: Check database logs

Why wrong: This order is incorrect because starting with credentials assumes the database is reachable, but if network connectivity is broken, no authentication attempt will succeed. Security groups and network checks should be done before credentials to avoid misleading error messages.

Step 1: Check security groups, Step 2: Check network connectivity, Step 3: Check credentials, Step 4: Check database logs, Step 5: Test direct connection

Why wrong: This order is incorrect because checking database logs before testing the direct connection is premature; logs are useful after confirming network and authentication issues are resolved. Also, security groups should be checked after verifying basic network connectivity, as they are a subset of network rules.

Step 1: Check network connectivity, Step 2: Check credentials, Step 3: Check security groups, Step 4: Test direct connection, Step 5: Check database logs

Why wrong: This order is incorrect because checking credentials before security groups can lead to confusion: a failed login might be due to IP restrictions in security groups rather than invalid credentials. Security groups should be verified before testing credentials to ensure the connection is allowed.

A unit test is failing; fix the test code.

Why wrong: An ECONNRESET error indicates a network connection was forcibly closed by the peer, which is fundamentally different from a unit test failure. Unit tests typically fail due to incorrect code logic or unmet assertions, resulting in specific assertion errors (e.g., 'expected X but got Y') rather than low-level network communication issues. Therefore, fixing test code would not resolve a network reset problem.

The npm cache is corrupted; clear the cache in CodeBuild.

Why wrong: AWS CodeBuild environments are ephemeral, meaning each build starts with a fresh, clean environment. There is no persistent npm cache that could become corrupted and carry over between builds. An ECONNRESET error specifically points to a problem with the network connection itself, not an issue with local cached data, making cache clearing irrelevant.

The npm token has expired; regenerate the token.

Why wrong: An expired or invalid npm token, typically used for authenticating with private package registries, would result in an HTTP 401 Unauthorized error, indicating an authentication failure. An ECONNRESET error, however, is a lower-level TCP error that means the connection was forcibly closed by the peer, pointing to a fundamental network connectivity issue rather than an authentication problem. Regenerating a token would not address a network reset.

The Lambda function is in a VPC without a NAT gateway.

Why wrong: A Lambda function in a VPC without a NAT Gateway would primarily face issues accessing external internet resources or certain AWS services outside the VPC. However, an AccessDeniedException when API Gateway attempts to invoke it is an IAM authorization failure, not a network routing problem. The invocation itself occurs over AWS's internal network, governed by IAM policies, not dependent on the Lambda's outbound internet access.

The Lambda function concurrency limit is exceeded.

Why wrong: Exceeding a Lambda function's concurrency limit results in throttling, which typically manifests as a TooManyRequestsException (HTTP 429) from the Lambda service. This error indicates a temporary resource unavailability due to high demand, not an authorization failure. An AccessDeniedException (HTTP 403) specifically signifies that the caller lacks the necessary IAM permissions to perform the requested action, which is a distinct issue from throttling.

API Gateway caching is enabled, returning stale responses.

Why wrong: API Gateway caching is designed to store and serve previously generated responses to improve performance and reduce backend load. If caching is enabled and returning stale data, it means API Gateway successfully retrieved a response from its cache, indicating successful prior interaction with the backend. This is a data freshness issue, not an authorization failure, and would not result in an `AccessDeniedException` when attempting to invoke the backend.