Testing Validation And Troubleshooting practice questions

A NAT Gateway with an EIP.

Why it fails: While this enables access, it's not the most secure or recommended approach compared to Interface Endpoints.

A

Why it fails: A

An Internet Gateway attached to the VPC.

Why it fails: Internet Gateways provide public access, which may violate security compliance.

A public subnet in the VPC configuration.

Why it fails: VPC endpoints do not require public subnets.

A

Why it fails: A

Disable the Guardrail and re-run the test to see if performance improves.

Why it fails: This would expose more PII, not help troubleshoot.

Route all traffic through a WAF to block PII patterns.

Why it fails: WAF is for HTTP/HTTPS filtering, not internal PII content within an LLM inference payload.

Increase the PII sensitivity threshold in the IAM policy.

Why it fails: Guardrail settings are configured in the Guardrail resource, not IAM.

Use AWS Step Functions to iterate through a test dataset and store results in S3.

Why it fails: This does not provide a built-in comparison interface for model performance.

Write a custom Python script to invoke both models and print the output to stdout.

Why it fails: While possible, it is not the most efficient or recommended AWS-managed way.

A

Why it fails: A

Deploy two separate endpoints and use Amazon CloudWatch dashboard for visual comparison.

Why it fails: CloudWatch is for monitoring metrics, not for evaluating model text quality.

A

Why it fails: A

Latency

Why it fails: This is a measurement of the overall duration, but monitoring it alone doesn't pinpoint the cause as context size.

Invocations

Why it fails: This measures the number of requests, not the size of the prompt.

OutputTokenCount

Why it fails: This measures the generated response, which is a result of the processing but not the cause of latency due to context.

Amazon SageMaker Model Monitor.

Why it fails: SageMaker Model Monitor is for tabular/ML models, not specifically for LLM text toxicity evaluation.

A

Why it fails: A

AWS Glue to sanitize data.

Why it fails: Glue is for data transformation, not model output evaluation.

AWS Lambda to call the model and manually inspect outputs.

Why it fails: Manual inspection is not automated validation.

Reduce the number of examples in the few-shot prompt.

Why it fails: Few-shot examples generally improve instruction adherence, not degrade it.

Change the model to a smaller parameter version.

Why it fails: Smaller models typically have worse instruction following capabilities.

A

Why it fails: A

Update the model's training data using a fine-tuning job.

Why it fails: Fine-tuning is a heavy-handed approach and not the first step for prompt instruction following issues.

Increase the top-p value.

Why it fails: Top-p also increases randomness.

A

Why it fails: A

Increase the maximum token length.

Why it fails: This only limits the output size, not the randomness.

Enable streaming in the API call.

Why it fails: Streaming changes the delivery method, not the generation behavior.

The IAM policy duration.

Why it fails: IAM policies do not have duration limits on individual requests.

The Bedrock base model throughput limit.

Why it fails: This causes 429 errors, not timeouts.

The VPC route table propagation.

Why it fails: This affects connectivity, not specific request timeouts.

A

Why it fails: A

AWS X-Ray service map.

Why it fails: X-Ray shows service latency and connectivity, not the internal decision logic of the model.

Amazon CloudWatch Logs for the Lambda function.

Why it fails: This shows the function's execution, but not the agent's logic for invoking it.

The Agent's IAM execution role policy.

Why it fails: An incorrect role would cause permission errors, not logical reasoning failures.

A

Why it fails: A

A

Why it fails: A

AWS WAF.

Why it fails: WAF does not natively inspect text payloads for PII.

Amazon Comprehend.

Why it fails: Comprehend can detect PII, but it requires manual integration into the pipeline.

Amazon Macie.

Why it fails: Macie is for S3 bucket data discovery, not real-time prompt filtering.

Deploy the model inside a VPC.

Why it fails: VPC deployment affects network isolation, not training data policy.

A

Why it fails: A

Enable data logging in CloudWatch.

Why it fails: Logging has no impact on model training practices.

Use a Provisioned Throughput model.

Why it fails: Provisioned Throughput is for capacity planning, not data privacy settings.

AWS Cloud9.

Why it fails: Cloud9 is an IDE, not a model prompt testing tool.

A

Why it fails: A

AWS Step Functions.

Why it fails: Step Functions is for workflow orchestration.

Amazon SageMaker Studio.

Why it fails: While it can be used, it is not specifically focused on prompt iteration for Bedrock models.

A

Why it fails: A

Use a smaller model for the agent.

Why it fails: Smaller models are generally worse at following complex JSON structures.

Increase the model's temperature.

Why it fails: Increasing temperature makes output less structured.

Add 'You must output JSON' to the system prompt.

Why it fails: System prompts are suggestions; a schema constraint is a technical enforcement.

AWS X-Ray service map.

Why it fails: X-Ray traces request paths through distributed applications and does not capture Bedrock model invocation metrics or token usage. It is tempting because X-Ray monitors service interactions, but Bedrock usage monitoring relies on CloudWatch metrics and invocation logging instead.

Amazon S3 bucket logging.

Why it fails: S3 bucket logging records access requests to S3 buckets and captures nothing about Bedrock model invocations or token consumption. It is tempting because S3 can store invocation logs, but the logging itself is configured through Bedrock's model invocation logging, not S3 bucket logging.

Increase the temperature.

Why it fails: This affects randomness, not verbosity.

Enable Guardrails to truncate output.

Why it fails: Guardrails are for content safety, not length control.

Use a different model family.

Why it fails: While different models have different tendencies, parameters and prompting are the primary tools.

Insecure API endpoint access.

Why it fails: While this is a general security issue, it is not specific to the LLM model itself.

Excessive API latency.

Why it fails: Latency is an availability issue, not a direct security threat.

AWS Config.

Why it fails: Useful for infrastructure compliance, but not model-level content compliance.

Amazon Inspector.

Why it fails: Used for vulnerability scanning of compute resources.

AWS Systems Manager.

Why it fails: Used for managing fleet configurations, not model safety.

Switch the vector store from Amazon OpenSearch Serverless to Amazon Aurora PostgreSQL with pgvector.

Why it fails: Changing the vector store is a significant architectural change that does not help diagnose the current issue. The problem could be in retrieval or generation, and switching stores introduces new variables. Diagnosis should first isolate the failing component without altering the system.

Increase the number of retrieved chunks by adjusting the maxResults parameter in the Retrieve API.

Why it fails: Adjusting maxResults changes the retrieval behavior, but without first inspecting what is currently retrieved, the developer cannot diagnose the root cause. This is a tuning action, not a diagnostic one. It may mask the problem or increase noise without confirming whether retrieval or generation is at fault.

Enable model invocation logging in Amazon Bedrock and inspect the retrieved passages in the logs.

Why it fails: Model invocation logging captures the request and response payloads, but it does not log the intermediate retrieval results from a Knowledge Base. The logs would show the final prompt and completion, not the passages retrieved. This makes it difficult to isolate whether retrieval failed.

Configure AWS X-Ray tracing on the Lambda function to trace the full request path.

Why it fails: X-Ray tracing is useful for distributed tracing, but it may not capture the agent's orchestration details or the model's reasoning. While it can help identify latency issues, it is not the primary source for diagnosing agent failures related to action group invocation. The two most direct sources are model invocation logs and Lambda logs.

Increase the Lambda function's memory allocation to improve performance.

Why it fails: Increasing memory might improve performance, but it does not provide diagnostic information. Without first identifying the cause, this change is speculative and could increase cost. The developer needs to gather logs and traces to understand the failure before tuning resources.

Change the agent's instruction to be more explicit about when to call the action group.

Why it fails: Modifying the instruction is a potential fix, but it does not help diagnose the current failures. The developer should first understand why the agent is failing by examining logs. Changing instructions without data could introduce new issues and does not provide visibility into the root cause.

The VPC endpoint policy is blocking traffic to the Bedrock runtime service.

Why it fails: A blocked VPC endpoint policy returns authorisation or connection errors, not HTTP 429 throttling responses. Throttling arises from request-rate limits on the model or cross-region inference profile. The option tempts because endpoint policies genuinely control Bedrock access, so they would be correct if the symptom were timeouts or access-denied failures.

The Lambda function timeout is set too low for the model response time.

Why it fails: A short timeout causes the invocation to fail with a timeout error, not a 429 response from Bedrock. Throttling is returned by the service before the function's execution window matters. The option tempts because timeout tuning genuinely affects long model responses, so it would be correct if the Lambda were terminating mid-inference.

The IAM role is missing the bedrock:InvokeModelWithResponseStream permission.

Why it fails: A missing InvokeModelWithResponseStream permission produces AccessDeniedException, not 429. Throttling is a rate signal independent of IAM authorisation. The option tempts because IAM permissions genuinely gate Bedrock calls, so they would be correct if the Lambda logged authorisation failures rather than too-many-requests responses.