AIP-C01 Testing Validation And Troubleshooting — Timed 10 Questions

Question 1

During development, an AI application fails to initialize because it cannot connect to the Amazon Bedrock API. The application runs inside a private VPC. What must be configured?

Accepted Answer

An Interface VPC Endpoint for Amazon Bedrock.. Interface VPC Endpoints (AWS PrivateLink) are required for VPC-based resources to communicate with Bedrock without going over the public internet.

Answer

A NAT Gateway with an EIP.

Answer

A

Answer

An Internet Gateway attached to the VPC.

Answer

A public subnet in the VPC configuration.

Question 2

A company is using Amazon Bedrock with Guardrails to filter PII. Users report that some PII is still passing through. What is the best approach to troubleshoot this?

Accepted Answer

Enable the 'trace' functionality in the InvokeModel API response to analyze Guardrail findings.. Checking the Guardrails assessment trace via the InvokeModel API response is necessary to see if the filter was triggered or bypassed.

Answer

A

Answer

Disable the Guardrail and re-run the test to see if performance improves.

Answer

Route all traffic through a WAF to block PII patterns.

Answer

Increase the PII sensitivity threshold in the IAM policy.

Question 3

A developer wants to evaluate a custom fine-tuned model before deploying it to production. What is the most efficient way to perform a side-by-side comparison with the base model?

Accepted Answer

Use Amazon Bedrock Model Evaluation jobs.. Amazon Bedrock Model Evaluation allows for human or automated evaluation tasks comparing multiple models.

Answer

Use AWS Step Functions to iterate through a test dataset and store results in S3.

Answer

Write a custom Python script to invoke both models and print the output to stdout.

Answer

A

Answer

Deploy two separate endpoints and use Amazon CloudWatch dashboard for visual comparison.

Question 4

A generative AI application using Amazon Bedrock is experiencing high latency. The developer suspects the issue is the size of the prompt context window. Which metric in Amazon CloudWatch should be monitored?

Accepted Answer

InputTokenCount. InputTokenCount is the primary metric indicating how much data is being sent to the model per request.

Answer

A

Answer

Latency

Answer

Invocations

Answer

OutputTokenCount

Question 5

A developer needs to validate that a fine-tuned model does not exhibit toxic behavior. Which AWS service should be used to automate this validation?

Accepted Answer

Amazon Bedrock Evaluation jobs with built-in toxicity metrics.. Amazon Bedrock Evaluation jobs allow for built-in toxicity and bias testing against standard benchmarks.

Answer

Amazon SageMaker Model Monitor.

Answer

A

Answer

AWS Glue to sanitize data.

Answer

AWS Lambda to call the model and manually inspect outputs.

Question 6

A developer is testing a model and notices that it frequently defaults to a specific tone regardless of the system prompt. What is the most effective troubleshooting technique?

Accepted Answer

Use the 'System' instruction field in the Messages API and increase the importance of the prompt framing.. System prompt injection or strict framing is often bypassed by models if the system instruction isn't strong enough or if the model's training bias is strong.

Answer

Reduce the number of examples in the few-shot prompt.

Answer

Change the model to a smaller parameter version.

Answer

A

Answer

Update the model's training data using a fine-tuning job.

Question 7

A developer notices that an application's inference results vary widely for the same input. What is the most appropriate setting to adjust for reproducibility?

Accepted Answer

Set the temperature parameter to 0.. The 'temperature' parameter controls randomness. Setting it to 0 results in deterministic, reproducible output.

Answer

Increase the top-p value.

Answer

A

Answer

Increase the maximum token length.

Answer

Enable streaming in the API call.

Question 8

You are troubleshooting a timeout error during long-running inference requests using streaming. What is the most likely configuration to check?

Accepted Answer

The SDK request timeout configuration.. If requests time out during streaming, the streaming response buffer or the client-side read timeout are likely misconfigured.

Answer

The IAM policy duration.

Answer

The Bedrock base model throughput limit.

Answer

The VPC route table propagation.

Answer

A

Question 9

A team is building an agentic workflow using Amazon Bedrock Agents. The agent is failing to correctly invoke a specific Lambda function tool. Where should the developer look to troubleshoot the reasoning process?

Accepted Answer

The 'Trace' field in the Bedrock Agent response.. The 'Trace' feature in Bedrock Agents provides a step-by-step breakdown of how the agent decided to call a function.

Answer

AWS X-Ray service map.

Answer

Amazon CloudWatch Logs for the Lambda function.

Answer

The Agent's IAM execution role policy.

Answer

A

Question 10

A company wants to identify PII in user prompts before sending them to a model. Which component of Amazon Bedrock is best suited for this?

Accepted Answer

Bedrock Guardrails.. Guardrails for Amazon Bedrock allows for the configuration of content filters including PII redaction.

Answer

A

Answer

AWS WAF.

Answer

Amazon Comprehend.

Answer

Amazon Macie.