AIP-C01 Testing Validation And Troubleshooting — 20 Questions

Question 1

During development, an AI application fails to initialize because it cannot connect to the Amazon Bedrock API. The application runs inside a private VPC. What must be configured?

Accepted Answer

An Interface VPC Endpoint for Amazon Bedrock.. Interface VPC Endpoints (AWS PrivateLink) are required for VPC-based resources to communicate with Bedrock without going over the public internet.

Answer

A NAT Gateway with an EIP.

Answer

A

Answer

An Internet Gateway attached to the VPC.

Answer

A public subnet in the VPC configuration.

Question 2

A company is using Amazon Bedrock with Guardrails to filter PII. Users report that some PII is still passing through. What is the best approach to troubleshoot this?

Accepted Answer

Enable the 'trace' functionality in the InvokeModel API response to analyze Guardrail findings.. Checking the Guardrails assessment trace via the InvokeModel API response is necessary to see if the filter was triggered or bypassed.

Answer

A

Answer

Disable the Guardrail and re-run the test to see if performance improves.

Answer

Route all traffic through a WAF to block PII patterns.

Answer

Increase the PII sensitivity threshold in the IAM policy.

Question 3

A developer wants to evaluate a custom fine-tuned model before deploying it to production. What is the most efficient way to perform a side-by-side comparison with the base model?

Accepted Answer

Use Amazon Bedrock Model Evaluation jobs.. Amazon Bedrock Model Evaluation allows for human or automated evaluation tasks comparing multiple models.

Answer

Use AWS Step Functions to iterate through a test dataset and store results in S3.

Answer

Write a custom Python script to invoke both models and print the output to stdout.

Answer

A

Answer

Deploy two separate endpoints and use Amazon CloudWatch dashboard for visual comparison.

Question 4

A generative AI application using Amazon Bedrock is experiencing high latency. The developer suspects the issue is the size of the prompt context window. Which metric in Amazon CloudWatch should be monitored?

Accepted Answer

InputTokenCount. InputTokenCount is the primary metric indicating how much data is being sent to the model per request.

Answer

A

Answer

Latency

Answer

Invocations

Answer

OutputTokenCount

Question 5

A developer needs to validate that a fine-tuned model does not exhibit toxic behavior. Which AWS service should be used to automate this validation?

Accepted Answer

Amazon Bedrock Evaluation jobs with built-in toxicity metrics.. Amazon Bedrock Evaluation jobs allow for built-in toxicity and bias testing against standard benchmarks.

Answer

Amazon SageMaker Model Monitor.

Answer

A

Answer

AWS Glue to sanitize data.

Answer

AWS Lambda to call the model and manually inspect outputs.

Question 6

A developer is testing a model and notices that it frequently defaults to a specific tone regardless of the system prompt. What is the most effective troubleshooting technique?

Accepted Answer

Use the 'System' instruction field in the Messages API and increase the importance of the prompt framing.. System prompt injection or strict framing is often bypassed by models if the system instruction isn't strong enough or if the model's training bias is strong.

Answer

Reduce the number of examples in the few-shot prompt.

Answer

Change the model to a smaller parameter version.

Answer

A

Answer

Update the model's training data using a fine-tuning job.

Question 7

A developer notices that an application's inference results vary widely for the same input. What is the most appropriate setting to adjust for reproducibility?

Accepted Answer

Set the temperature parameter to 0.. The 'temperature' parameter controls randomness. Setting it to 0 results in deterministic, reproducible output.

Answer

Increase the top-p value.

Answer

A

Answer

Increase the maximum token length.

Answer

Enable streaming in the API call.

Question 8

You are troubleshooting a timeout error during long-running inference requests using streaming. What is the most likely configuration to check?

Accepted Answer

The SDK request timeout configuration.. If requests time out during streaming, the streaming response buffer or the client-side read timeout are likely misconfigured.

Answer

The IAM policy duration.

Answer

The Bedrock base model throughput limit.

Answer

The VPC route table propagation.

Answer

A

Question 9

A team is building an agentic workflow using Amazon Bedrock Agents. The agent is failing to correctly invoke a specific Lambda function tool. Where should the developer look to troubleshoot the reasoning process?

Accepted Answer

The 'Trace' field in the Bedrock Agent response.. The 'Trace' feature in Bedrock Agents provides a step-by-step breakdown of how the agent decided to call a function.

Answer

AWS X-Ray service map.

Answer

Amazon CloudWatch Logs for the Lambda function.

Answer

The Agent's IAM execution role policy.

Answer

A

Question 10

A company wants to identify PII in user prompts before sending them to a model. Which component of Amazon Bedrock is best suited for this?

Accepted Answer

Bedrock Guardrails.. Guardrails for Amazon Bedrock allows for the configuration of content filters including PII redaction.

Answer

A

Answer

AWS WAF.

Answer

Amazon Comprehend.

Answer

Amazon Macie.

Question 11

An organization is concerned about data privacy and wants to ensure that no user data is used to train foundation models. Which setting ensures this?

Accepted Answer

Configure the 'Data Sharing' settings in the Bedrock console to disable data collection.. AWS provides a 'no training' commitment by default for Bedrock and allows users to opt-out of data collection through settings.

Answer

Deploy the model inside a VPC.

Answer

A

Answer

Enable data logging in CloudWatch.

Answer

Use a Provisioned Throughput model.

Question 12

Which AWS tool provides a graphical interface to test and iterate on prompts with different models before integrating them into an application?

Accepted Answer

Amazon Bedrock Playgrounds.. Amazon Bedrock Prompt Management provides a playground for testing and saving prompts.

Answer

AWS Cloud9.

Answer

A

Answer

AWS Step Functions.

Answer

Amazon SageMaker Studio.

Question 13

A developer is troubleshooting an error where an agent fails to call a tool because the input JSON is malformed. How can the developer force the agent to follow a specific output schema?

Accepted Answer

Use the 'Inference Configuration' to enforce a strict JSON schema.. The 'Inference Configuration' allows for structured output enforcement in supported models.

Answer

A

Answer

Use a smaller model for the agent.

Answer

Increase the model's temperature.

Answer

Add 'You must output JSON' to the system prompt.

Question 14

An application deployed on AWS Lambda is receiving 429 Too Many Requests errors when calling Amazon Bedrock. The developer has verified that account-level quotas have not been reached. What is the most likely cause?

Accepted Answer

The application is exceeding the tokens-per-minute (TPM) limit for the specific model in that AWS Region.. Model-specific throughput limits or Provisioned Throughput quotas are distinct from account-level service quotas.

Answer

The VPC endpoint policy is blocking traffic to the Bedrock runtime service.

Answer

The Lambda function timeout is set too low for the model response time.

Answer

The IAM role is missing the bedrock:InvokeModelWithResponseStream permission.

Question 15

A developer is using Amazon Bedrock to build a customer support chatbot. The model is hallucinating frequently when retrieving internal company knowledge. Which evaluation methodology should the developer implement to quantify the ground truth accuracy of the RAG pipeline?

Accepted Answer

Implement a RAGAS-based evaluation pipeline to measure faithfulness and answer relevance.. Retrieval-Augmented Generation (RAG) evaluation requires metrics like RAGAS or TruLens to measure faithfulness, answer relevance, and context precision against a golden dataset.

Answer

Increase the temperature setting to 1.0 to encourage more creative responses.

Answer

Use Amazon CloudWatch logs to calculate the latency of every inference call.

Answer

Run a standard BLEU score evaluation against the model's training data.

Question 16

A developer is creating a test suite for an LLM application. Which THREE of the following metrics are commonly used to evaluate RAG system performance?

Accepted Answer

Answer Relevance. Faithfulness, answer relevance, and context precision are the key components of RAGAS evaluation.

Answer

Model Training Loss

Answer

Inference Latency

Question 17

Which THREE of the following are valid ways to monitor Amazon Bedrock model usage?

Accepted Answer

AWS CloudTrail logs.. CloudWatch metrics, AWS CloudTrail logs, and Usage billing reports are the primary ways to monitor usage.

Answer

AWS X-Ray service map.

Answer

Amazon S3 bucket logging.

Question 18

A developer is troubleshooting an Amazon Bedrock Agent that is frequently hitting errors. Which TWO of the following steps should be taken to isolate the issue?

Accepted Answer

Check CloudWatch Logs for the associated Action Group Lambda functions.. Examining the trace and checking the Lambda function logs are the standard diagnostic paths for Agent failures.

Answer

Restart the Bedrock service.

Answer

Increase the model's temperature to 1.0.

Answer

Re-train the underlying base model.

Question 19

A developer is testing an application and finds that the model output is too verbose. Which TWO of the following parameters or techniques can help control the response length?

Accepted Answer

Reduce the 'max_tokens' parameter.. Max tokens and system instructions are the primary ways to limit verbosity.

Answer

Increase the temperature.

Answer

Enable Guardrails to truncate output.

Answer

Use a different model family.

Question 20

You are auditing an AI application for security risks. Which THREE of the following are potential threats that should be tested against the model?

Accepted Answer

Toxic or biased output generation.. Prompt injection, PII leakage, and model output toxicity are key security concerns for LLM applications.

Answer

Insecure API endpoint access.

Answer

Excessive API latency.