AIP-C01 Testing Validation And Troubleshooting — All Questions With Answers

Question 1

During development, an AI application fails to initialize because it cannot connect to the Amazon Bedrock API. The application runs inside a private VPC. What must be configured?

Accepted Answer

An Interface VPC Endpoint for Amazon Bedrock.. Interface VPC Endpoints (AWS PrivateLink) are required for VPC-based resources to communicate with Bedrock without going over the public internet.

Answer

A NAT Gateway with an EIP.

Answer

A

Answer

An Internet Gateway attached to the VPC.

Answer

A public subnet in the VPC configuration.

Question 2

A company is using Amazon Bedrock with Guardrails to filter PII. Users report that some PII is still passing through. What is the best approach to troubleshoot this?

Accepted Answer

Enable the 'trace' functionality in the InvokeModel API response to analyze Guardrail findings.. Checking the Guardrails assessment trace via the InvokeModel API response is necessary to see if the filter was triggered or bypassed.

Answer

A

Answer

Disable the Guardrail and re-run the test to see if performance improves.

Answer

Route all traffic through a WAF to block PII patterns.

Answer

Increase the PII sensitivity threshold in the IAM policy.

Question 3

A developer wants to evaluate a custom fine-tuned model before deploying it to production. What is the most efficient way to perform a side-by-side comparison with the base model?

Accepted Answer

Use Amazon Bedrock Model Evaluation jobs.. Amazon Bedrock Model Evaluation allows for human or automated evaluation tasks comparing multiple models.

Answer

Use AWS Step Functions to iterate through a test dataset and store results in S3.

Answer

Write a custom Python script to invoke both models and print the output to stdout.

Answer

A

Answer

Deploy two separate endpoints and use Amazon CloudWatch dashboard for visual comparison.

Question 4

A generative AI application using Amazon Bedrock is experiencing high latency. The developer suspects the issue is the size of the prompt context window. Which metric in Amazon CloudWatch should be monitored?

Accepted Answer

InputTokenCount. InputTokenCount is the primary metric indicating how much data is being sent to the model per request.

Answer

A

Answer

Latency

Answer

Invocations

Answer

OutputTokenCount

Question 5

A developer needs to validate that a fine-tuned model does not exhibit toxic behavior. Which AWS service should be used to automate this validation?

Accepted Answer

Amazon Bedrock Evaluation jobs with built-in toxicity metrics.. Amazon Bedrock Evaluation jobs allow for built-in toxicity and bias testing against standard benchmarks.

Answer

Amazon SageMaker Model Monitor.

Answer

A

Answer

AWS Glue to sanitize data.

Answer

AWS Lambda to call the model and manually inspect outputs.

Question 6

A developer is testing a model and notices that it frequently defaults to a specific tone regardless of the system prompt. What is the most effective troubleshooting technique?

Accepted Answer

Use the 'System' instruction field in the Messages API and increase the importance of the prompt framing.. System prompt injection or strict framing is often bypassed by models if the system instruction isn't strong enough or if the model's training bias is strong.

Answer

Reduce the number of examples in the few-shot prompt.

Answer

Change the model to a smaller parameter version.

Answer

A

Answer

Update the model's training data using a fine-tuning job.

Question 7

A developer notices that an application's inference results vary widely for the same input. What is the most appropriate setting to adjust for reproducibility?

Accepted Answer

Set the temperature parameter to 0.. The 'temperature' parameter controls randomness. Setting it to 0 results in deterministic, reproducible output.

Answer

Increase the top-p value.

Answer

A

Answer

Increase the maximum token length.

Answer

Enable streaming in the API call.

Question 8

You are troubleshooting a timeout error during long-running inference requests using streaming. What is the most likely configuration to check?

Accepted Answer

The SDK request timeout configuration.. If requests time out during streaming, the streaming response buffer or the client-side read timeout are likely misconfigured.

Answer

The IAM policy duration.

Answer

The Bedrock base model throughput limit.

Answer

The VPC route table propagation.

Answer

A

Question 9

A team is building an agentic workflow using Amazon Bedrock Agents. The agent is failing to correctly invoke a specific Lambda function tool. Where should the developer look to troubleshoot the reasoning process?

Accepted Answer

The 'Trace' field in the Bedrock Agent response.. The 'Trace' feature in Bedrock Agents provides a step-by-step breakdown of how the agent decided to call a function.

Answer

AWS X-Ray service map.

Answer

Amazon CloudWatch Logs for the Lambda function.

Answer

The Agent's IAM execution role policy.

Answer

A

Question 10

A company wants to identify PII in user prompts before sending them to a model. Which component of Amazon Bedrock is best suited for this?

Accepted Answer

Bedrock Guardrails.. Guardrails for Amazon Bedrock allows for the configuration of content filters including PII redaction.

Answer

A

Answer

AWS WAF.

Answer

Amazon Comprehend.

Answer

Amazon Macie.

Question 11

An organization is concerned about data privacy and wants to ensure that no user data is used to train foundation models. Which setting ensures this?

Accepted Answer

Configure the 'Data Sharing' settings in the Bedrock console to disable data collection.. AWS provides a 'no training' commitment by default for Bedrock and allows users to opt-out of data collection through settings.

Answer

Deploy the model inside a VPC.

Answer

A

Answer

Enable data logging in CloudWatch.

Answer

Use a Provisioned Throughput model.

Question 12

Which AWS tool provides a graphical interface to test and iterate on prompts with different models before integrating them into an application?

Accepted Answer

Amazon Bedrock Playgrounds.. Amazon Bedrock Prompt Management provides a playground for testing and saving prompts.

Answer

AWS Cloud9.

Answer

A

Answer

AWS Step Functions.

Answer

Amazon SageMaker Studio.

Question 13

A developer is troubleshooting an error where an agent fails to call a tool because the input JSON is malformed. How can the developer force the agent to follow a specific output schema?

Accepted Answer

Use the 'Inference Configuration' to enforce a strict JSON schema.. The 'Inference Configuration' allows for structured output enforcement in supported models.

Answer

A

Answer

Use a smaller model for the agent.

Answer

Increase the model's temperature.

Answer

Add 'You must output JSON' to the system prompt.

Question 14

Which THREE of the following are valid ways to monitor Amazon Bedrock model usage?

Accepted Answer

AWS CloudTrail logs.. CloudWatch metrics, AWS CloudTrail logs, and Usage billing reports are the primary ways to monitor usage.

Answer

AWS X-Ray service map.

Answer

Amazon S3 bucket logging.

Question 15

A developer is testing an application and finds that the model output is too verbose. Which TWO of the following parameters or techniques can help control the response length?

Accepted Answer

Reduce the 'max_tokens' parameter.. Max tokens and system instructions are the primary ways to limit verbosity.

Answer

Increase the temperature.

Answer

Enable Guardrails to truncate output.

Answer

Use a different model family.

Question 16

You are auditing an AI application for security risks. Which THREE of the following are potential threats that should be tested against the model?

Accepted Answer

Toxic or biased output generation.. Option A (Toxic or biased output generation) is correct because AI models can reproduce or amplify harmful stereotypes and offensive language from their training data, so red-team testing must probe for toxicity and bias in generated content. Option D (Prompt injection attacks) is correct because attackers can embed adversarial instructions in user input or retrieved content to override system prompts, exfiltrate data, or hijack the model's behavior, making it a core AI-specific threat to test. Option E (PII data leakage in the response) is correct because models trained on or given access to sensitive data may memorize and regurgitate personally identifiable information, so testing for unintended PII disclosure is essential. Option B (Insecure API endpoint access) is not selected because it concerns the surrounding application's authentication and authorization controls rather than a threat against the model itself. Option C (Excessive API latency) is not selected because latency is a performance and availability concern, not a security threat to the model.

Answer

Insecure API endpoint access.

Answer

Excessive API latency.

Question 17

A company is setting up a governance framework for AI development. Which TWO of the following tools should they use to enforce safety and compliance in Amazon Bedrock?

Accepted Answer

Amazon Bedrock Model Evaluation jobs.. Guardrails and Model Evaluation jobs are the primary tools for enforcing safety and assessing compliance.

Answer

AWS Config.

Answer

Amazon Inspector.

Answer

AWS Systems Manager.

Question 18

A developer is building a Retrieval Augmented Generation (RAG) application using Amazon Bedrock Knowledge Bases. During testing, the developer observes that the model frequently provides incorrect answers even though the correct information exists in the source documents. The developer wants to diagnose whether the issue is in retrieval or generation. Which approach should the developer take to isolate the problem?

Accepted Answer

Use the RetrieveAndGenerate API with the returnSourceDocuments parameter set to true and examine the returned source documents.. To isolate whether the issue is in retrieval or generation, the developer needs visibility into the retrieved documents. The RetrieveAndGenerate API with returnSourceDocuments provides exactly that. If the source documents contain the correct information but the answer is wrong, the generation step is at fault. If the source documents are incorrect, the retrieval step needs tuning.

Answer

Switch the vector store from Amazon OpenSearch Serverless to Amazon Aurora PostgreSQL with pgvector.

Answer

Increase the number of retrieved chunks by adjusting the maxResults parameter in the Retrieve API.

Answer

Enable model invocation logging in Amazon Bedrock and inspect the retrieved passages in the logs.

Question 19

A developer is troubleshooting an Amazon Bedrock agent that intermittently fails to complete user requests. The agent uses an action group with an AWS Lambda function to query a database. The developer needs to identify the cause of the failures. Which TWO actions should the developer take to gather the necessary diagnostic information? (Choose two.)

Accepted Answer

Review the AWS Lambda function's Amazon CloudWatch Logs for errors and timeouts.. To diagnose an agent's intermittent failures, the developer needs visibility into both the agent's orchestration and the action group's execution. Enabling model invocation logging captures the agent's reasoning and API calls, while reviewing Lambda CloudWatch Logs reveals errors in the action group. Together, these provide a complete picture of where the failure occurs.

Answer

Configure AWS X-Ray tracing on the Lambda function to trace the full request path.

Answer

Increase the Lambda function's memory allocation to improve performance.

Answer

Change the agent's instruction to be more explicit about when to call the action group.

Question 20

An application deployed on AWS Lambda is receiving 429 Too Many Requests errors when calling Amazon Bedrock. The developer has verified that account-level quotas have not been reached. What is the most likely cause?

Accepted Answer

The application is exceeding the tokens-per-minute (TPM) limit for the specific model in that AWS Region.. A 429 Too Many Requests from Bedrock, despite account-level quotas being fine, most commonly means the application is hitting the per-model tokens-per-minute (TPM) or requests-per-minute (RPM) limit for that specific model in that Region. Bedrock enforces model-specific quotas that are separate from account-level limits, so a single high-volume model can throttle even when overall account usage is low.

Answer

The VPC endpoint policy is blocking traffic to the Bedrock runtime service.

Answer

The Lambda function timeout is set too low for the model response time.

Answer

The IAM role is missing the bedrock:InvokeModelWithResponseStream permission.

AIP-C01 Testing Validation And Troubleshooting — All Questions With Answers

During development, an AI application fails to initialize because it cannot connect to the Amazon Bedrock API. The application runs inside a private VPC. What must be configured?

A company is using Amazon Bedrock with Guardrails to filter PII. Users report that some PII is still passing through. What is the best approach to troubleshoot this?

A developer wants to evaluate a custom fine-tuned model before deploying it to production. What is the most efficient way to perform a side-by-side comparison with the base model?

A generative AI application using Amazon Bedrock is experiencing high latency. The developer suspects the issue is the size of the prompt context window. Which metric in Amazon CloudWatch should be monitored?

A developer needs to validate that a fine-tuned model does not exhibit toxic behavior. Which AWS service should be used to automate this validation?

A developer is testing a model and notices that it frequently defaults to a specific tone regardless of the system prompt. What is the most effective troubleshooting technique?

A developer notices that an application's inference results vary widely for the same input. What is the most appropriate setting to adjust for reproducibility?

You are troubleshooting a timeout error during long-running inference requests using streaming. What is the most likely configuration to check?

A team is building an agentic workflow using Amazon Bedrock Agents. The agent is failing to correctly invoke a specific Lambda function tool. Where should the developer look to troubleshoot the reasoning process?

A company wants to identify PII in user prompts before sending them to a model. Which component of Amazon Bedrock is best suited for this?

An organization is concerned about data privacy and wants to ensure that no user data is used to train foundation models. Which setting ensures this?

Which AWS tool provides a graphical interface to test and iterate on prompts with different models before integrating them into an application?

A developer is troubleshooting an error where an agent fails to call a tool because the input JSON is malformed. How can the developer force the agent to follow a specific output schema?

Which THREE of the following are valid ways to monitor Amazon Bedrock model usage?

A developer is testing an application and finds that the model output is too verbose. Which TWO of the following parameters or techniques can help control the response length?

You are auditing an AI application for security risks. Which THREE of the following are potential threats that should be tested against the model?

A company is setting up a governance framework for AI development. Which TWO of the following tools should they use to enforce safety and compliance in Amazon Bedrock?

An application deployed on AWS Lambda is receiving 429 Too Many Requests errors when calling Amazon Bedrock. The developer has verified that account-level quotas have not been reached. What is the most likely cause?

A developer is using Amazon Bedrock to build a customer support chatbot. The model is hallucinating frequently when retrieving internal company knowledge. Which evaluation methodology should the developer implement to quantify the ground truth accuracy of the RAG pipeline?

A developer is creating a test suite for an LLM application. Which THREE of the following metrics are commonly used to evaluate RAG system performance?

A developer is troubleshooting an Amazon Bedrock Agent that is frequently hitting errors. Which TWO of the following steps should be taken to isolate the issue?

Which THREE of the following are valid components of an Amazon Bedrock Agent test plan?

A developer is validating a generative AI application that uses Amazon Bedrock. The application must produce deterministic, repeatable outputs for regression testing. Which TWO configuration changes should the developer make to achieve this? (Choose two.)

AIP-C01 Testing Validation And Troubleshooting — All Questions With Answers

During development, an AI application fails to initialize because it cannot connect to the Amazon Bedrock API. The application runs inside a private VPC. What must be configured?

A company is using Amazon Bedrock with Guardrails to filter PII. Users report that some PII is still passing through. What is the best approach to troubleshoot this?

A developer wants to evaluate a custom fine-tuned model before deploying it to production. What is the most efficient way to perform a side-by-side comparison with the base model?

A generative AI application using Amazon Bedrock is experiencing high latency. The developer suspects the issue is the size of the prompt context window. Which metric in Amazon CloudWatch should be monitored?

A developer needs to validate that a fine-tuned model does not exhibit toxic behavior. Which AWS service should be used to automate this validation?

A developer is testing a model and notices that it frequently defaults to a specific tone regardless of the system prompt. What is the most effective troubleshooting technique?

A developer notices that an application's inference results vary widely for the same input. What is the most appropriate setting to adjust for reproducibility?

You are troubleshooting a timeout error during long-running inference requests using streaming. What is the most likely configuration to check?

A team is building an agentic workflow using Amazon Bedrock Agents. The agent is failing to correctly invoke a specific Lambda function tool. Where should the developer look to troubleshoot the reasoning process?

A company wants to identify PII in user prompts before sending them to a model. Which component of Amazon Bedrock is best suited for this?

An organization is concerned about data privacy and wants to ensure that no user data is used to train foundation models. Which setting ensures this?

Which AWS tool provides a graphical interface to test and iterate on prompts with different models before integrating them into an application?

A developer is troubleshooting an error where an agent fails to call a tool because the input JSON is malformed. How can the developer force the agent to follow a specific output schema?

Which THREE of the following are valid ways to monitor Amazon Bedrock model usage?

A developer is testing an application and finds that the model output is too verbose. Which TWO of the following parameters or techniques can help control the response length?

You are auditing an AI application for security risks. Which THREE of the following are potential threats that should be tested against the model?

A company is setting up a governance framework for AI development. Which TWO of the following tools should they use to enforce safety and compliance in Amazon Bedrock?

An application deployed on AWS Lambda is receiving 429 Too Many Requests errors when calling Amazon Bedrock. The developer has verified that account-level quotas have not been reached. What is the most likely cause?

A developer is using Amazon Bedrock to build a customer support chatbot. The model is hallucinating frequently when retrieving internal company knowledge. Which evaluation methodology should the developer implement to quantify the ground truth accuracy of the RAG pipeline?

A developer is creating a test suite for an LLM application. Which THREE of the following metrics are commonly used to evaluate RAG system performance?

A developer is troubleshooting an Amazon Bedrock Agent that is frequently hitting errors. Which TWO of the following steps should be taken to isolate the issue?

Which THREE of the following are valid components of an Amazon Bedrock Agent test plan?

A developer is validating a generative AI application that uses Amazon Bedrock. The application must produce deterministic, repeatable outputs for regression testing. Which TWO configuration changes should the developer make to achieve this? (Choose two.)