AIP-C01 Testing Validation And Troubleshooting Practice Test 2 — 15 Questions

Question 1

You are auditing an AI application for security risks. Which THREE of the following are potential threats that should be tested against the model?

Accepted Answer

Toxic or biased output generation.. Option A (Toxic or biased output generation) is correct because AI models can reproduce or amplify harmful stereotypes and offensive language from their training data, so red-team testing must probe for toxicity and bias in generated content. Option D (Prompt injection attacks) is correct because attackers can embed adversarial instructions in user input or retrieved content to override system prompts, exfiltrate data, or hijack the model's behavior, making it a core AI-specific threat to test. Option E (PII data leakage in the response) is correct because models trained on or given access to sensitive data may memorize and regurgitate personally identifiable information, so testing for unintended PII disclosure is essential. Option B (Insecure API endpoint access) is not selected because it concerns the surrounding application's authentication and authorization controls rather than a threat against the model itself. Option C (Excessive API latency) is not selected because latency is a performance and availability concern, not a security threat to the model.

Answer

Insecure API endpoint access.

Answer

Excessive API latency.

Question 2

A company is setting up a governance framework for AI development. Which TWO of the following tools should they use to enforce safety and compliance in Amazon Bedrock?

Accepted Answer

Amazon Bedrock Model Evaluation jobs.. Guardrails and Model Evaluation jobs are the primary tools for enforcing safety and assessing compliance.

Answer

AWS Config.

Answer

Amazon Inspector.

Answer

AWS Systems Manager.

Question 3

A developer is building a Retrieval Augmented Generation (RAG) application using Amazon Bedrock Knowledge Bases. During testing, the developer observes that the model frequently provides incorrect answers even though the correct information exists in the source documents. The developer wants to diagnose whether the issue is in retrieval or generation. Which approach should the developer take to isolate the problem?

Accepted Answer

Use the RetrieveAndGenerate API with the returnSourceDocuments parameter set to true and examine the returned source documents.. To isolate whether the issue is in retrieval or generation, the developer needs visibility into the retrieved documents. The RetrieveAndGenerate API with returnSourceDocuments provides exactly that. If the source documents contain the correct information but the answer is wrong, the generation step is at fault. If the source documents are incorrect, the retrieval step needs tuning.

Answer

Switch the vector store from Amazon OpenSearch Serverless to Amazon Aurora PostgreSQL with pgvector.

Answer

Increase the number of retrieved chunks by adjusting the maxResults parameter in the Retrieve API.

Answer

Enable model invocation logging in Amazon Bedrock and inspect the retrieved passages in the logs.

Question 4

A developer is troubleshooting an Amazon Bedrock agent that intermittently fails to complete user requests. The agent uses an action group with an AWS Lambda function to query a database. The developer needs to identify the cause of the failures. Which TWO actions should the developer take to gather the necessary diagnostic information? (Choose two.)

Accepted Answer

Review the AWS Lambda function's Amazon CloudWatch Logs for errors and timeouts.. To diagnose an agent's intermittent failures, the developer needs visibility into both the agent's orchestration and the action group's execution. Enabling model invocation logging captures the agent's reasoning and API calls, while reviewing Lambda CloudWatch Logs reveals errors in the action group. Together, these provide a complete picture of where the failure occurs.

Answer

Configure AWS X-Ray tracing on the Lambda function to trace the full request path.

Answer

Increase the Lambda function's memory allocation to improve performance.

Answer

Change the agent's instruction to be more explicit about when to call the action group.

Question 5

An application deployed on AWS Lambda is receiving 429 Too Many Requests errors when calling Amazon Bedrock. The developer has verified that account-level quotas have not been reached. What is the most likely cause?

Accepted Answer

The application is exceeding the tokens-per-minute (TPM) limit for the specific model in that AWS Region.. A 429 Too Many Requests from Bedrock, despite account-level quotas being fine, most commonly means the application is hitting the per-model tokens-per-minute (TPM) or requests-per-minute (RPM) limit for that specific model in that Region. Bedrock enforces model-specific quotas that are separate from account-level limits, so a single high-volume model can throttle even when overall account usage is low.

Answer

The VPC endpoint policy is blocking traffic to the Bedrock runtime service.

Answer

The Lambda function timeout is set too low for the model response time.

Answer

The IAM role is missing the bedrock:InvokeModelWithResponseStream permission.

Question 6

A developer is using Amazon Bedrock to build a customer support chatbot. The model is hallucinating frequently when retrieving internal company knowledge. Which evaluation methodology should the developer implement to quantify the ground truth accuracy of the RAG pipeline?

Accepted Answer

Implement a RAGAS-based evaluation pipeline to measure faithfulness and answer relevance.. RAGAS (Retrieval-Augmented Generation Assessment) is a framework specifically designed to evaluate RAG pipelines. It measures faithfulness (whether the answer is grounded in the retrieved context) and answer relevance (how relevant the answer is to the question). These metrics directly quantify ground truth accuracy and help identify hallucinations. Therefore, implementing a RAGAS-based evaluation is the correct approach.

Answer

Increase the temperature setting to 1.0 to encourage more creative responses.

Answer

Use Amazon CloudWatch logs to calculate the latency of every inference call.

Answer

Run a standard BLEU score evaluation against the model's training data.

Question 7

A developer is creating a test suite for an LLM application. Which THREE of the following metrics are commonly used to evaluate RAG system performance?

Accepted Answer

Answer Relevance. Faithfulness, answer relevance, and context precision are the key components of RAGAS evaluation.

Answer

Model Training Loss

Answer

Inference Latency

Question 8

A developer is troubleshooting an Amazon Bedrock Agent that is frequently hitting errors. Which TWO of the following steps should be taken to isolate the issue?

Accepted Answer

Check CloudWatch Logs for the associated Action Group Lambda functions.. Option C is correct because Bedrock Agents invoke Action Group Lambda functions to fulfill API operations, and their execution logs, errors, and timeouts are captured in CloudWatch Logs, making them the primary place to isolate backend failures. Option D is correct because the Agent's Trace output shows the step-by-step orchestration (pre-processing, orchestration, knowledge base/action group invocation, and post-processing), revealing exactly which stage or invocation failed. Option A is wrong because Bedrock is a fully managed service with no customer-accessible restart operation, and restarting would not diagnose the root cause. Option B is wrong because raising temperature to 1.0 only increases response randomness and would not isolate an error. Option E is wrong because base foundation models cannot be retrained by customers; customization is done via fine-tuning or continued pre-training, which is unrelated to troubleshooting.

Answer

Restart the Bedrock service.

Answer

Increase the model's temperature to 1.0.

Answer

Re-train the underlying base model.

Question 9

Which THREE of the following are valid components of an Amazon Bedrock Agent test plan?

Accepted Answer

Validating the reasoning path of the agent.. Testing reasoning, tool invocation accuracy, and safety constraints are critical components of an agent test plan.

Answer

Verifying the S3 bucket encryption.

Answer

Measuring model training time.

Question 10

A generative AI application deployed on AWS Lambda invokes an Amazon Bedrock model and occasionally receives a ThrottlingException with the message 'Too many requests'. The developer has already confirmed that the Lambda function's reserved concurrency is well below the account limit. The application must remain reliable without over-provisioning. Which action should the developer take to handle these throttling errors most effectively?

Accepted Answer

Implement an exponential backoff and jitter retry strategy using the AWS SDK for the Bedrock InvokeModel API call.. ThrottlingException from Amazon Bedrock indicates that the request rate exceeded the account's quota. The correct approach is to implement retries with exponential backoff and jitter, which is a standard resilience pattern for transient errors. This allows the application to recover gracefully without manual quota increases, and it works well in distributed systems like Lambda.

Answer

Configure the Lambda function to use provisioned concurrency to ensure immediate scaling.

Answer

Increase the Lambda function's memory size to improve CPU performance and reduce invocation time.

Answer

Switch the model invocation to use the Amazon Bedrock Converse API instead of InvokeModel.

Question 11

A developer is troubleshooting an Amazon Bedrock Agent that fails to complete a multi-step task. CloudWatch Logs show that the agent successfully invokes the action group's Lambda function, but the agent then replies to the user that it cannot proceed. The Lambda function returns a JSON object with a top-level key "result" instead of the key the agent expects. Which change will most likely resolve the issue?

Accepted Answer

Modify the Lambda function to return the response in the format defined by the action group's API schema, including the required response body structure.. Bedrock Agents rely on the action group's OpenAPI schema to validate and parse the Lambda response. When the function returns a payload whose structure does not match the schema, the agent cannot extract the expected fields and aborts the orchestration, even though the invocation itself succeeded. Correcting the Lambda to emit the schema-defined response body is the direct fix.

Answer

Enable the agent's "return control" option so the agent can bypass the Lambda response and generate the final answer directly.

Answer

Add a new action group to the agent that duplicates the same Lambda function but uses a different API schema version.

Answer

Increase the Lambda function's memory size to 1024 MB and extend the timeout to 5 minutes to allow the agent more time to parse the response.

Question 12

A developer is testing a retrieval-augmented generation (RAG) application built on Amazon Bedrock Knowledge Bases. For a specific question, the model returns a confident but incorrect answer, and the retrieved chunks shown in the test console are only loosely related to the question. The developer wants to diagnose whether the retrieval step or the generation step is at fault. Which approach is most effective?

Accepted Answer

Use the Retrieve API to fetch chunks for the question and inspect their relevance scores and content independently of generation.. To separate retrieval failures from generation failures, you must inspect the retrieved chunks independently. The Retrieve API returns the chunks and relevance scores without invoking the generation model, so you can verify whether the correct context was available. If the right context is present but the answer is wrong, generation is at fault; otherwise, retrieval tuning is needed.

Answer

Add a guardrail that blocks unsupported answers and observe whether the response is filtered, indicating a generation problem.

Answer

Switch the knowledge base to a different embedding model and re-ingest all documents, then compare the final answers.

Answer

Increase the model's temperature to 1.0 and re-run the query to see if the answer changes, then compare outputs.

Question 13

A developer is validating a generative AI application that uses Amazon Bedrock. The application must produce deterministic, repeatable outputs for regression testing. Which TWO configuration changes should the developer make to achieve this? (Choose two.)

Accepted Answer

Provide a fixed random seed using the model's supported seed parameter, if available for that model.. Deterministic regression testing requires minimizing sampling randomness. Setting temperature to 0 makes token selection greedy, and using a fixed seed where supported further stabilizes outputs across identical requests. These two settings together provide the most repeatable behavior available from Bedrock models, which is essential for comparing outputs across application versions.

Answer

Enable streaming responses to capture partial outputs for comparison.

Answer

Increase the maxTokenCount parameter to the model's maximum to ensure complete answers.

Answer

Set the topP parameter to 1.0 and leave temperature at its default value.

Question 14

A developer is testing an Amazon Bedrock application and wants to verify that the model's responses do not contain personally identifiable information (PII) before the application displays them to users. The developer needs a managed way to detect and block PII in both prompts and responses. Which AWS feature should be used?

Accepted Answer

Amazon Bedrock Guardrails with the sensitive information filter configured for PII.. Amazon Bedrock Guardrails include a sensitive information filter that can detect and block or mask PII in both input prompts and model responses. Configuring this filter enforces PII policy at inference time without custom code, satisfying the need for a managed solution that covers both directions of the interaction.

Answer

Amazon Comprehend PII detection called manually after each inference to filter outputs.

Answer

AWS WAF with a custom rule to inspect request and response bodies for PII patterns.

Answer

Amazon Macie to scan the model's responses for PII before display.

Question 15

A developer is troubleshooting a generative AI application that calls Amazon Bedrock. Intermittently, invocations fail with a ThrottlingException. CloudWatch metrics show that the account's on-demand quota for the model is being exceeded during peak hours. The developer wants to reduce throttling without changing the model or application logic. Which solution is most appropriate?

Accepted Answer

Implement exponential backoff with jitter in the AWS SDK retry configuration and request a quota increase for the model.. Intermittent ThrottlingException from Bedrock indicates the account is exceeding its on-demand quota. Exponential backoff with jitter allows retries to succeed when throttling is transient, and requesting a quota increase raises the sustained capacity limit. This combination reduces throttling without modifying the model or application logic, addressing both immediate and long-term needs.

Answer

Reduce the maxTokenCount parameter to 1 to lower token consumption per request.

Answer

Switch the application to use a different AWS Region where the model is available and retry all requests there.

Answer

Enable provisioned throughput for the model and remove all retry logic from the application.