Be able to design a Bedrock Agent test plan, isolate agent errors with trace and CloudWatch, and evaluate RAG with the right metrics. The key is separating retrieval failures from generation failures before changing prompts or models.
Start practicing
Testing Validation And Troubleshooting — choose a session length
Free · No account required
Domain overview
This domain covers how to verify, evaluate, and debug generative AI workloads on AWS. It is tested through scenario questions about Amazon Bedrock Agents, Knowledge Bases, and RAG pipelines, asking you to pick valid test-plan components, isolate failure causes, choose evaluation metrics, and diagnose confident-but-wrong answers from retrieved context.
Exam objectives
Building Amazon Bedrock Agent test plans covering action groups, orchestration, and session behavior
Isolating Bedrock Agent errors using CloudWatch Logs, trace, and invocation metrics
Selecting RAG evaluation metrics such as faithfulness, relevance, and context precision
Diagnosing RAG failures by inspecting retrieved chunks, embeddings, and Knowledge Base configuration
Treating a confident wrong answer as a model problem instead of checking whether retrieval returned irrelevant or missing chunks.
Assuming all RAG metrics measure the same thing; confusing answer relevance with context relevance or faithfulness.
Debugging Bedrock Agents only from the application side while ignoring agent trace, action group Lambda logs, and CloudWatch metrics.
Click any question to see the full explanation and answer options, or start a focused practice session above.
An application deployed on AWS Lambda is receiving 429 Too Many Requests errors when calling Amazon Bedrock. The developer has verified that account-level quotas have not been reached. What is the most likely cause?
2A developer is using Amazon Bedrock to build a customer support chatbot. The model is hallucinating frequently when retrieving internal company knowledge. Which evaluation methodology should the developer implement to quantify the ground truth accuracy of the RAG pipeline?
3A developer is creating a test suite for an LLM application. Which THREE of the following metrics are commonly used to evaluate RAG system performance?
4A developer is troubleshooting an Amazon Bedrock Agent that is frequently hitting errors. Which TWO of the following steps should be taken to isolate the issue?
5Which THREE of the following are valid components of an Amazon Bedrock Agent test plan?
6A generative AI application deployed on AWS Lambda invokes an Amazon Bedrock model and occasionally receives a ThrottlingException with the message 'Too many requests'. The developer has already confirmed that the Lambda function's reserved concurrency is well below the account limit. The application must remain reliable without over-provisioning. Which action should the developer take to handle these throttling errors most effectively?
7A developer is troubleshooting an Amazon Bedrock Agent that fails to complete a multi-step task. CloudWatch Logs show that the agent successfully invokes the action group's Lambda function, but the agent then replies to the user that it cannot proceed. The Lambda function returns a JSON object with a top-level key "result" instead of the key the agent expects. Which change will most likely resolve the issue?
8A developer is testing a retrieval-augmented generation (RAG) application built on Amazon Bedrock Knowledge Bases. For a specific question, the model returns a confident but incorrect answer, and the retrieved chunks shown in the test console are only loosely related to the question. The developer wants to diagnose whether the retrieval step or the generation step is at fault. Which approach is most effective?
9A developer is validating a generative AI application that uses Amazon Bedrock. The application must produce deterministic, repeatable outputs for regression testing. Which TWO configuration changes should the developer make to achieve this? (Choose two.)
10A developer is testing an Amazon Bedrock application and wants to verify that the model's responses do not contain personally identifiable information (PII) before the application displays them to users. The developer needs a managed way to detect and block PII in both prompts and responses. Which AWS feature should be used?
11A developer is troubleshooting a generative AI application that calls Amazon Bedrock. Intermittently, invocations fail with a ThrottlingException. CloudWatch metrics show that the account's on-demand quota for the model is being exceeded during peak hours. The developer wants to reduce throttling without changing the model or application logic. Which solution is most appropriate?
12A developer is using Amazon Bedrock Guardrails to filter out hate speech and violence in a chatbot application. During testing, the developer notices that some prompts containing subtle hate speech are not being blocked. The developer wants to improve the detection rate without increasing false positives on benign content. Which action should the developer take?
Be able to design a Bedrock Agent test plan, isolate agent errors with trace and CloudWatch, and evaluate RAG with the right metrics. The key is separating retrieval failures from generation failures before changing prompts or models.
The Courseiva AIP-C01 question bank contains 12 questions in the Testing Validation And Troubleshooting domain. Click any question to see the full explanation and answer breakdown.
Start with a 10-question focused session to identify your baseline accuracy in this domain. Read every explanation — even for questions you answer correctly — to understand the reasoning. Once you score consistently above 80%, move to a 20–30 question session to confirm depth before moving to the next domain.
Yes — the session launcher on this page draws questions exclusively from the Testing Validation And Troubleshooting domain. Choose 10, 20, 30, or 50 questions for a focused session, or click individual questions to review them one by one.
Save your results, see per-domain analytics, and get readiness scores — free, for every certification.
Sign Up FreeFree forever · Every certification included