Scenario practice questions

Refer to the exhibit. The application is hitting rate limits during peak hours. What is the best architectural change to improve operational resilience?

Exhibit

2024-10-12 14:02:01 ERROR: Anthropic.Client: 429 Too Many Requests (Rate limit exceeded)
2024-10-12 14:02:02 WARN: Retrying in 500ms...
2024-10-12 14:02:03 WARN: Retrying in 1000ms...

Trap 1: Increase the timeout duration of the API calls indefinitely.

Indefinite timeouts are dangerous and can cause thread exhaustion in the application, leading to cascading failures. It does not address the 429 error and will only make the system less responsive. Effective error handling must be active and intelligent, not simply passive waiting for longer periods.

Trap 2: Bypass the API gateway and send requests directly to the model's…

Bypassing the API gateway is a violation of security protocols and operational governance. It does not prevent rate limiting, as the model provider's service will still enforce these limits at the infrastructure level. This approach is unstable, unauthorized, and provides no solution to the rate-limiting problem.

Trap 3: Disable all retries to prevent the application from crashing.

Disabling retries means that every transient error results in a failed user request, leading to a poor user experience. While it prevents cascading failure, it is not an effective solution to the underlying need for high availability. Intelligent retries are essential for robust LLM application design.

A
Increase the timeout duration of the API calls indefinitely.
Why it fails: Indefinite timeouts are dangerous and can cause thread exhaustion in the application, leading to cascading failures. It does not address the 429 error and will only make the system less responsive. Effective error handling must be active and intelligent, not simply passive waiting for longer periods.
B
Implement an exponential backoff strategy with jitter.
Exponential backoff with jitter is the recommended strategy for handling transient rate limits. By increasing the wait time between retries and adding randomness, the application avoids overwhelming the API upon recovery. This ensures a stable, resilient architecture that handles traffic spikes gracefully without persistent errors.
C
Bypass the API gateway and send requests directly to the model's backend IP.
Why it fails: Bypassing the API gateway is a violation of security protocols and operational governance. It does not prevent rate limiting, as the model provider's service will still enforce these limits at the infrastructure level. This approach is unstable, unauthorized, and provides no solution to the rate-limiting problem.
D
Disable all retries to prevent the application from crashing.
Why it fails: Disabling retries means that every transient error results in a failed user request, leading to a poor user experience. While it prevents cascading failure, it is not an effective solution to the underlying need for high availability. Intelligent retries are essential for robust LLM application design.

Question 2hardmultiple choice

A regulated healthcare client is reviewing your Claude-based patient-intake summarization tool. Their compliance officer asks how you will handle a scenario where the model produces a clinically inaccurate summary. Which response demonstrates the strongest lifecycle and stakeholder communication practice?

Trap 1: State that the tool is a decision-support aid and that all…

While clinician review is essential, framing the entire answer as a liability transfer is adversarial and evasive. It ignores the client's legitimate question about how the system will be monitored, corrected, and improved. Compliance officers expect vendors to own their part of the control environment, and deflecting entirely to the clinician undermines the partnership the engagement depends on.

Trap 2: Offer to switch to a smaller, faster model so that summaries are…

Speed does not reduce clinical inaccuracy and can increase it, since smaller models generally produce weaker summarization. This option also misidentifies the compliance concern as a latency problem rather than a correctness and accountability problem. Proposing a model downgrade in a regulated context without evaluation evidence would raise new questions about your change-management discipline.

Trap 3: Guarantee that the model will not produce inaccurate summaries…

No LLM can be guaranteed to eliminate clinically inaccurate summaries, and Anthropic's documentation explicitly frames safety training as risk reduction, not elimination. Making this guarantee misrepresents the technology and creates an untenable liability position. A compliance officer who later observes an error would lose trust in every other assurance the team provides, which is far more damaging than an honest limitation statement.

A
State that the tool is a decision-support aid and that all liability rests with the clinician who reviews the output.
Why it fails: While clinician review is essential, framing the entire answer as a liability transfer is adversarial and evasive. It ignores the client's legitimate question about how the system will be monitored, corrected, and improved. Compliance officers expect vendors to own their part of the control environment, and deflecting entirely to the clinician undermines the partnership the engagement depends on.
B
Offer to switch to a smaller, faster model so that summaries are generated quickly enough for clinicians to catch errors in real time.
Why it fails: Speed does not reduce clinical inaccuracy and can increase it, since smaller models generally produce weaker summarization. This option also misidentifies the compliance concern as a latency problem rather than a correctness and accountability problem. Proposing a model downgrade in a regulated context without evaluation evidence would raise new questions about your change-management discipline.
C
Guarantee that the model will not produce inaccurate summaries because Anthropic's safety training prevents harmful outputs.
Why it fails: No LLM can be guaranteed to eliminate clinically inaccurate summaries, and Anthropic's documentation explicitly frames safety training as risk reduction, not elimination. Making this guarantee misrepresents the technology and creates an untenable liability position. A compliance officer who later observes an error would lose trust in every other assurance the team provides, which is far more damaging than an honest limitation statement.
D
Describe an incident response process with human-in-the-loop review, error logging, root-cause analysis, and a defined escalation path to the clinical team.
This response acknowledges that errors are possible and demonstrates a controlled, auditable process for detecting, containing, and learning from them. Human review before clinical use, structured logging for traceability, and a clear escalation path are exactly the controls a compliance officer needs to see. It also keeps the clinician accountable for the final decision, which aligns the AI system with existing clinical governance.

Question 3hardmultiple choice

A research agent uses Claude with an extended thinking budget to analyze a 200-page regulatory filing. Mid-analysis it must call a `fetch_footnote` tool whose result is essential to the conclusion. The architect wants the tool result to be incorporated without discarding the model's prior reasoning. Which approach best achieves this?

Trap 1: Store the footnote in an external vector store and instruct the…

Deferring retrieval adds an extra round trip and does not guarantee the same footnote is surfaced, since vector search returns approximate matches. The agent needs the exact result now to continue its reasoning; storing it elsewhere introduces retrieval risk and latency without benefit. This also fails to preserve the tool_use/tool_result pairing the API requires.

Trap 2: Append the tool result as a new user message and restart the…

Restarting discards the accumulated reasoning and burns the entire thinking budget again, which is both wasteful and likely to produce a different chain that may not reach the same point. It also risks exceeding the context window on a 200-page document. The goal is to preserve prior reasoning, and this approach explicitly throws it away.

Trap 3: Summarize the footnote with a separate cheap model and paste the…

Mutating the system prompt mid-conversation breaks prompt cache assumptions and can invalidate prior reasoning context, and a secondary model's summary may drop legally material detail. It also severs the link between the tool call and its result, which the API expects to be paired. This introduces lossy transformation where exact fidelity matters.

A
Store the footnote in an external vector store and instruct the agent to retrieve it again later if needed.
Why it fails: Deferring retrieval adds an extra round trip and does not guarantee the same footnote is surfaced, since vector search returns approximate matches. The agent needs the exact result now to continue its reasoning; storing it elsewhere introduces retrieval risk and latency without benefit. This also fails to preserve the tool_use/tool_result pairing the API requires.
B
Append the tool result as a new user message and restart the analysis from the beginning of the filing.
Why it fails: Restarting discards the accumulated reasoning and burns the entire thinking budget again, which is both wasteful and likely to produce a different chain that may not reach the same point. It also risks exceeding the context window on a 200-page document. The goal is to preserve prior reasoning, and this approach explicitly throws it away.
C
Summarize the footnote with a separate cheap model and paste the summary into the system prompt before the next call.
Why it fails: Mutating the system prompt mid-conversation breaks prompt cache assumptions and can invalidate prior reasoning context, and a secondary model's summary may drop legally material detail. It also severs the link between the tool call and its result, which the API expects to be paired. This introduces lossy transformation where exact fidelity matters.
D
Insert the tool result into the conversation as a tool_result block associated with the original tool_use, then continue the same turn so the model resumes from its existing reasoning.
Returning the result as a tool_result block tied to the preceding tool_use preserves the message sequence the model already reasoned over, letting it continue the same turn without re-deriving prior steps. This is the canonical way to feed external data into an in-progress reasoning chain. It keeps the thinking budget intact and respects the API's tool-use contract.

Question 4easymultiple choice

A fintech startup wants to use Claude to generate marketing copy that references competitor products by name and makes performance comparisons. Legal counsel asks the architect which governance step is most appropriate before this capability goes live.

Trap 1: Increase the model's temperature setting so the copy sounds more…

Temperature affects randomness and stylistic variation in generated text, not legal exposure. Raising it could actually increase the chance of unsupported or defamatory comparative claims. The architect's concern is whether the use case is permissible and compliant, which sampling parameters cannot address, so this change would not satisfy counsel's request.

Trap 2: Enable prompt caching to lower the cost of generating large volumes…

Prompt caching reduces latency and token cost for repeated prefixes. It has no relationship to whether comparative advertising against named competitors is permitted. Adopting it would leave the underlying policy and regulatory risk untouched, so it does not answer the governance question counsel posed before launch.

Trap 3: Add a spell-check and grammar pass to the generated marketing copy…

Proofreading improves readability but says nothing about whether naming competitors and making performance claims is allowed under Anthropic's policies or advertising rules. A perfectly spelled comparative claim can still be unlawful or out of policy. This control addresses presentation quality, not the eligibility and compliance question counsel raised.

A
Establish a documented review process that classifies the use case against Anthropic's Usage Policies and applicable advertising regulations before deployment.
Generating comparative claims about named competitors raises both usage-policy and regulatory questions, so a documented classification and review step is the right governance gate. It ensures the use case is assessed against Anthropic's policies and advertising law before any content is produced. The other measures improve output quality or performance but do not resolve the legal and policy eligibility question.
B
Increase the model's temperature setting so the copy sounds more creative and varied.
Why it fails: Temperature affects randomness and stylistic variation in generated text, not legal exposure. Raising it could actually increase the chance of unsupported or defamatory comparative claims. The architect's concern is whether the use case is permissible and compliant, which sampling parameters cannot address, so this change would not satisfy counsel's request.
C
Enable prompt caching to lower the cost of generating large volumes of marketing variants.
Why it fails: Prompt caching reduces latency and token cost for repeated prefixes. It has no relationship to whether comparative advertising against named competitors is permitted. Adopting it would leave the underlying policy and regulatory risk untouched, so it does not answer the governance question counsel posed before launch.
D
Add a spell-check and grammar pass to the generated marketing copy before publication.
Why it fails: Proofreading improves readability but says nothing about whether naming competitors and making performance claims is allowed under Anthropic's policies or advertising rules. A perfectly spelled comparative claim can still be unlawful or out of policy. This control addresses presentation quality, not the eligibility and compliance question counsel raised.

Question 5mediummultiple choice

Refer to the exhibit. Your agent is receiving this error during a high-concurrency operation. Which implementation correctly handles this scenario?

Exhibit

{
  "error": "RateLimitError",
  "message": "Too many requests in the last 60 seconds",
  "retry_after": 5
}

Trap 1: Discard the task and return a failure message to the user.

Discarding the task is a poor user experience. Rate limits are often transient. A well-designed agent should have the capability to wait and retry automatically before giving up, which allows it to handle temporary spikes in traffic without manual user intervention or task failure.

Trap 2: Immediately retry the request without any delay.

Immediate retries will almost certainly fail again and potentially exacerbate the rate limit issue. This behavior constitutes a 'retry storm' that can lead to permanent blacklisting of your API key. You must always wait for the specified time or use exponential backoff to allow the system to stabilize.

Trap 3: Switch to a backup API key and continue immediately.

Using backup keys to bypass rate limits can lead to account suspension. Most providers track rate limits by organization or underlying billing entity, not just by individual key. You should architect for resiliency through backoff, rather than attempting to circumvent the limits set by the API provider.

A
Discard the task and return a failure message to the user.
Why it fails: Discarding the task is a poor user experience. Rate limits are often transient. A well-designed agent should have the capability to wait and retry automatically before giving up, which allows it to handle temporary spikes in traffic without manual user intervention or task failure.
B
Immediately retry the request without any delay.
Why it fails: Immediate retries will almost certainly fail again and potentially exacerbate the rate limit issue. This behavior constitutes a 'retry storm' that can lead to permanent blacklisting of your API key. You must always wait for the specified time or use exponential backoff to allow the system to stabilize.
C
Wait for the duration specified in 'retry_after' before retrying.
Respecting the 'retry_after' duration is the standard way to handle rate limiting. It aligns your agent's behavior with the server's requirements, ensuring that your retry attempts occur after the limit has reset. This is the most efficient and polite way to interact with rate-limited external APIs.
D
Switch to a backup API key and continue immediately.
Why it fails: Using backup keys to bypass rate limits can lead to account suspension. Most providers track rate limits by organization or underlying billing entity, not just by individual key. You should architect for resiliency through backoff, rather than attempting to circumvent the limits set by the API provider.

Question 6mediummultiple choice

Which pattern is most suitable for an agent that must balance the need for speed (latency) versus the need for correctness in a customer support scenario?

Trap 1: Always prioritize speed by using the smallest, fastest model…

Prioritizing speed at the cost of correctness leads to poor customer satisfaction when the answers provided are wrong or hallucinatory. In customer support, accuracy is paramount. An architecture that ignores verification in favor of speed fails to address the fundamental requirement of providing reliable, helpful assistance to the user.

Trap 2: Force the user to wait for verification before showing any response.

Forcing users to wait for a comprehensive check creates a frustrating experience and high bounce rates. Modern expectations demand immediate interactivity. An architecture that blocks the UI for verification is outdated; systems must be designed to handle background processing while maintaining a fluid and responsive interface for the end-user.

Trap 3: Use a RAG-only architecture to guarantee factual accuracy.

RAG improves factual accuracy by grounding the model, but it does not inherently manage latency or the trade-off between speed and correctness. Even with RAG, you still need an architectural strategy to handle user expectations regarding response time and to verify the generated outputs, making it incomplete as a solution.

A
Always prioritize speed by using the smallest, fastest model available.
Why it fails: Prioritizing speed at the cost of correctness leads to poor customer satisfaction when the answers provided are wrong or hallucinatory. In customer support, accuracy is paramount. An architecture that ignores verification in favor of speed fails to address the fundamental requirement of providing reliable, helpful assistance to the user.
B
Use a dual-path architecture with immediate streaming and background verification.
This architecture provides the best of both worlds: immediate feedback for the user and a secondary validation layer that ensures the content is accurate. This pattern manages the trade-off between perceived latency and result quality, delivering a superior user experience without sacrificing the precision required for high-quality support.
C
Force the user to wait for verification before showing any response.
Why it fails: Forcing users to wait for a comprehensive check creates a frustrating experience and high bounce rates. Modern expectations demand immediate interactivity. An architecture that blocks the UI for verification is outdated; systems must be designed to handle background processing while maintaining a fluid and responsive interface for the end-user.
D
Use a RAG-only architecture to guarantee factual accuracy.
Why it fails: RAG improves factual accuracy by grounding the model, but it does not inherently manage latency or the trade-off between speed and correctness. Even with RAG, you still need an architectural strategy to handle user expectations regarding response time and to verify the generated outputs, making it incomplete as a solution.

Question 7hardmulti select

When conducting a risk assessment for a new Claude-based customer support bot, which TWO factors should be prioritized as 'High Risk' according to Anthropic's safety guidelines?

Trap 1: Generating personalized marketing copy for a retail website.

Marketing copy generation is generally considered a low-risk application of AI. While it requires quality control for brand alignment, the potential for causing significant harm or violating fundamental safety principles is minimal compared to regulated fields like healthcare or finance.

Trap 2: Summarizing publicly available news articles for internal research.

Summarization of public data is a standard, low-risk use case. The primary risk is hallucination, which can be managed with grounding techniques. It does not involve the sensitive personal data or high-stakes outcomes that characterize high-risk AI deployments.

Trap 3: Translating internal training manuals into multiple languages.

Language translation is a productivity-enhancing task that carries low safety risk. While accuracy is important for the quality of the training, it does not typically involve the legal, safety, or ethical complexities found in high-risk categories like medical diagnostics or automated law enforcement.

A
Providing automated, unreviewed medical or legal advice.
Medical and legal domains are considered high-risk because incorrect information can lead to severe personal harm or legal liability. Anthropic's safety guidelines emphasize that AI should not replace professional judgment in these areas without significant human-in-the-loop oversight and clear disclaimers to the end-user.
B
Generating personalized marketing copy for a retail website.
Why it fails: Marketing copy generation is generally considered a low-risk application of AI. While it requires quality control for brand alignment, the potential for causing significant harm or violating fundamental safety principles is minimal compared to regulated fields like healthcare or finance.
C
Summarizing publicly available news articles for internal research.
Why it fails: Summarization of public data is a standard, low-risk use case. The primary risk is hallucination, which can be managed with grounding techniques. It does not involve the sensitive personal data or high-stakes outcomes that characterize high-risk AI deployments.
D
Automated processing of loan applications without human review.
Automated financial decision-making can lead to systemic bias and unfair outcomes, which are major governance concerns. Such systems must be carefully audited for fairness and transparency, and typically require a human to make the final determination to comply with financial regulations and ethical standards.
E
Translating internal training manuals into multiple languages.
Why it fails: Language translation is a productivity-enhancing task that carries low safety risk. While accuracy is important for the quality of the training, it does not typically involve the legal, safety, or ethical complexities found in high-risk categories like medical diagnostics or automated law enforcement.