Hard Difficulty Questions

Reactive Tool Calling (ReAct)

Why it fails: The ReAct pattern focuses on the immediate next step based on the current observation. While flexible, it can easily lose the 'big picture' in long migrations with many dependencies, leading the agent to wander off-task or fail to recognize that a previous failure has invalidated the entire migration plan.

Zero-shot prompting with all tools

Why it fails: Asking a model to perform a complex migration in a single turn without a structured loop is highly likely to result in failure. The model cannot anticipate the output of intermediate steps, leading to hallucinations about data states or the generation of invalid commands that haven't been tested against real-world results.

Few-shot prompting with migration examples

Why it fails: Few-shot examples help with formatting and style but do not provide the structural state management needed for complex, multi-dependency workflows. Examples cannot cover every edge case in a migration, and the agent still lacks a mechanism to track its current progress through a long-running, stateful sequence of operations.

Parallel Sub-tasking

Why it fails: Parallel execution is inappropriate for tasks with strict sequential dependencies. If step B requires the output of step A, running them simultaneously will cause step B to fail. This architecture is designed for independent workloads and would break the logic required for a coherent, step-by-step data migration process.

Catch the error and retry the request immediately in a loop.

Why it fails: Immediate retries can lead to a thundering herd problem, where the application continuously hammers the API, worsening the rate limiting issue. This approach ignores the need for backoff, likely resulting in repeated failures and potential service degradation for other parts of the application or other API users.

Increase the timeout duration for all API calls in the application.

Why it fails: Increasing timeouts does not address the fundamental issue of rate limiting. If the API is rejecting requests due to volume, waiting longer for a response will not result in a successful request. It only delays the error handling process and may lead to resource exhaustion in the client application.

Switch to a synchronous architecture to serialize all API requests.

Why it fails: Serializing requests eliminates concurrency but significantly decreases throughput and user experience. It is not a scalable solution for high-load scenarios. A robust design should support high concurrency through intelligent request management and backoff strategies rather than artificially constraining the system to sequential processing, which limits growth potential.

Increase the temperature parameter to 1.5 to maximize response creativity.

Why it fails: High temperature settings increase the randomness of the model's output, which is counterproductive for high-stakes analysis. Increased randomness introduces greater instability and potential for hallucinated figures, directly undermining the accuracy and reliability required for financial decision-making processes within the enterprise environment.

Require human intervention for every prompt sent to the API to guarantee zero errors.

Why it fails: Requiring human intervention for every prompt renders the automated AI solution non-scalable and impractical for high-volume financial workflows. Governance should focus on automated guardrails and exception-based manual review rather than forcing a human-in-the-loop requirement for every single interaction, which ruins operational efficiency.

Disable all safety filters to allow the model to process complex financial jargon.

Why it fails: Disabling safety filters removes critical guardrails that prevent the model from generating harmful, biased, or nonsensical content. This increases operational risk without providing a meaningful benefit to accuracy, as safety filters do not inherently prevent the model from understanding or processing complex financial terminology.

Enable the model to have full administrative access to the file system.

Why it fails: Granting administrative access is a major security vulnerability. An agent should operate within a heavily sandboxed environment with the minimum set of permissions required. Full access would allow a prompt-injection attack to potentially compromise the entire infrastructure and lead to catastrophic data leaks or system destruction.

Embed all user data directly into the system prompt for faster access.

Why it fails: Putting sensitive user data in the system prompt increases the risk of exposure and makes it impossible to manage permissions effectively. It also bloats the context window unnecessarily. Data should be fetched dynamically with appropriate security controls, rather than hard-coded into the persistent system prompt.

Physical security of the data centers housing the TPU/GPU clusters.

Why it fails: The physical security of the hardware and the environment where the models are trained and hosted is the responsibility of Anthropic and its cloud infrastructure partners. Customers have no control over data center access or hardware maintenance, as these are managed at the platform and infrastructure layers.

Fine-tuning the base model's Constitutional AI principles.

Why it fails: Constitutional AI principles are baked into the model's core during the training and RLHF phases by Anthropic researchers. Customers cannot modify these foundational safety guardrails; instead, they must use system prompts and external moderation tools to layer their own specific business rules on top of the model.

Hardcode system prompts to ensure the model behavior cannot change over time.

Why it fails: Hardcoding prompts is a maintenance nightmare. It prevents updating instructions to improve quality or safety and makes version control impossible. Over time, as your application needs evolve, being unable to update prompts will force you into difficult refactoring sessions, which is the opposite of productive architecture.

Use the largest available context window for every single request to maximize intelligence.

Why it fails: Using the maximum context window for every request is wasteful and inefficient. It increases costs significantly and can introduce unnecessary latency. Architects should advise using the smallest context window necessary for the task, which improves efficiency and keeps costs predictable, demonstrating a mature approach to resource management.

Increase the timeout duration of the API calls indefinitely.

Why it fails: Indefinite timeouts are dangerous and can cause thread exhaustion in the application, leading to cascading failures. It does not address the 429 error and will only make the system less responsive. Effective error handling must be active and intelligent, not simply passive waiting for longer periods.

Bypass the API gateway and send requests directly to the model's backend IP.

Why it fails: Bypassing the API gateway is a violation of security protocols and operational governance. It does not prevent rate limiting, as the model provider's service will still enforce these limits at the infrastructure level. This approach is unstable, unauthorized, and provides no solution to the rate-limiting problem.

Disable all retries to prevent the application from crashing.

Why it fails: Disabling retries means that every transient error results in a failed user request, leading to a poor user experience. While it prevents cascading failure, it is not an effective solution to the underlying need for high availability. Intelligent retries are essential for robust LLM application design.

Assure them that the model is smart enough to handle PII.

Why it fails: Trusting a model to 'handle' PII autonomously is a massive, irresponsible risk. The architect must proactively implement controls to prevent PII from reaching the model or to ensure it is handled according to strict privacy policies. This answer shows a complete lack of understanding of data privacy requirements for LLMs.

Tell them the model does not store any data anyway.

Why it fails: This response is misleading and fails to address the risk of data exposure during inference. Even if data isn't permanently stored, it is still transmitted and processed. Focusing only on storage ignores the critical risks involved in the transit and processing of data, which is where many leaks occur.

Suggest moving the project to an on-premise LLM to ensure security.

Why it fails: Moving to on-premise infrastructure is a major strategic change that may not be feasible or necessary. The architect should first address the security controls for the current implementation. Proposing this as a knee-jerk reaction avoids the actual problem and likely introduces significant new costs and deployment complexities.

The input parameters do not match the JSON schema

Why it fails: If the input parameters were invalid according to the defined schema, the error would typically occur during the validation phase before the tool is even executed. However, the exhibit shows the model successfully generating the call, suggesting the problem lies in the subsequent step of the orchestration loop, not the schema.

The tool_use ID is missing the required prefix

Why it fails: The provided ID 'tool_102' is a valid string representation within the content block. While Anthropic IDs often follow specific formats like 'toolu_', the API itself accepts various string identifiers. A prefix mismatch would not stop the orchestrator from attempting a response; it would more likely result in an API validation error.

The assistant content block lacks a stop_reason

Why it fails: The 'stop_reason' is a property of the API response metadata, not a field that resides within the content array of the assistant's message. The absence of metadata in the JSON snippet provided does not indicate an architectural failure, as the exhibit focuses on the structure of the message content itself.

Disable all logging to protect user privacy.

Why it fails: Disabling all logging removes the ability to audit and debug issues, which harms reliability and incident response. Privacy can be protected through redaction and access controls rather than eliminating logs entirely. Without logs, teams cannot investigate failures or improve the system, making this practice detrimental to operational enablement.

Set the model's temperature to a high value to encourage diverse responses.

Why it fails: High temperature increases randomness, which reduces reliability and consistency. For a production assistant, deterministic and accurate responses are preferred. High temperature can lead to hallucinations and unpredictable behavior, undermining safety. Therefore, this practice is counterproductive for the stated goal.

Log all user prompts and model outputs for debugging and auditing.

Why it fails: While logging can aid debugging, logging all prompts and outputs may introduce privacy and security risks, especially if they contain sensitive data. It also does not inherently improve reliability or safety; in fact, without proper safeguards, logs can become a liability. The scenario asks for practices that ensure reliability and safety, so logging alone is insufficient and potentially harmful.

Embed the entire corpus into the system prompt so the model always has full context available.

Why it fails: Embedding a full corpus in the system prompt is impractical for any realistic size and would vastly exceed context limits while dramatically increasing cost and latency. It also does not solve relevance: a model faced with an enormous undifferentiated context can still attend to the wrong passages. This approach bypasses retrieval rather than improving it and is not a viable architectural change for the described system.

Increase the retrieval count to 50 so the correct chunk is more likely to be included somewhere in the context.

Why it fails: Retrieving more chunks increases the chance that the correct passage appears, but it also floods the context with additional irrelevant material, worsening the citation problem and raising token cost. The scenario explicitly asks for improvement without increasing the number of retrieved chunks. More candidates without better ranking does not fix the precision issue caused by overlapping terminology.

Lower the model's temperature to zero so it sticks more closely to the retrieved text.

Why it fails: Temperature affects sampling randomness, not the relevance of retrieved chunks. At temperature zero the model is more deterministic, but it will still see the same irrelevant chunks and may still cite them. The root cause is retrieval precision, not sampling. Lowering temperature does not promote the correct passage or demote noisy ones, so the grounding problem persists.

Require the model to emit a SHA-256 hash of its own reasoning text inside the tool input, and verify that hash in the downstream trade service before execution.

Why it fails: Self-reported hashes are generated by the same model whose behavior is under audit, so they provide no independent guarantee and can be fabricated or inconsistent across retries. Hashing free-form reasoning also produces unstable values because tokenization and sampling vary. Without a platform-issued identifier tying the tool call to the assistant turn, the downstream service cannot actually verify provenance.

Route all trades through a single orchestrator agent that logs a natural-language summary of each decision to an append-only ledger after the trade is confirmed.

Why it fails: A post-hoc natural-language summary is not cryptographically bound to the originating model turn; it can be paraphrased, truncated, or hallucinated by the summarizer. Logging after confirmation also breaks the causal order, since the trade already executed before any record existed. This approach gives narrative auditability but not the deterministic, per-instruction traceability the regulation requires.

Enable prompt caching on the system prompt so that every trade instruction inherits a stable cache key that can be presented to auditors as proof of origin.

Why it fails: Prompt caching reduces latency and cost by reusing processed prefixes; its cache keys are an optimization detail, not an intent-binding identifier. A cache key does not uniquely tie a specific trade instruction to the model turn that produced it, and cache entries can expire or be evicted. Auditors would be unable to reconstruct which model output authorized which trade, so this fails the traceability mandate.

Enable verbose logging of all prompts and responses for forensic analysis.

Why it fails: Verbose logging can aid investigations but does not prevent data leakage; it may even increase risk if logs contain sensitive data. It is a detective, not preventive, control. The risk committee requires controls to prevent leakage, so logging alone is insufficient and could introduce additional privacy concerns if not properly secured.

Require all users to sign an acceptable use policy before accessing the summarization tool.

Why it fails: An acceptable use policy is a administrative control that sets expectations but does not technically prevent data leakage. Users may still inadvertently or maliciously share sensitive data. It is not as effective as technical controls like DLP or redaction, which enforce policies automatically and consistently.

Use a private, dedicated instance of Claude with no internet access and strict network segmentation.

Why it fails: While network segmentation is a good security practice, Claude is a cloud-based service accessed via API, so a private instance without internet access is not a standard deployment option. This option misrepresents the architecture and does not directly address data leakage or privacy compliance in the context of using a hosted LLM service.

Enable verbose debug logging of every prompt and response for later forensic review.

Why it fails: Verbose logging is valuable for investigation but is detective, not preventive. It records what a compromised agent did without restricting what it can do, so refunds and record edits would still occur at full scale. Forensic evidence helps after an incident, but the risk committee asked for controls that limit the damage an agent can cause while it is misbehaving.

Schedule quarterly reviews of the agent's conversation transcripts to identify emerging misuse patterns.

Why it fails: Periodic transcript review supports trend analysis and program improvement, but it operates on a quarterly cadence and cannot contain an active incident. Between reviews, a compromised agent could issue unlimited refunds and alter orders. As a governance rhythm it complements preventive controls, yet it does not reduce the maximum impact of a single malicious or erroneous action.

Publish an acceptable use policy that prohibits employees from manipulating the agent.

Why it fails: A policy communicates expectations and supports disciplinary action, but it does not technically constrain the agent's capabilities. A malicious external user or injected instruction is unaffected by an internal policy document. Since the goal is limiting blast radius from a compromised agent, a non-enforceable policy statement cannot substitute for scoped permissions or approval gates.

Route every request through a gateway that logs the full request and response payloads with the model identifier attached.

Why it fails: Payload logging is valuable for incident forensics and quality review, but logs are observational evidence, not a control over model behavior. A log showing that responses changed overnight does not prevent the change, attribute it to a decision-maker, or provide a rollback path. Regulators asking for attributable, reversible change management need the pinning and evaluation record instead.

Enable prompt caching on all production requests so that previously validated prompts are reused verbatim.

Why it fails: Prompt caching reduces latency and cost by reusing prefix computations, but it does not version model weights or record behavioral change. Cached prompts still execute against whatever model is currently aliased, so a silent upgrade would change outputs while the cache layer reports nothing. It is a performance feature, not an auditability control.

Store the system prompt in a version-controlled repository and require pull-request review before it is merged.

Why it fails: Version-controlling the system prompt governs one input to the system, and review discipline is genuinely useful. However, the model itself can change beneath an unchanged prompt when an alias such as claude-3-5-sonnet-latest silently advances. This practice therefore cannot demonstrate that model behavior changes are tracked or reversible, which is the committee's stated obligation.

Enable Anthropic's Zero Data Retention option in the Anthropic Console for the bank's organization.

Why it fails: Zero Data Retention is an Anthropic Console arrangement that governs how Anthropic handles data for direct API usage; it does not change which AWS regions process a Bedrock inference profile request. It also says nothing about recording which human approved a given invocation. Relying on it here leaves both the geographic processing constraint and the attribution requirement unresolved, making it the wrong control for this architecture.

Configure the Bedrock guardrail with a PII filter set to ANONYMIZE on the input and the model response.

Why it fails: Bedrock guardrails with a PII filter can detect and mask or block sensitive data in prompts and completions, which reduces exposure, but this control operates on content rather than on where inference physically executes. It does not constrain the cross-region profile to EEA regions, and it creates no per-inference record tying the request to a named human approver, so the residency and attribution requirements remain unmet.

Add a system prompt instructing Claude to refuse any request containing a customer name or account number.

Why it fails: Prompt-level instructions are probabilistic and can be overridden by adversarial or merely unusual inputs, so they cannot serve as a regulatory control. More fundamentally, a refusal instruction changes nothing about where the inference runs or whether the call is logged against a named approver. Treating model behavior as the enforcement point confuses a safety nudge with an architectural guarantee that the regulator will accept.

A list of all stakeholders who attended the project kickoff meeting.

Why it fails: Kickoff attendance records are historical project artifacts, not evidence of operational readiness. Stakeholders at kickoff may have changed roles, and attendance implies no ongoing responsibility. The steering committee needs current, forward-looking commitments about who operates the service and under what terms, so a historical attendee list adds no assurance about post-launch support, monitoring, or accountability.

A complete archive of every prompt version ever tested during development.

Why it fails: Historical prompt archives are useful for traceability and audits, but they do not demonstrate operational ownership. Keeping every experimental version also creates confusion about which prompt is authoritative in production. The readiness question is who will operate and maintain the service, not what was tried during development. An archive can support a runbook, but it cannot substitute for documented ownership and support commitments.

A slide deck summarizing the assistant's key features for the marketing team.

Why it fails: A marketing overview communicates capabilities to external or internal audiences but says nothing about operational accountability. It does not identify who fixes incidents, how quality is monitored, or what service levels apply. Presenting it as readiness evidence would leave the steering committee without confirmation that support and maintenance are staffed, which is the core concern of a go-live handover review.

Rely solely on the model's inherent ability to distinguish between user intent and instructions.

Why it fails: Assuming the model can perfectly identify and ignore malicious injections is a security failure. Models are susceptible to sophisticated prompt hacking techniques, and relying on their internal judgment without external validation or input sanitization leaves the application vulnerable to unauthorized command execution and data exfiltration.

Force users to sign a legal agreement regarding acceptable use before accessing the UI.

Why it fails: Legal agreements are reactive and provide no technical mitigation against malicious intent. While important for corporate policy, they do not address the technical reality of prompt injection vulnerabilities. A governance strategy must prioritize technical controls over passive legal deterrents to effectively manage security risk in production.

Disable system prompts entirely to prevent attackers from overriding them.

Why it fails: System prompts are essential for defining model personality, safety boundaries, and task parameters. Disabling them destroys the functionality of the AI application. The correct approach is to secure the prompt through structural design and input validation, not by removing necessary system-level behavioral instructions.

Store all raw user input in unencrypted public buckets.

Why it fails: Storing data in unencrypted public buckets is a severe security failure. It violates almost all compliance frameworks and exposes sensitive data to unauthorized access. Data must always be encrypted at rest and in transit, with strict access control lists (ACLs) to manage who can view the stored information.

Disable all authentication to improve system latency.

Why it fails: Disabling authentication is a major security vulnerability that allows unauthorized access to your resources. Security and compliance always take precedence over minor latency improvements. Properly implemented authentication is a requirement for any enterprise system to ensure that only authorized users or services interact with the AI model.

Blame the provider for not having enough memory for our specific application needs.

Why it fails: Blaming the provider for industry-standard context limits is unprofessional. Architects should understand the constraints of the tools they select and design systems that work within those limits. This response fails to provide a solution and reflects poorly on your capability to manage and architect around known system limitations.

Suggest that we switch to a model with a larger context window, regardless of cost.

Why it fails: Suggesting a hardware or model upgrade without considering the business impact is irresponsible. Even with larger windows, there are performance and cost trade-offs. An architect must evaluate the entire system architecture, including the cost of latency and token usage, before proposing an increase in model capability or context size.

Tell the stakeholder that the code is wrong and needs to be rewritten entirely.

Why it fails: The error is not necessarily a bug in the code, but an architectural limitation regarding input size. Telling the stakeholder the code is 'wrong' creates unnecessary panic and ignores the fact that the application logic may be sound, just misaligned with the current model's token capacity for specific inputs.