Hard Difficulty Questions

Switch every production workload immediately from Claude 3.5 Sonnet to Claude 3 Opus to maximize parallel processing.

Why it fails: Claude 3 Opus consumes more tokens per response than Sonnet, so switching every workload raises expenditure rather than reducing it. Opus suits complex reasoning tasks where Sonnet's quality falls short, but applying it universally to millions of routine support queries ignores the cost driver the team is auditing.

Disable response streaming to batch full payloads and reduce HTTP connection overhead.

Why it fails: Disabling streaming does not reduce token consumption, since input and output tokens are billed identically regardless of delivery method. Streaming affects perceived latency, not cost. It would be correct when a client cannot handle server-sent events or requires complete payloads for downstream processing.

Hardcode maximum output tokens to five thousand across all API calls regardless of task complexity.

Why it fails: Setting excessively high maximum output token limits creates financial exposure if the model hallucinates or loops. It does not reduce costs; instead, it risks unexpected billing spikes if response lengths are unconstrained by prompt design or application logic.

Manually apply the fix to the code and restart the Claude Code session.

Why it fails: Restarting the session after a manual fix is inefficient and disrupts the agent's internal state. Claude Code is designed to process error outputs and apply fixes iteratively within a single session, maintaining context and allowing it to learn from previous failures during the development process.

Disable the agent's ability to run commands to prevent further errors.

Why it fails: Disabling command execution prevents the agent from verifying its own fixes. Claude Code requires shell access to run tests and validate changes. Without this capability, the agent cannot confirm if its proposed solution resolves the underlying issue, rendering the debugging process incomplete and unreliable.

Run 'npm install' to ensure all dependencies are resolved before proceeding.

Why it fails: While dependency management is important, the error message indicates a 'TypeError' which is typically a logic issue rather than a missing package. Running install does not resolve code-level errors where a function call is targeting an undefined variable or module reference within the project code.

Check the total cost of API usage in the Anthropic dashboard.

Why it fails: Cost is a lagging indicator. By the time you notice an anomalous cost, the damage from an attack may already be done. Cost monitoring is essential for financial management, but it is not a technical tool for distinguishing between malicious traffic patterns and legitimate high-frequency user interactions in real-time.

Limit all users to one request per minute.

Why it fails: Setting a draconian rate limit negatively impacts the user experience and breaks legitimate applications that require high throughput. A one-size-fits-all approach is not a valid security strategy; it is a blunt instrument that hinders functionality and fails to address the nuances of modern, sophisticated traffic patterns.

Disable all external access to the API immediately.

Why it fails: Disabling access is a nuclear option that accomplishes the goal of stopping a DDoS attack, but it also stops all legitimate business. This is not a balanced or effective security practice for maintaining service availability. Effective security involves graceful mitigation, not total system shutdown in response to spikes.

As a single, complete JSON object in the first stream event.

Why it fails: In a streaming response, the goal is to provide data as it is generated. Waiting for the entire tool call to be ready before sending any data would defeat the purpose of streaming. Instead, the API sends the tool call incrementally, just like it does with standard text content blocks.

Via a separate WebSocket channel dedicated to tool execution.

Why it fails: Anthropic's API uses Server-Sent Events (SSE) for streaming, not WebSockets. All content, whether text or tool calls, is delivered over the same SSE connection. There is no separate channel for tools; they are just different types of content blocks within the same unified message stream provided by the API.

As a Base64 encoded string at the end of the message stream.

Why it fails: Tool arguments are sent as plain text fragments of a JSON string, not Base64 encoded. Encoding would add unnecessary overhead and complexity to the streaming process. The standard approach allows for immediate, character-by-character processing if needed, though most developers will wait for the complete JSON string before attempting to parse it.

The model's maximum token limit setting.

Why it fails: The max_tokens parameter caps the number of output tokens the model can generate, but it does not directly determine the actual token cost. The cost is based on the actual tokens used, not the limit. Setting a high max_tokens does not increase cost unless the model actually generates that many tokens.

The geographic region of the API endpoint.

Why it fails: Anthropic's pricing for the Claude API is not region-dependent; it is based on token usage regardless of the endpoint's location. Therefore, the geographic region does not directly contribute to token cost. This option is a distractor that might confuse developers familiar with other cloud services that have regional pricing.

The number of tokens in the model's response.

Why it fails: Output tokens are billed separately and do not contribute to input token cost. If the question asked about total token cost, output tokens would be a factor, but here we are focusing on input token cost. Therefore, this option is incorrect for the specified cost component.

Send the system prompt as a user message instead of a system prompt to avoid extra charges.

Why it fails: Anthropic charges the same rate for tokens regardless of whether they appear in the system prompt or user message. Moving the system prompt to a user message does not change the token count or cost. It may also break the intended instruction hierarchy. This is not a valid cost-reduction strategy and could harm response quality.

Increase the max_tokens parameter to allow the model to generate longer responses.

Why it fails: Increasing max_tokens raises the maximum possible output length, which can increase cost if the model generates more tokens. It does not reduce input token costs, which are the primary expense here. This action would likely increase overall spending and is not a cost-saving measure. The focus should be on reducing input tokens or caching repeated content.

Switch to a larger model with a higher token limit to process the entire history more efficiently.

Why it fails: Larger models generally have higher per-token costs and do not inherently reduce the number of tokens processed. Using a larger model would likely increase expenses, not decrease them. The goal is cost reduction, so moving to a more expensive model contradicts the objective. This option does not address the underlying cost drivers.

Add a system prompt instructing Claude not to store or remember any patient information, and send the notes through the public API.

Why it fails: A system prompt is an instruction to the model, not an enforceable data-handling control. It cannot prevent the notes from being transmitted to Anthropic or bound what the service does with them. Relying on model behavior for a compliance requirement leaves both the egress and retention conditions unmet.

Enable zero data retention on the Anthropic account so prompts and completions are not stored, and continue sending full notes over the public API.

Why it fails: Zero data retention addresses storage after processing, not the requirement that notes never leave the company network. The full note still travels to Anthropic's servers and is processed there. The residency requirement remains violated even though retention is disabled.

Switch to the Anthropic API through AWS PrivateLink so traffic stays on the private network, and keep sending full notes in the prompt.

Why it fails: PrivateLink protects data in transit from the application to the API endpoint, but the raw notes are still sent to Anthropic for processing. The requirement that notes never leave the company network is not met, because the model receives the full text. Transport privacy alone does not address data residency or retention.

Use the public Anthropic API with a zero-data-retention agreement and rely on TLS 1.3 in transit.

Why it fails: Zero data retention addresses Anthropic's storage of the data, but the payload still traverses the public internet to Anthropic-operated endpoints. The requirement states that PHI must not leave the controlled network boundary. Encryption in transit protects confidentiality on the wire but does not keep data inside the boundary, so this fails the first condition.

Self-host an open-weights model on-premises and route all summarization through it instead of Claude.

Why it fails: This satisfies the network boundary and retention conditions, but it no longer uses Claude, which the scenario specifies as the required model. Substituting a different model changes the capability, evaluation, and clinical quality assumptions the company built around Claude. The question asks for a deployment approach for Claude, so replacing the model is out of scope even though it is technically secure.

Use the Anthropic API with client-side field-level encryption of PHI fields before sending and decrypt only in the client.

Why it fails: If PHI is encrypted before transmission and Claude cannot decrypt it, the model cannot meaningfully summarize the content, defeating the use case. If keys are shared so Claude can decrypt, PHI effectively leaves the boundary. Either way, this does not satisfy both the boundary and retention requirements while preserving the summarization function.

Include the full text of all previously processed chunks in every new request.

Why it fails: Including all prior chunks recreates the original context-length problem and will exceed the context window well before the manual is finished. It also inflates cost quadratically as the pipeline progresses. The developer explicitly has a fixed token budget, so this approach is both technically infeasible and economically unacceptable for a 200-page document.

Ask the model to summarize each chunk twice and concatenate both outputs to increase coverage.

Why it fails: Summarizing twice and concatenating doubles output tokens and often produces redundant or conflicting statements, which worsens consistency rather than improving it. Duplication does not create cross-chunk awareness, so a term defined in chunk 3 still has no way to influence chunk 27. It also consumes the fixed token budget without adding grounding.

Lower the temperature to zero and rely on deterministic sampling to guarantee identical facts across chunks.

Why it fails: Temperature zero reduces randomness but cannot guarantee factual consistency across chunks, because each chunk is processed with different input text and no shared memory. Determinism affects sampling variance, not information flow between calls. Facts absent from a given chunk's input will still be absent from its summary regardless of temperature.

Lower the temperature to zero and add the sentence 'Respond only with JSON' to the end of the user message.

Why it fails: Temperature zero reduces sampling variance but does not eliminate structural drift, because the model still chooses whether to emit a preamble. A trailing instruction is also the weakest position for a formatting rule and can be outweighed by the earlier descriptive schema, so malformed outputs can persist.

Wrap the schema description in <json_schema> tags and instruct Claude to output only what matches the schema.

Why it fails: Delimiting the schema improves clarity, but the developer already described the schema precisely and still saw an 8 percent failure rate. Tagging alone does not constrain the first emitted token, so the model can still preface the object or fence it, leaving the parser broken.

Increase max_tokens so the model has room to finish the JSON object without truncation.

Why it fails: Truncation would produce an incomplete object, but the observed failures are complete outputs with a preamble or code fence. Raising max_tokens does nothing to stop the model from emitting 'Sure, here is the JSON:' or wrapping the result in backticks, so it does not address the reported defect.

Move the handbook to the end of the prompt and place the user question and instructions after it.

Why it fails: Placing the long document last pushes the actual question and instructions into a trailing position where attention is weaker, and the document still competes for attention across its length. This does not fix the positional degradation for late-document content and can harm instruction adherence. The scenario needs better use of the whole context, not a reordering that buries the query.

Place the long handbook in the middle of the prompt and surround it with the instructions and question at both ends.

Why it fails: Content in the middle of a long context tends to receive the weakest attention, so relocating the handbook there would likely worsen recall for the sections that already fail. Duplicating instructions at both ends adds tokens without restoring attention to mid-document policy text. This approach misunderstands where degradation concentrates and does not improve retrieval of late content.

Split the handbook into two equal prompts sent sequentially, asking Claude to remember the first half when answering the second.

Why it fails: The Messages API is stateless, so the model cannot 'remember' a prior call unless the full history is resent, which reintroduces the same long-context problem. Splitting also risks losing cross-references between sections. This does not address positional attention degradation and adds coordination complexity, so it is not the best fix for the stated symptom.

Deny the current edit and re-prompt Claude Code to only edit source files.

Why it fails: Re-prompting may steer this particular turn, but it provides no durable protection. In a later session or a differently worded request, Claude Code could again attempt to rewrite the generated stubs, and nothing in the environment would stop it. The scenario explicitly requires protection that persists beyond the current interaction, which a one-time denial cannot guarantee.

Set the editor to read-only mode for the `gen/` directory before starting the session.

Why it fails: Editor-level read-only flags govern the human's editing surface, not the tools Claude Code invokes. Claude Code writes through its own file-editing and shell capabilities, which bypass editor UI settings. Relying on the editor leaves the generated stubs fully writable by the agent, so the requirement would not actually be met.

Commit the generated stubs so Claude Code treats them as read-only.

Why it fails: Version control status does not make files read-only to Claude Code; a committed file is still editable and the change would simply appear as a new diff. This option reflects a misconception that tracking a file in Git prevents agent modification. The scenario needs an enforced guard, not an incidental artifact of repository history.

Instruct Claude Code in CLAUDE.md to never run dangerous commands.

Why it fails: Guidance in CLAUDE.md is contextual advice, not an enforced control. The model may still propose a dangerous command, especially under ambiguous instructions, and there is no hard block. While documenting intent is good practice, it does not by itself reduce the risk of arbitrary execution in a reliable, verifiable way, so it is not one of the two practices that actually constrain behavior.

Run the build inside a container with no network access and a read-only filesystem.

Why it fails: Container isolation is a valid defense-in-depth measure, but the scenario asks for practices that reduce risk while still allowing the needed commands to run as part of the build task. A read-only filesystem would often prevent the build from writing artifacts, breaking the task. Isolation changes the environment rather than governing which commands Claude Code may execute, so it does not directly address the stated concern within the workflow.

Enable `--dangerously-skip-permissions` to avoid prompts during the build.

Why it fails: Skipping permissions removes the confirmation step and allows Claude Code to run commands without review, which increases rather than reduces risk. The scenario is specifically about reducing the chance of arbitrary or destructive commands, so disabling safeguards is counterproductive. This flag is intended for isolated environments where risk is acceptable, not for a cautious build workflow.

Cache the results of expensive tool calls on the server side and return cached responses for identical requests.

Why it fails: Caching can improve performance, but it introduces complexity and potential staleness. For infrastructure management, data changes frequently, and cached responses could lead to incorrect actions. The scenario emphasizes efficiency and scalability, but caching is not a primary practice for scaling; statelessness and idempotency are more fundamental. Caching might be useful in specific cases but is not one of the two best practices here.

Use a single global lock to serialize all tool calls, preventing concurrent execution.

Why it fails: A global lock would severely limit scalability by forcing all requests to be processed one at a time. It would create a bottleneck and reduce throughput. For a scalable MCP server, concurrency should be managed carefully, but global locking is counterproductive. Statelessness and idempotency are better approaches.

Store session state in memory on each server instance to track client interactions.

Why it fails: Storing session state in memory ties clients to specific instances, which hinders horizontal scaling and can cause issues if an instance fails. For a scalable server, state should be externalized or avoided. Stateless design is preferred. This practice would make scaling more difficult, not easier.

The user is only responsible for providing the initial prompt.

Why it fails: Providing the initial prompt is only the start. The user must actively participate by reviewing proposals, handling interactive prompts, and verifying the results. A 'set it and forget it' approach is incompatible with the safety-first design of the tool and leads to poor project outcomes.

The user should avoid interfering to prevent conflicts.

Why it fails: Interference is not the right term for active oversight. The user must monitor the agent's work, intervene when necessary, and guide the process. Allowing the agent to proceed without human supervision increases the risk of subtle bugs and architectural errors that only a human can detect.

The user must perform all terminal commands manually.

Why it fails: The agent is designed to execute terminal commands autonomously. Requiring the user to run every command manually would remove the primary efficiency benefit of the tool. The user should observe and authorize, but not perform, the agent's primary autonomous tasks unless specific intervention is required.

The server must use a unique API key for every tool call it receives.

Why it fails: While authentication is important, requiring a unique API key for every individual tool call is not a standard requirement of MCP and would add significant management overhead. Typically, authentication is handled at the transport or session level when the host first connects to the server.

The server must rotate its connection ID every sixty seconds for privacy.

Why it fails: Frequent rotation of connection IDs is not a feature or requirement of MCP and would likely cause instability in the host-server communication. The protocol relies on stable sessions to manage the state of resources and tools. Privacy is better handled through proper data masking and secure transport layers.

All MCP tool results must be manually approved by a human operator.

Why it fails: While 'human-in-the-loop' is a valid pattern for high-risk actions, it is not a mandatory security requirement for all MCP implementations. Forcing manual approval for every result would destroy the speed and utility of most AI agents. Security should instead be built into the server's logic and permissions.

Deleting arbitrary user files without asking.

Why it fails: Deleting files without user authorization is a high-risk action that violates safety standards. The agent must always ask for explicit confirmation before removing files to prevent data loss, ensuring that the developer remains in control of their filesystem and can prevent accidental deletion of important assets.

Exfiltrating environment variables to remote servers.

Why it fails: Exfiltrating sensitive environment variables is a malicious action. Claude Code is designed to protect user privacy and does not send sensitive local environment data to external servers, prioritizing security and preventing potential credential leakage that could compromise the developer's infrastructure and system integrity.

text_update

Why it fails: While 'text_update' sounds descriptive of what is happening, it is not a valid event type in the Anthropic streaming protocol. The API uses a more structured approach with content blocks to allow for future flexibility, such as sending multiple types of data (text and tool results) in a single stream.

chunk_received

Why it fails: The term 'chunk_received' is a generic networking term and does not correspond to a specific event in the Anthropic API specification. Using incorrect event names in your event listener logic will cause the stream to appear broken or unresponsive, as the application will fail to catch the actual data packets.

stream_open

Why it fails: The connection to a stream is typically handled by the HTTP library, and while the stream opens, there is no specific SSE event named 'stream_open' sent by the Anthropic server. The first operational event developers should look for to begin processing the actual API data is the message_start event.

System prompts can be passed interchangeably as the first item in the messages array with a system role or via the top-level system parameter.

Why it fails: The messages array does not accept a system role; attempting to pass system messages inside the messages array results in an API validation error because the system parameter must be supplied as a distinct top-level request field.

Tool use definitions and tool choices are configured exclusively within the system prompt block as XML tags rather than through dedicated API parameters.

Why it fails: Tools are configured using structured JSON schemas in the dedicated top-level tools parameter rather than relying solely on raw XML tags inside the system prompt, ensuring robust programmatic parsing and validation.

Images provided in multimodal requests must be supplied as base64-encoded strings inside content blocks with explicitly defined media types.

Why it fails: While base64 encoding is supported for image content blocks along with media types, images can also be referenced via URL streams in specific environments or handled through distinct content types depending on the exact SDK version.

The tool is not licensed for use in CI/CD pipelines.

Why it fails: Licensing is not the primary barrier. The fundamental technical hurdle is the interactive architecture. While one could technically run it, the agent's need for human validation and natural language feedback makes it ill-suited for the automated, headless environments typical of CI/CD, where non-deterministic AI behavior is undesirable.

The tool requires an internet connection for every step.

Why it fails: While it does connect to the API, network connectivity is not the bottleneck. Many CLI tools connect to the internet in CI/CD. The core issue is the agentic nature of the tool, which is meant for human collaboration, not for autonomous code verification in a scripted pipeline environment.

It cannot execute shell commands reliably.

Why it fails: The agent is quite capable of executing shell commands. However, it executes them as part of a reasoning loop designed for a developer to follow. In a CI/CD environment, this extra reasoning layer adds overhead and potential for failure that standard scripts are specifically designed to avoid.