CCAR-F Context and Reliability Practice Test 3 — 15 Questions

Question 1

An enterprise application uses Claude to summarize complex legal documents. The system occasionally hallucinates specific clause numbers when the context window contains multiple similar contracts. Which strategy most effectively improves factual reliability for specific data extraction?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Use a retrieval-augmented generation (RAG) architecture to inject only relevant text segments into the context window.. Grounding the model with high-precision retrieval augmented generation (RAG) is the gold standard for reducing hallucinations in document processing. By forcing the model to operate strictly within the provided retrieved context, the probability of hallucinating external data decreases significantly. This approach is essential for architectural reliability, as it bounds the model's creative potential, ensuring that legal summaries remain tethered to the provided document artifacts rather than probabilistic memory.

Answer

Increase the system prompt's temperature setting to 1.0 to ensure more creative clause identification.

Answer

Implement a few-shot prompting strategy using examples of common legal clause structures.

Answer

Enable verbose chain-of-thought prompting to force the model to explain its reasoning process.

Question 2

An application uses Claude 3.5 Sonnet to summarize technical logs. Users report that the model occasionally ignores specific error codes when logs exceed 50,000 tokens. Which strategy best ensures consistent reliability for long-context tasks?

Accepted Answer

Pre-process logs into smaller, overlapping chunks and use a multi-step aggregation approach for summarization.. Long-context retrieval requires managing token density and model attention span. By implementing a sliding window or chunking strategy with overlapping segments, you ensure the model maintains context across boundaries. This prevents the 'lost in the middle' phenomenon where models prioritize information at the beginning or end of a prompt. Reliable context management is critical for technical applications where missing a single error code can lead to incorrect diagnostic conclusions.

Answer

Increase the temperature setting to 1.2 to encourage more creative scanning of the input text.

Answer

Use a system prompt to explicitly instruct the model to pay attention to all tokens regardless of their position.

Answer

Switch to a smaller model version to ensure faster processing of the large log files.

Question 3

Refer to the exhibit. The developer notices that Claude occasionally truncates the sentiment analysis report. What is the most likely cause based on the JSON configuration?

Accepted Answer

The 'max_tokens' limit is insufficient for the length of the required sentiment analysis report.. The 'max_tokens' parameter in the JSON configuration limits the total number of tokens the model can generate. If the model's analysis exceeds this value, the response will be cut off, resulting in an incomplete output. This is a common reliability issue when the expected output length is variable or misunderstood. Setting a sufficiently high 'max_tokens' ensures the model has enough headroom to complete its reasoning and output process.

Answer

The temperature is set to 0, which prevents the model from generating longer, more descriptive responses.

Answer

The model is receiving an error because the messages array is malformed.

Answer

The model version 'claude-3-5-sonnet-20240620' does not support sentiment analysis tasks.

Question 4

Which property of a well-structured prompt contributes most significantly to the reliability of a model's output in a zero-shot scenario?

Accepted Answer

Providing clear, domain-specific instructions and output format constraints.. Clear, role-based, and specific instructions define the 'context' in which the model operates. Providing explicit constraints (such as 'no markdown', 'format as JSON', or 'be concise') reduces ambiguity. Reliability is highest when the model's output space is narrow and clearly defined. Without these constraints, models may interpret requests differently, leading to unpredictable formatting or content that violates the requirements of the downstream consuming application.

Answer

Using a highly complex and verbose prompt to cover all possible edge cases.

Answer

Writing the prompt in a conversational tone to make the model feel more comfortable.

Answer

Omitting examples to allow the model to show its full range of reasoning.

Question 5

A legal-tech team runs a Claude-powered contract review service in production. Compliance auditors require a reproducible record of every model response for any given input, but the team also wants to adopt newer model improvements without redeploying code. They want to keep prompt text unchanged and avoid maintaining provider-specific SDK logic. Which approach best satisfies both the reproducibility requirement and the adoption of newer model improvements?

Accepted Answer

Route every request through a versioned internal gateway that records the exact model identifier and full request payload, and let the gateway map a logical alias such as claude-3-5-sonnet-latest to a concrete dated snapshot at request time.. Reproducibility requires capturing the exact model identifier and request payload, while the ability to adopt improvements without code changes requires an indirection layer that resolves a logical alias to a dated snapshot. A versioned gateway provides both: it records what actually ran and lets operations rotate the alias by configuration. Pinning in code, multi-model voting, or relying on prompt determinism each fails at least one of the stated constraints.

Answer

Send the same prompt to all available Claude model versions on every request and store the responses, then let a downstream voting service select the majority answer for the audit record.

Answer

Pin the model to a dated snapshot identifier such as claude-3-5-sonnet-20240620 for all production traffic and treat any model upgrade as a code change reviewed by compliance.

Answer

Freeze the system prompt and temperature in a configuration file so that identical inputs always yield identical outputs, and rely on that determinism instead of recording the model identifier.

Question 6

A customer support platform routes conversations to Claude 3.5 Sonnet via the Messages API. The system prompt includes 40,000 tokens of product documentation, and each turn appends the full prior transcript. After roughly 30 exchanges, agents report that Claude starts contradicting earlier troubleshooting steps it gave in the same conversation and drifts from the documented procedures. Latency and cost have also grown steadily. Which architectural change best addresses the reliability degradation while controlling cost?

Accepted Answer

Keep only the last few turns verbatim, and maintain a rolling structured summary of resolved issues and confirmed product facts that is re-injected each turn alongside the static documentation.. Long sessions accumulate raw transcript tokens until earlier, authoritative content competes with stale or contradictory turns. Bounding the history and replacing it with a maintained, re-injected summary preserves the decisions that matter while keeping the static documentation prominent. This controls both the reliability drift and the runaway token cost, which prompt caching or model swaps alone cannot fix.

Answer

Increase the temperature to 1.0 so the model explores more of the documentation and naturally self-corrects when it detects contradictions.

Answer

Enable prompt caching on the static system prompt and raise the max_tokens parameter so the model has more room to restate earlier steps.

Answer

Switch the model to Claude 3 Haiku for the long sessions, since a faster model will complete each turn before the context window fills and drift begins.

Question 7

A customer support platform uses Claude to answer policy questions by retrieving relevant help-center articles and inserting them into the prompt. Agents report that for questions whose answer spans two separate articles, Claude often cites only one article and omits the other. Logs show both articles were retrieved and included. Which change to the context assembly is most likely to resolve this?

Accepted Answer

Wrap each retrieved article in clearly labeled XML tags with source identifiers, and instruct Claude to synthesize across all provided sources before answering.. The failure is a context-structuring problem: two retrieved articles are present but Claude anchors on one. Delimiting each article with labeled tags and explicitly instructing synthesis across all sources gives the model clear document boundaries and a reason to reconcile them. Output limits, temperature, and text merging do not change how Claude parses or prioritizes the supplied context.

Answer

Increase the max_tokens parameter so Claude has room to mention both articles in its reply.

Answer

Concatenate both articles into a single unbroken text block so Claude treats them as one source.

Answer

Lower the temperature setting so Claude produces more deterministic and complete citations.

Question 8

You are architecting a customer-support assistant that maintains a persistent knowledge base across sessions. After several weeks, users report that the assistant confidently cites policy details that were never in the source documents. Which architectural change most directly addresses this reliability failure?

Accepted Answer

Require the assistant to answer only from retrieved passages and to return a sentinel value when retrieval yields no supporting passage.. The reported failure is unsupported factual claims, so the fix must tie answers to verifiable source content. Constraining the assistant to retrieved passages and defining a no-answer sentinel gives the system an explicit grounding contract and a detectable failure state. Determinism settings, longer outputs, and accuracy exhortations change style or length but never establish whether a cited policy actually exists.

Answer

Increase the max_tokens parameter so the assistant has more room to explain its reasoning before answering.

Answer

Add a system prompt instruction telling the assistant to never make mistakes and to always be accurate.

Answer

Lower the temperature parameter to zero so the assistant produces more deterministic responses.

Question 9

A regulated insurance workflow requires Claude to extract coverage limits from policy documents and return them as JSON. During testing, the model occasionally emits prose commentary before the JSON, breaking the downstream parser. The team cannot change the parser and must guarantee the response begins with a valid JSON object. Which technique most reliably enforces that?

Accepted Answer

Prefill the assistant turn with an opening brace so Claude continues from that point.. Prefilling the assistant turn with an opening brace constrains generation so the response starts inside a JSON object, satisfying a parser that cannot tolerate leading prose. Instructions and retries are probabilistic, temperature changes add variance, and shortening inputs does not affect output formatting. For a hard formatting guarantee, the prefill is the reliable control.

Answer

Shorten the policy documents before extraction so the model has less text to summarize.

Answer

Add the phrase 'respond only in JSON' to the end of the system prompt and retry on parse failure.

Answer

Raise the temperature to encourage the model to explore more structured output formats.

Question 10

A development team is building an internal document Q&A tool. They want Claude to answer strictly from uploaded policy PDFs and to decline questions the documents do not cover. Which prompt structuring technique best supports this requirement?

Accepted Answer

Provide the documents inside XML tags and instruct Claude to answer only using content within those tags, declining otherwise.. Strict document-scoped answering needs two things: unambiguous boundaries around the source material and an explicit instruction about what to do when the sources are silent. Delimiting the PDFs in XML tags gives Claude a referable container, while the decline instruction supplies the fallback behavior. Sampling tweaks, example counts, and knowledge-first ordering all leave the sourcing boundary undefined.

Answer

Ask Claude to answer from its general knowledge first, then compare with the documents afterward.

Answer

Increase the number of few-shot examples to fifty so the model learns the answering pattern.

Answer

Set the top_p parameter to 0.1 to make the model choose only high-probability tokens.

Question 11

A travel assistant maintains a long conversation with a user over several days. The system prompt defines a strict booking policy, but by the fortieth turn Claude begins offering refunds that violate that policy. The team wants to keep the full history for continuity while preventing policy erosion. Which architecture best addresses this?

Accepted Answer

Re-assert the booking policy as a system-level instruction periodically or before policy-sensitive actions.. Policy drift over long conversations is countered by keeping the governing instruction salient near the decision point. Periodically re-asserting the booking policy, or injecting it before policy-sensitive actions, preserves full history while restoring the constraint's influence. Truncation, larger output limits, and user-driven reminders do not address the loss of instruction salience across many turns.

Answer

Delete older turns once the conversation exceeds thirty messages so only recent context remains.

Answer

Ask the user to restate the booking policy at the start of each session.

Answer

Increase the model's max_tokens so it has more room to recall the policy when answering.

Question 12

You are building a customer support assistant using the Claude API. The assistant must answer questions based solely on a provided knowledge base of 20 product manuals. You want to ensure that when the answer is not in the knowledge base, Claude explicitly states 'I don't know' rather than fabricating an answer. Which technique is most reliable for achieving this?

Accepted Answer

Include a system prompt instructing Claude to answer only from the provided context and to respond with 'I don't know' if the answer is not present.. The most reliable way to ensure Claude answers only from provided context and admits ignorance is to explicitly instruct it via a system prompt. This leverages the model's instruction-following ability to constrain its responses. Other methods like lowering temperature or fine-tuning do not directly enforce this behavior and may still result in hallucinations when the answer is missing.

Answer

Use a larger context window to include all 20 manuals in every request.

Answer

Fine-tune Claude on a dataset of questions and correct answers from the manuals.

Answer

Set the temperature parameter to 0 to reduce randomness in responses.

Question 13

An architect is designing a multi-turn contract-review assistant. During long sessions, earlier redline decisions get contradicted in later turns because the model loses track of prior conclusions. The team cannot shorten sessions. Which approach best preserves decision consistency across the full session?

Accepted Answer

Maintain a structured running summary of decisions and inject it into each new turn alongside the most recent exchange.. The contradiction pattern stems from decision state being implicit in a long transcript. A structured running summary makes prior redline conclusions an explicit, compact artifact that is re-injected every turn, so consistency no longer depends on the model surfacing distant text. Window size, higher randomness, and user restatement all fail to externalize and enforce that state.

Answer

Raise the temperature so the model explores alternative interpretations and avoids repeating itself.

Answer

Rely on the model's extended context window to retain all prior turns without additional structure.

Answer

Ask the user to restate all prior decisions at the start of every turn.

Question 14

You are architecting a customer-support assistant that uses Claude with a 200K-token context window. Each session accumulates roughly 40K tokens of chat history, and you also inject a 30K-token product manual on every turn. Latency has become unacceptable because the full payload is re-sent each call. Which change best reduces latency while preserving the assistant's knowledge of earlier turns?

Accepted Answer

Enable prompt caching on the stable prefix (manual plus prior turns) so repeated tokens are served from cache instead of being reprocessed on each call.. Prompt caching is the architectural lever for repeated large prefixes: the manual and accumulated history are stable across turns, so caching their processed state avoids re-encoding tens of thousands of tokens on every request. Caching preserves the full context the assistant needs, unlike truncation, and it targets input processing cost, unlike max_tokens or batching, which address output length or throughput rather than interactive latency.

Answer

Batch all user questions into a single nightly asynchronous request using the Message Batches API.

Answer

Raise the max_tokens parameter so Claude can emit a longer response and finish the answer in fewer round trips.

Answer

Switch to a smaller model with a shorter context window and truncate the manual to the first 8K tokens.

Question 15

You are designing a customer support assistant that must always respond in a calm, professional tone. During testing, you notice that after several turns of heated user complaints, Claude's replies become abrupt and less empathetic. You need the most reliable way to prevent this tonal drift throughout a long conversation. What should you do?

Accepted Answer

Use a system prompt that explicitly defines the persona and tone, and keep it unchanged across all turns.. Tonal drift in long conversations is best mitigated by placing the desired persona and tone in the system prompt, which Claude treats as a persistent instruction set. Unlike user-turn reminders or hidden assistant turns, the system prompt remains constant and authoritative, helping the model maintain consistent behavior even when user messages are emotionally charged. Lowering temperature affects randomness, not persona adherence.

Answer

Reduce the model's temperature to 0 so that responses become deterministic and tone cannot vary.

Answer

Add a reminder of the desired tone at the beginning of each user message.

Answer

After each user message, insert a hidden assistant turn that models the desired tone.