AI0-001 AI Implementation and Operations Practice Test 9 — 15 Questions

Question 1

A hospital has deployed an AI triage assistant that summarizes patient intake notes and suggests an acuity level for the emergency department. Clinicians report that the assistant sometimes produces confident but unsupported acuity suggestions. The operations team must add safeguards appropriate for a high-stakes clinical deployment. (Choose two.)

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Display the source note spans that the model used for each acuity suggestion so clinicians can verify the evidence before accepting it.. The reported failure is confident but unsupported suggestions, so the safeguards must both expose the evidence behind each suggestion and keep a clinician as the accountable decision-maker. Attribution to source note spans makes the claim checkable, and mandatory confirmation or override prevents an unverified acuity level from entering the patient record.

Answer

Retrain the assistant on the hospital's historical triage outcomes and redeploy it as the primary acuity assigner.

Answer

Lower the model's temperature and top-p sampling values so its generated summaries are more deterministic.

Answer

Increase the model's context window so it can ingest the patient's entire longitudinal chart in one request.

Question 2

A bank operates a credit-scoring model in production. Auditors require the team to reproduce the exact score a specific applicant received six months ago, including the model version, the feature values, and the code path used. Which capability must the team have in place to satisfy this requirement?

Accepted Answer

Full lineage logging that captures the model artifact version, the raw input record, the engineered feature values, and the inference request metadata for every prediction.. Auditability of an individual prediction demands that the model version, the post-engineering feature values, and the request context all be captured at the moment of inference. Only comprehensive lineage logging binds those elements to a specific decision so it can be replayed later. Registries, aggregate dashboards, and shadow deployments each cover a different concern and none retains the record-level detail an auditor needs to reconstruct one score.

Answer

A model registry that stores every trained model version with its performance metrics and promotion status.

Answer

Periodic shadow deployment of the current model against the previous version to compare scoring behavior.

Answer

A dashboard that tracks aggregate approval rates and score distributions over time to demonstrate stable model behavior.

Question 3

An e-commerce company operates an AI recommendation service. After a marketing campaign, the operations team notices that inference costs have tripled while request volume has only doubled. They need to reduce cost per inference without degrading recommendation quality. Which two actions should the team take? (Choose two.)

Accepted Answer

Apply model quantization or a distilled smaller model for the recommendation ranking stage, validating quality against offline metrics.. Cost per inference falls when each execution does more useful work or requires fewer resources. Dynamic batching amortizes execution overhead across concurrent requests, and quantization or distillation reduces the compute needed per prediction. Adding replicas, disabling caching, or upgrading instance size raises capacity or work without improving efficiency, so they do not meet the goal.

Answer

Disable caching of recommendation results so every request is computed fresh from the model.

Answer

Move the recommendation service to a larger GPU instance type with more memory.

Answer

Increase the number of replicas behind the load balancer so each instance handles fewer requests.

Question 4

An AI operations team is designing a rollback strategy for a fraud-detection model served behind a feature flag. A new model version shows degraded precision after release. The team wants to restore the previous behavior within minutes without redeploying code or losing the ability to collect data on the new version. Which approach best meets these requirements?

Accepted Answer

Route traffic back to the prior model version by toggling the feature flag, while continuing shadow-mode evaluation of the new version. A feature flag allows instant switching between model versions without touching application code, and shadow mode lets the suspect version keep receiving copied traffic for evaluation while customers are served by the proven model. Redeployment is too slow, threshold tuning does not restore validated behavior, and retraining cannot act within minutes while labels accumulate.

Answer

Lower the model's decision threshold so fewer transactions are flagged as fraudulent

Answer

Keep the new model live and immediately retrain it on the most recent labeled fraud cases

Answer

Rebuild the container image with the prior model artifact and redeploy through the CI/CD pipeline

Question 5

A retail bank deploys an AI model that approves or declines small-business loan applications. Regulators require the bank to explain any adverse decision to the applicant in plain language. The model is a gradient-boosted ensemble over dozens of features, and the bank's data scientists cannot easily describe why a specific applicant was declined. Which approach best satisfies the regulatory requirement?

Accepted Answer

Apply a local explanation technique such as SHAP values to identify the features that most influenced the individual applicant's score and translate them into plain-language reasons.. Regulatory adverse-action requirements demand case-specific reasons for each declined applicant. Local explanation techniques such as SHAP values attribute an individual prediction to its contributing features, which can then be rendered in plain language. Global feature importance, aggregate audit results, and swapping in a simpler model either describe the population rather than the individual or disrupt the production system without addressing the disclosure need.

Answer

Publish the model's global feature importance rankings alongside each decision notice so applicants see which factors matter most overall.

Answer

Replace the gradient-boosted ensemble with a single decision tree so the decision path itself can be shown to the applicant.

Answer

Provide applicants with the model's overall accuracy and fairness audit results to demonstrate that the system is trustworthy.

Question 6

A logistics company runs an AI route-optimization service that calls a hosted large language model to interpret free-text driver notes and convert them into structured stop instructions. The service works in testing, but in production many requests fail with rate-limit and timeout errors during the morning dispatch window. The team wants the service to survive these failures without losing driver instructions. Which approach should the team implement?

Accepted Answer

Add retry with exponential backoff and jitter, plus an idempotency key so repeated attempts do not create duplicate stop instructions.. The failures are transient throttling and timeout conditions, which are exactly what retry with exponential backoff and jitter is designed to absorb. Adding an idempotency key makes those retries safe by guaranteeing that a repeated request produces one set of stop instructions rather than duplicates.

Answer

Switch the service to send all of the morning's driver notes in a single batched request to reduce the total call count.

Answer

Raise the client-side request timeout to ten minutes so slow responses are allowed to complete.

Answer

Cache the model's responses and serve cached structured instructions whenever a similar driver note is submitted.

Question 7

An operations team runs a computer-vision model that flags manufacturing defects on an assembly line. Auditors require evidence that any single prediction can be reconstructed and explained months later. The team already logs model version, input image hash, and prediction score. Which additional logging practice best satisfies the audit requirement?

Accepted Answer

Log the preprocessed feature vector, the model version identifier, and the explanation artifacts such as saliency or SHAP values for each inference.. Auditability of an individual prediction requires the exact inputs the model saw, the version of the model that scored them, and the explanation artifacts generated at inference time. Capturing the preprocessed feature vector, model version identifier, and attributions such as SHAP or saliency values lets auditors replay and justify any single decision. Aggregates, raw frames, or reviewer names cannot reconstruct the original reasoning.

Answer

Log the prediction score together with the operator who reviewed the flagged unit.

Answer

Log the raw camera frames indefinitely and rely on the current model to regenerate explanations on demand.

Answer

Log only the aggregate daily defect rate and the model's overall precision.

Question 8

A retailer's recommendation service runs on a managed inference endpoint. During a flash sale, request volume triples and the endpoint's response time exceeds the acceptable threshold. The operations team must reduce latency quickly without retraining the model. Which action should they take first?

Accepted Answer

Increase the number of provisioned inference instances behind the endpoint so requests are distributed across more compute.. The described symptom is a capacity shortfall during a traffic spike, so the fastest effective remedy is horizontal scaling of the serving tier. Adding inference instances distributes load and cuts queueing delay without touching the model. Retraining, trimming logging, and extending timeouts either take too long or merely conceal the latency, leaving the underlying bottleneck in place.

Answer

Raise the client-side request timeout so slow responses are tolerated instead of failing.

Answer

Lower the endpoint's logging verbosity and disable request tracing to reduce per-request overhead.

Answer

Retrain the model with a smaller architecture so each inference completes faster.

Question 9

A media company uses a generative AI assistant to draft customer responses. After an update to the underlying foundation model, agents report that responses sometimes include fabricated policy details. The operations team must detect this regression quickly and prevent fabricated content from reaching customers. Which combination of controls is most appropriate?

Accepted Answer

Add automated groundedness and hallucination checks against the approved policy knowledge base, and require agent approval before any response is sent.. Detecting fabricated policy details requires verifying generated text against the authoritative source, which groundedness or hallucination checks provide. Preventing delivery requires a gate such as agent approval before sending. Rollback, prompt tweaks, and retrospective sampling either do not detect unsupported claims or act too late to protect customers.

Answer

Roll back to the previous foundation model version and monitor customer satisfaction scores for improvement.

Answer

Enable request logging and review a sample of conversations weekly for quality issues.

Answer

Lower the model's maximum output token limit and add a system prompt instructing it to be accurate.

Question 10

A hospital's AI triage assistant produces recommendations that clinicians frequently override. An operations review finds the model was trained on data from a different patient population than the one currently served. Which action most directly addresses the root cause of the low acceptance rate?

Accepted Answer

Retrain or fine-tune the model using data representative of the current patient population and revalidate its performance. The review identified a population mismatch between training data and the patients now being served, so the corrective action must close that gap. Retraining or fine-tuning on representative local data and revalidating performance targets the cause directly. Confidence displays, override documentation, and higher refresh rates are peripheral measures that do not change what the model has learned.

Answer

Increase the model's inference frequency so recommendations refresh more often during a shift

Answer

Mandate that clinicians document a reason each time they override a recommendation

Answer

Add a confidence score display so clinicians can see how certain the model is about each recommendation

Question 11

A media company serves personalized article recommendations through a model that is retrained weekly. After a major news event, engagement metrics show that recommendations became stale within hours because the model had not yet seen the new topic. The engineering team wants recommendations to reflect breaking topics within minutes without retraining the whole model. Which approach should the team implement?

Accepted Answer

Add a real-time retrieval layer that surfaces recently published articles by semantic similarity to the user's current session, and blend those candidates with the model's ranked output.. The gap is that breaking articles never enter the candidate set until the next weekly training cycle, so no amount of reweighting or diversity tuning can surface them. A real-time retrieval layer that matches fresh articles to the live session and blends them into the ranking closes that gap within minutes while preserving existing personalization.

Answer

Shorten the retraining cadence from weekly to hourly so the model learns the new topic sooner.

Answer

Increase the weight of the recency feature in the existing ranking model and redeploy the updated weights.

Answer

Lower the diversity threshold so the recommender is allowed to show a wider spread of topics to each user.

Question 12

An insurance company operates an AI claims-triage model that flags suspicious claims for human review. After six months in production, the operations team observes that the model's precision has fallen steadily while recall has stayed roughly constant, and the volume of false-positive flags has grown. The data science team suspects the input data pipeline is the cause rather than the model weights. Which TWO operational checks should the team perform first to diagnose the problem? (Choose two.)

Accepted Answer

Audit the feature engineering and ETL pipeline for silent failures such as nulls, default values, or unit changes that alter feature semantics.. Stable recall with falling precision points to inputs that no longer match training conditions rather than to the model weights. Comparing production feature distributions against the training baseline detects drift or schema changes, and auditing the ETL and feature engineering stages uncovers silent failures such as nulls, defaults, or unit changes. Retraining, threshold changes, or more reviewers treat symptoms without identifying the pipeline defect.

Answer

Increase the model's decision threshold so fewer claims are flagged as suspicious.

Answer

Retrain the model immediately on the last six months of production data to restore precision.

Answer

Expand the human review team so more flagged claims can be examined manually while the model remains unchanged.

Question 13

A support team deploys a retrieval-augmented generation assistant that answers questions from internal policy documents. Users report that the assistant confidently invents policy details that do not appear in any document. The team wants to reduce these fabricated answers without retraining the language model. Which change is MOST effective?

Accepted Answer

Improve retrieval relevance and instruct the model to answer only from retrieved passages, returning a fallback when evidence is absent.. Hallucination in a retrieval-augmented assistant is typically a grounding failure. Improving retrieval relevance delivers the correct source passages, and a prompt that restricts answers to retrieved evidence with a defined fallback prevents the model from filling gaps from its own parameters. Sampling temperature, retrieval volume, and prompt length do not address the underlying grounding problem.

Answer

Increase the number of retrieved documents returned to the prompt without changing ranking quality.

Answer

Shorten the system prompt so the model has more room for its own reasoning.

Answer

Raise the model's temperature setting to make responses more varied.

Question 14

A hospital's clinical decision support model was validated at 94 percent accuracy on a held-out set. After go-live, clinicians report that the model's suggestions are frequently irrelevant for elderly patients, even though overall accuracy in the monitoring dashboard has barely moved. Which monitoring practice would have surfaced this problem?

Accepted Answer

Tracking performance metrics disaggregated by patient demographic and clinical subgroups, with alerts when any subgroup falls below its own threshold.. The clue is that aggregate accuracy held steady while a specific patient group received poor suggestions, which is the classic signature of a subgroup masked by population averaging. Disaggregated performance monitoring with per-segment thresholds is the practice that reveals it. Aggregate accuracy alerts, latency metrics, and input drift detection each observe a different dimension and cannot confirm that elderly patients specifically are being served badly.

Answer

Measuring inference latency and throughput to confirm the model responds within clinical workflow time limits.

Answer

Monitoring overall model accuracy and alerting when it drops more than two percentage points from the validation baseline.

Answer

Comparing the live distribution of input features against the training distribution to detect covariate shift.

Question 15

A hospital deploys an AI model that summarizes clinical notes for physicians. Before go-live, the AI team must verify that the model does not reproduce patient identifiers in its summaries when they are not clinically necessary. Which activity is the MOST appropriate for this verification?

Accepted Answer

Run a red-team evaluation with prompts designed to elicit protected health information and measure leakage rates.. Verifying that a summarizer suppresses unnecessary identifiers requires exercising the model with prompts that attempt to elicit protected health information and measuring how often leakage occurs. Red-teaming produces that empirical evidence against a threshold. Perplexity, training-data scrubbing, and model cards are useful complements but none demonstrates the deployed model's output behavior under adversarial or realistic clinical prompts.

Answer

Verify that the training dataset was de-identified using an automated named-entity recognition scrubber.

Answer

Confirm the model card documents the intended use and known limitations of the summarization system.

Answer

Compare the model's perplexity on the clinical corpus against its perplexity on public text.