AI-300 Practice Test 9 — 10 Questions

Question 1

You are setting up an evaluation suite for your LLM. Which THREE metrics are commonly provided by the 'Built-in' evaluators in Azure AI Prompt Flow?

Accepted Answer

Coherence. Azure AI Prompt Flow's built-in evaluators include Coherence (A), which measures how logically and naturally the generated text flows; Groundedness (B), which checks whether the response is supported by the provided source/context; and Relevance (C), which assesses how well the answer addresses the user's question. These three are standard quality metrics exposed by the built-in evaluation flow for assessing LLM outputs. Deployment cost (D) is an operational/cost metric, not a built-in evaluator metric, and training duration (E) relates to model training rather than response evaluation, so neither belongs here.

Answer

Deployment cost

Answer

Training duration

Question 2

You are building a RAG application. Where should you store the indexed documents for optimal retrieval performance?

Accepted Answer

Azure AI Search. Azure AI Search is the native indexer and retriever for AI Foundry RAG applications.

Answer

Azure Blob Storage

Answer

Azure Cache for Redis

Answer

Azure SQL Database

Question 3

You are troubleshooting a failed agent deployment. Which TWO areas should you inspect first?

Accepted Answer

The endpoint availability status.. Option B (the endpoint availability status) is correct because a failed agent deployment often stems from the target endpoint being unreachable, unhealthy, or misconfigured, so checking its availability status quickly reveals connectivity or service-health problems that would block deployment. Option C (the 'Deployment Logs' in the Azure AI Foundry portal) is correct because these logs capture the specific error messages, stack traces, and failure reasons generated during the deployment attempt, making them the most direct source for diagnosing why the agent failed. Options A, D, and E do not belong: the Azure status page only reports broad, platform-wide outages rather than issues specific to your agent deployment, user profile settings are unrelated to deployment execution, and the billing usage report reflects cost consumption rather than deployment health or errors.

Answer

The Azure status page.

Answer

The user profile settings.

Answer

The billing usage report.

Question 4

You need to monitor the data drift of a model deployed in Azure ML. What is the first step you must take?

Accepted Answer

Create a Data Drift Monitor. In Azure Machine Learning, data drift detection is a first-class feature that requires you to create a Data Drift Monitor on a deployed model endpoint or dataset. The monitor defines the baseline dataset, target dataset, features to track, and the drift metric (e.g., population stability index) and schedule. Only after the monitor exists can alerts and metrics be surfaced.

Answer

Configure Application Insights

Answer

Run a manual evaluation script

Answer

Enable Azure Monitor logs

Question 5

You are designing the infrastructure for a multi-agent system. Which THREE of the following are essential components to ensure agent orchestration scalability?

Accepted Answer

Horizontal Pod Autoscaling (HPA) for inference services.. Option B is correct because Horizontal Pod Autoscaling (HPA) dynamically adjusts the number of pod replicas for inference services based on metrics like CPU utilization or custom metrics, which is essential for scaling agent workloads elastically as demand fluctuates. Option C is correct because a message queue such as Azure Service Bus decouples agent tasks, enabling asynchronous communication, load leveling, and reliable task distribution across multiple agents, which is fundamental to orchestration scalability. Option D is correct because a distributed cache like Azure Cache for Redis provides low-latency shared state management, allowing agents to coordinate, share session context, and avoid bottlenecks from repeated database reads in a scaled-out system. Option A is not essential for orchestration scalability, since Azure Data Lake Storage is a storage solution for large-scale data and model artifacts, not a runtime scaling or coordination component. Option E is not essential either, as Azure Bastion provides secure RDP/SSH access to virtual machines for management purposes and does not contribute to agent orchestration scalability.

Answer

Azure Data Lake Storage for model weights.

Answer

Azure Bastion for secure management.

Question 6

Which THREE actions are essential when managing a 'Golden Dataset' for LLM evaluation?

Accepted Answer

Versioning the dataset to track changes. Option A (Versioning the dataset to track changes) is correct because a golden dataset must be immutable and auditable, so semantic versioning or hashes (e.g., Git, DVC, or dataset version tags) let you reproduce evaluation results and correlate score changes with specific dataset revisions. Option D (Updating the dataset as model capabilities change) is correct because as models improve, static benchmarks saturate; curating harder or newly relevant cases (e.g., new task types, edge cases, or regressions) keeps the evaluation discriminative and aligned with current deployment goals. Option E (Curating high-quality prompt-response pairs) is correct because the golden dataset's reference answers must be accurate, unambiguous, and representative, typically validated by domain experts, since noisy or biased ground truth invalidates every metric computed against it. Option B is not essential because encryption with public keys only is an asymmetric-encryption detail, not a dataset-management requirement; confidentiality is handled by access controls and, if needed, symmetric encryption or KMS. Option C is not essential because deleting logs after each test run destroys the audit trail and reproducibility evidence needed to compare runs and debug regressions.

Answer

Encrypting the data with public keys only

Answer

Deleting logs after each test run

Question 7

You want to automate the evaluation of your LLM application using a 'Golden Dataset'. What is the primary purpose of this dataset in an MLOps pipeline?

Accepted Answer

To serve as a baseline for measuring performance improvements. A Golden Dataset serves as the ground truth to compare model outputs against during automated evaluation runs.

Answer

To increase the model training speed

Answer

To reduce the number of tokens used

Answer

To generate new prompts for users

Question 8

You want to evaluate your prompt engineering changes quantitatively. Which method is most reliable for comparing two prompt versions?

Accepted Answer

Running the prompts against a benchmark evaluation dataset.. Running both prompt versions against a benchmark evaluation dataset produces quantitative, reproducible metrics (accuracy, F1, BLEU, groundedness, etc.) that allow direct statistical comparison. This removes subjectivity and isolates the effect of the prompt change, which is the only reliable way to determine whether one prompt outperforms another.

Answer

Monitoring the total request count.

Answer

Asking developers for subjective feedback.

Answer

Checking if the model is running on GPT-4.

Question 9

You are designing a quality assurance gate for your model. If a model output has a 'Violence' score of 0.8 according to Azure AI Content Safety, what is the best practice to handle it?

Accepted Answer

Filter out the output and return a generic refusal. A high score indicates a potential violation, and the output should be blocked or sanitized before reaching the end user.

Answer

Allow the output but log it as an error

Answer

Retrain the model immediately

Answer

Ignore the score if the latency is high

Question 10

Your team uses a custom Docker image for training. You need to update the image in the Azure Container Registry (ACR). What is the recommended workflow to ensure Azure ML picks up the latest version?

Accepted Answer

Update the image tag and refresh the Environment. Azure ML references container images through Environment objects, which pin a specific image tag (or digest) from ACR. To pick up a new image, you must push the updated image with a new tag (or overwrite the tag and force a refresh), then update the Environment definition to reference that tag and re-register it. This ensures training jobs pull the correct, latest image rather than a cached older version.

Answer

Modify the storage account permissions

Answer

Restart the compute cluster

Answer

Delete and recreate the workspace