AI-300 Practice Test 3 — 25 Questions

Question 1

Which TWO of the following are recommended methods for identifying 'hallucinations' in a RAG system?

Accepted Answer

Using an LLM-as-a-judge to compare against ground truth. Comparing output to source context (groundedness) and using a reference-based evaluation are standard methods.

Answer

Checking for grammatical errors

Answer

Measuring model response speed

Answer

Counting total prompt tokens

Question 2

You are configuring a connection to a vector store. What is the best way to handle the secret credential?

Accepted Answer

Store in Azure Key Vault and reference in the connection.. Key Vault is the centralized store for all application secrets in Azure, which should be linked to the project connection.

Answer

Hardcode the key in the flow configuration.

Answer

Pass it as a parameter in the code.

Answer

Store it in an environment variable.

Question 3

A team is scaling a GenAI application that utilizes multiple models via model-as-a-service. You notice that inference requests are being throttled. What is the most effective infrastructure-level adjustment to handle the increased load?

Accepted Answer

Purchase and deploy Provisioned Throughput Units (PTUs) for the specific model deployment.. Provisioned throughput units (PTUs) allow for dedicated capacity, which is the standard way to prevent throttling for high-scale GenAI workloads in Azure.

Answer

Change the model version to a lower-parameter variant.

Answer

Move the deployment to a different subscription within the same Azure region.

Answer

Increase the number of instances in the associated Azure Kubernetes Service (AKS) cluster.

Question 4

You want to automate the evaluation of your LLM application using a 'Golden Dataset'. What is the primary purpose of this dataset in an MLOps pipeline?

Accepted Answer

To serve as a baseline for measuring performance improvements. A Golden Dataset serves as the ground truth to compare model outputs against during automated evaluation runs.

Answer

To increase the model training speed

Answer

To reduce the number of tokens used

Answer

To generate new prompts for users

Question 5

Which THREE features does Azure AI Foundry provide to help developers?

Accepted Answer

Evaluation.. Model Catalog, Prompt Flow, and Evaluation are key developer-facing features in Azure AI Foundry.

Answer

Azure Active Directory creation.

Answer

Direct physical hardware access.

Question 6

When using Prompt Flow for GenAI, where should you store your evaluation results to visualize them over time?

Accepted Answer

Azure Machine Learning workspace. Prompt Flow logs evaluation runs to the Azure Machine Learning workspace, where they can be viewed in the Runs tab.

Answer

GitHub repository

Answer

Azure Blob Storage

Answer

Local file system

Question 7

When configuring a private endpoint for your Azure AI resource, which TWO of the following must be set up?

Accepted Answer

A virtual network with a dedicated subnet.. A private endpoint requires a target subnet and a private DNS zone for resolution.

Answer

A load balancer.

Answer

A firewall rule allowing all traffic.

Answer

A public IP address.

Question 8

Your team is deploying a large language model in Azure AI Foundry. You need to ensure that the inference endpoint remains highly available across different regions. Which configuration should you implement?

Accepted Answer

Enable global load balancing using Azure Front Door to route requests to multiple regional endpoint deployments.. Azure AI Foundry supports multi-region endpoints via Azure Front Door or traffic manager integration to distribute requests across regional inference endpoints for high availability.

Answer

Configure a Load Balancer at the VNet level for the model endpoint.

Answer

Set the model deployment to 'High Availability' mode in the Azure AI Foundry portal.

Answer

Replicate the underlying storage account for the model weights across regions.

Question 9

You are deploying a model that requires a high-memory compute for inference. You are using a 'Managed Online Endpoint'. Where do you specify the instance type for this deployment?

Accepted Answer

In the 'Deployment' YAML file under 'instance_type'.. The instance type is defined in the 'Deployment' configuration object, typically in the 'instance_type' field.

Answer

In the 'Endpoint' YAML file.

Answer

In the 'Environment' definition.

Answer

In the 'Workspace' settings.

Question 10

You are setting up a CI/CD pipeline for GenAI models. You need to evaluate the model's performance on a benchmark dataset before it is promoted to the production endpoint. Which Azure AI Foundry capability should be integrated into your pipeline?

Accepted Answer

Prompt Flow evaluation runs.. Evaluation runs in Prompt Flow allow for automated testing of model quality against datasets, which is a required step for a robust GenAI CI/CD pipeline.

Answer

Model catalog benchmarking.

Answer

Azure AI Content Safety scanning.

Answer

Azure Machine Learning model drift monitor.

Question 11

You are building a RAG application and notice that the model sometimes hallucinates information not present in the retrieved documents. Which evaluation metric should you prioritize to mitigate this?

Accepted Answer

Groundedness. Groundedness specifically assesses whether the generated response is derived from the retrieved documents.

Answer

Relevance

Answer

Performance

Answer

Fluency

Question 12

You want to evaluate how well your model adheres to specific brand guidelines. Which evaluation method is best suited for this?

Accepted Answer

Custom LLM-as-a-judge evaluation. Custom evaluation using a judge model (LLM-as-a-judge) configured with a rubric allows for checking specific style or brand compliance.

Answer

Token usage threshold alert

Answer

Default coherence metric

Answer

Automated regression testing of code

Question 13

You are optimizing the cost of your GenAI infrastructure. You have several agents running in Prompt Flow that are idle for large portions of the day. Which runtime configuration should be modified?

Accepted Answer

Set the 'automatic_shutdown' property on the Prompt Flow runtime compute instance.. Setting the 'idle_time_before_shutdown' or similar inactivity policies in the Prompt Flow runtime ensures compute resources are deallocated when not in use.

Answer

Change the VM SKU to a burstable instance type.

Answer

Implement a script to delete the runtime when the flow is finished.

Answer

Set the concurrency limit to zero.

Question 14

A developer is troubleshooting an agent that is failing to connect to an external tool. Which THREE of the following infrastructure settings should be verified?

Accepted Answer

The Azure AI Foundry 'Connections' configuration for the specific tool.. Connection failures often stem from incorrect credential management, network blocking, or misconfigured environment variables that define the external tool's access.

Answer

The Azure Resource Group location.

Answer

The model version of the LLM.

Question 15

Which TWO of the following strategies are commonly used to optimize for cost in Generative AI?

Accepted Answer

Selecting the smallest capable model for the task.. Caching and choosing smaller, task-appropriate models are the primary levers for cost management.

Answer

Increasing the context window size to 128k.

Answer

Enabling verbose logging.

Answer

Using the highest possible temperature.

Question 16

You are monitoring an Azure OpenAI deployment and need to identify if a model is outputting content that violates safety policies. Which Azure AI Content Safety feature should you enable to categorize harmful content?

Accepted Answer

Content Safety API text analysis. The Content Safety service provides classification categories such as Hate, Self-Harm, Sexual, and Violence to filter model outputs.

Answer

Azure Monitor Logs

Answer

Application Insights Profiler

Answer

Prompt Flow Guardrails

Question 17

You need to detect 'jailbreak' attempts in your RAG application. You are implementing a custom evaluation pipeline. Which technique is most effective for identifying adversarial inputs designed to bypass system instructions?

Accepted Answer

Adversarial input classification using a dedicated judge model. Adversarial evaluation using a 'jailbreak' detection model or prompt-based evaluation is the standard approach to identify bypass attempts.

Answer

Token usage tracking

Answer

Latency threshold monitoring

Answer

Semantic similarity scoring

Question 18

You are reviewing the 'Run History' in Azure Machine Learning. You want to compare the training time of two different experiments. Which UI feature should you use?

Accepted Answer

The 'Experiments' tab, then 'Compare' button.. The 'Charts' or 'Metrics' comparison view in the Azure ML Studio allows you to visualize and compare metrics across multiple runs.

Answer

The 'Models' registry list.

Answer

The 'Endpoints' dashboard.

Answer

The 'Notebooks' file browser.

Question 19

You are evaluating an LLM application using Prompt Flow. You want to measure the 'Groundedness' of the model response relative to the retrieved context. Which evaluator should you configure?

Accepted Answer

Groundedness evaluator. The Groundedness evaluator in Prompt Flow checks if the response is supported by the context provided.

Answer

Relevance evaluator

Answer

Coherence evaluator

Answer

Fluency evaluator

Question 20

You have an automated model training pipeline that is failing intermittently due to compute availability. What should you configure to ensure the pipeline is more resilient?

Accepted Answer

Configure 'retry' settings for the step.. You should configure 'retry' settings in the 'PipelineStep' definition to handle transient compute errors.

Answer

Increase the 'max_nodes' of the cluster.

Answer

Set the 'allow_reuse' parameter to True.

Answer

Use 'LowPriority' compute.