AI-300 Practice Test 3 — 30 Questions

Question 1

You are building a RAG application and notice that the model sometimes hallucinates information not present in the retrieved documents. Which evaluation metric should you prioritize to mitigate this?

Accepted Answer

Groundedness. Groundedness specifically assesses whether the generated response is derived from the retrieved documents.

Answer

Relevance

Answer

Performance

Answer

Fluency

Question 2

You want to evaluate how well your model adheres to specific brand guidelines. Which evaluation method is best suited for this?

Accepted Answer

Custom LLM-as-a-judge evaluation. Custom evaluation using a judge model (LLM-as-a-judge) configured with a rubric allows for checking specific style or brand compliance.

Answer

Token usage threshold alert

Answer

Default coherence metric

Answer

Automated regression testing of code

Question 3

You are optimizing the cost of your GenAI infrastructure. You have several agents running in Prompt Flow that are idle for large portions of the day. Which runtime configuration should be modified?

Accepted Answer

Set the 'automatic_shutdown' property on the Prompt Flow runtime compute instance.. Setting the 'idle_time_before_shutdown' or similar inactivity policies in the Prompt Flow runtime ensures compute resources are deallocated when not in use.

Answer

Change the VM SKU to a burstable instance type.

Answer

Implement a script to delete the runtime when the flow is finished.

Answer

Set the concurrency limit to zero.

Question 4

A developer is troubleshooting an agent that is failing to connect to an external tool. Which THREE of the following infrastructure settings should be verified?

Accepted Answer

The Azure AI Foundry 'Connections' configuration for the specific tool.. Connection failures often stem from incorrect credential management, network blocking, or misconfigured environment variables that define the external tool's access.

Answer

The Azure Resource Group location.

Answer

The model version of the LLM.

Question 5

Which TWO of the following strategies are commonly used to optimize for cost in Generative AI?

Accepted Answer

Selecting the smallest capable model for the task.. Caching and choosing smaller, task-appropriate models are the primary levers for cost management.

Answer

Increasing the context window size to 128k.

Answer

Enabling verbose logging.

Answer

Using the highest possible temperature.

Question 6

You are monitoring an Azure OpenAI deployment and need to identify if a model is outputting content that violates safety policies. Which Azure AI Content Safety feature should you enable to categorize harmful content?

Accepted Answer

Content Safety API text analysis. The Content Safety service provides classification categories such as Hate, Self-Harm, Sexual, and Violence to filter model outputs.

Answer

Azure Monitor Logs

Answer

Application Insights Profiler

Answer

Prompt Flow Guardrails

Question 7

You need to detect 'jailbreak' attempts in your RAG application. You are implementing a custom evaluation pipeline. Which technique is most effective for identifying adversarial inputs designed to bypass system instructions?

Accepted Answer

Adversarial input classification using a dedicated judge model. Adversarial evaluation using a 'jailbreak' detection model or prompt-based evaluation is the standard approach to identify bypass attempts.

Answer

Token usage tracking

Answer

Latency threshold monitoring

Answer

Semantic similarity scoring

Question 8

You are reviewing the 'Run History' in Azure Machine Learning. You want to compare the training time of two different experiments. Which UI feature should you use?

Accepted Answer

The 'Experiments' tab, then 'Compare' button.. The 'Charts' or 'Metrics' comparison view in the Azure ML Studio allows you to visualize and compare metrics across multiple runs.

Answer

The 'Models' registry list.

Answer

The 'Endpoints' dashboard.

Answer

The 'Notebooks' file browser.

Question 9

You are evaluating an LLM application using Prompt Flow. You want to measure the 'Groundedness' of the model response relative to the retrieved context. Which evaluator should you configure?

Accepted Answer

Groundedness evaluator. The Groundedness evaluator in Prompt Flow checks if the response is supported by the context provided.

Answer

Relevance evaluator

Answer

Coherence evaluator

Answer

Fluency evaluator

Question 10

You have an automated model training pipeline that is failing intermittently due to compute availability. What should you configure to ensure the pipeline is more resilient?

Accepted Answer

Configure 'retry' settings for the step.. You should configure 'retry' settings in the 'PipelineStep' definition to handle transient compute errors.

Answer

Increase the 'max_nodes' of the cluster.

Answer

Set the 'allow_reuse' parameter to True.

Answer

Use 'LowPriority' compute.

Question 11

Which THREE metrics are critical for monitoring a production Generative AI system?

Accepted Answer

Latency (Time-to-first-token).. Latency, token usage, and error rates are the standard pillars of LLM monitoring.

Answer

The hardware temperature of the Azure server.

Answer

Model training loss.

Question 12

Your team wants to perform 'Red Teaming' on your application. Which activity describes this process correctly?

Accepted Answer

Controlled adversarial testing to find safety and security gaps. Red Teaming involves adversarial testing to find edge cases, safety flaws, and security vulnerabilities.

Answer

Unit testing code snippets

Answer

Automated performance testing for latency

Answer

A/B testing two different model versions

Question 13

Which of the following is a key component of an observability strategy for Generative AI applications?

Accepted Answer

Monitoring both technical and quality metrics. Observability requires tracking both technical metrics (latency, errors) and quality metrics (groundedness, safety).

Answer

Hard-coding all model responses

Answer

Using only manual review processes

Answer

Disabling logging to save costs

Question 14

When designing a content safety policy, which THREE categories are explicitly supported by the Azure AI Content Safety API?

Accepted Answer

Sexual. Azure AI Content Safety supports Hate, Sexual, Violence, and Self-Harm categories.

Answer

Political bias

Answer

Code syntax errors

Question 15

A model is exhibiting data drift. You have created a drift monitor. What is the next step to automate the retraining?

Accepted Answer

Configure an Event Grid subscription to trigger a pipeline.. After creating a monitor, you set up an 'Event Grid' subscription to notify an Azure Function or Logic App to trigger the pipeline.

Answer

Update the dataset version.

Answer

Manually restart the compute cluster.

Answer

Create a 'RetrainingTrigger' in the model registry.

Question 16

You are defining an Azure Machine Learning environment for a training job. The environment requires a specific set of Python libraries. What is the best practice for defining these dependencies?

Accepted Answer

Define dependencies in a 'conda.yaml' file.. Using a 'conda.yaml' file is the best practice for managing reproducible Python environments in Azure ML.

Answer

Hardcode pip install commands in the training script.

Answer

Install libraries via a startup script in the compute cluster.

Answer

Pre-install them on the virtual machine image.

Question 17

Which Azure Machine Learning resource provides a pre-configured environment for development?

Accepted Answer

Compute Instance.. A 'Compute Instance' is a managed, cloud-based development environment that comes pre-installed with the Azure ML SDK, Jupyter, and other tools.

Answer

Inference Cluster.

Answer

Compute Cluster.

Answer

Managed Endpoint.

Question 18

You are preparing a model for deployment. Which THREE items should you include in the model package?

Accepted Answer

Model weights (pickle/onnx).. Model file, environment requirements, and inference code.

Answer

Training dataset.

Answer

Pipeline definition.

Question 19

You are implementing a retraining trigger for a demand forecasting model. You want to trigger a pipeline execution only when the drift metric for the 'Price' feature exceeds a predefined threshold. Which service should you integrate with Azure Machine Learning?

Accepted Answer

Azure Event Grid and Logic Apps.. Azure Machine Learning Data Drift Monitors can be configured to emit events to Azure Event Grid, which can then trigger an Azure Logic App or Azure Function to start the pipeline.

Answer

Azure Batch schedules.

Answer

Azure Data Factory triggers.

Answer

Azure Monitor alerts on logs.

Question 20

You are configuring a CI/CD pipeline for model deployment. Which THREE actions must you perform to ensure model traceability?

Accepted Answer

Register the model in the Azure ML Model Registry with versioning. Model registry registration, metadata tagging, and linking to the source code commit are standard MLOps practices for traceability.

Answer

Use a shared local folder for model artifacts

Answer

Delete all training logs after deployment