Microsoft Azure AI Fundamentals AI-900 (AI-900) — Questions 226–300

Frequency penalty reduces repetition by penalizing tokens that have already appeared, which does not increase randomness or creativity; it only discourages repeating the same words or phrases.

Top_p (nucleus sampling) controls the cumulative probability threshold for token selection, not the randomness of the output. Increasing top_p does not directly increase randomness; it limits the pool of tokens to those with high probability, which can actually reduce diversity.

Presence penalty reduces the likelihood of repeating any token that has appeared in the text so far, which encourages novelty but does not directly increase randomness. The question asks for more randomness, which is controlled by temperature.

If the question were: 'The model keeps repeating the same phrases across multiple headlines. Which parameter should be increased to reduce this repetition?' then frequency penalty would be correct.

A question asks: 'Which parameter should be adjusted to ensure the model only considers tokens that make up the top 90% of probability mass, thereby filtering out low-probability tokens?' In that scenario, top_p is the correct parameter to set to 0.9.

A question asks: 'The marketing team finds that generated headlines reuse the same words too often. Which parameter should they increase to penalize token repetition?' In that case, presence penalty would be correct.

Candidates may confuse frequency penalty with randomness because both affect output diversity, but frequency penalty specifically targets repetition rather than overall creativity.

Candidates may confuse top_p with temperature because both influence output diversity. They might think that adjusting the probability threshold (top_p) increases randomness, but it actually controls the size of the candidate token set, not the distribution's shape.

Candidates may confuse 'penalty' with 'randomness' or think that penalizing presence increases diversity, which they equate with creativity, but the parameter for randomness is temperature.

Setting temperature to 0.0 makes output more deterministic but does not filter offensive language; it only reduces randomness.

The frequency_penalty parameter reduces repetition of token sequences but does not filter offensive or inappropriate content. It cannot block specific categories of language like hate speech or profanity.

Configuring max_tokens limits the length of generated text but does not filter offensive content; it only truncates output after a token count.

If the question asked how to make the model produce more focused, less creative outputs (e.g., for factual Q&A), setting temperature to 0.0 would be correct.

A question asks: 'The model is generating repetitive ad copy. Which parameter should be adjusted to reduce repetition?' In that scenario, frequency_penalty is the correct answer.

When a question asks how to prevent excessively long responses that exceed API cost or latency limits, setting max_tokens to an appropriate value would be correct.

Candidates may think lowering temperature reduces undesirable outputs, confusing randomness control with content safety filtering.

Candidates may confuse content filtering with other model parameters, thinking that penalizing frequency can somehow reduce offensive language, or they may not fully understand the distinct roles of safety filters versus generation controls.

Candidates may mistakenly believe that limiting output length can prevent offensive content, or they confuse token limits with content moderation controls.

Fine-tuning adjusts model behavior on custom data but does not enforce content safety filters; it can even amplify biases if the dataset contains problematic content. Azure OpenAI's content filtering system is specifically designed to block offensive, harmful, or violent outputs regardless of the model's training.

Increasing the token limit per response controls the maximum length of generated text, but does not prevent offensive or harmful content. Content filtering is required to enforce brand safety guidelines.

Prompt engineering can guide model outputs but does not provide a systematic, configurable filter to block offensive language, harmful stereotypes, or violent themes as required by brand guidelines.

A company needs the model to generate marketing copy in a specific brand voice (e.g., formal, humorous) and has a large dataset of approved examples. Fine-tuning would adapt the model to that style, whereas content filtering alone cannot achieve stylistic alignment.

A question asks: 'Which configuration should be adjusted to allow the model to generate longer marketing copy for a detailed product description?' In that scenario, increasing the token limit per response would be correct.

When the question asks how to improve the relevance or style of generated content without changing the underlying model, such as 'A company wants to ensure marketing copy consistently uses a friendly tone without retraining the model. Which technique should they use?'

Candidates may think fine-tuning can 'teach' the model to avoid certain topics, but they overlook that fine-tuning does not guarantee safety compliance and that Azure provides a dedicated content filtering service for that purpose.

Candidates may mistakenly think that limiting output length can reduce harmful content, or they confuse token limits with content moderation controls.

Candidates may overestimate the power of prompt engineering to enforce safety constraints, confusing it with the built-in content filtering that Azure OpenAI provides for responsible AI.

Dense captioning generates descriptive captions for regions of an image, but it does not provide structured object detection with bounding boxes and counts per category, which is required for counting items per product category.

Azure Machine Learning with a pre-trained YOLO model is not a managed Azure Computer Vision service; it requires custom model training and deployment, whereas Custom Vision provides a simpler, integrated solution for custom object detection without managing infrastructure.

A company wants to automatically generate natural language descriptions of scenes in images, such as 'a shelf with soft drinks and chips', without needing to count or classify specific object categories. They have labeled images with region descriptions.

A question where a company needs to build a custom object detection model but has specific requirements for using a pre-trained model (e.g., YOLO) due to edge deployment constraints, and they have the expertise to manage the ML lifecycle on Azure Machine Learning.

Candidates may think dense captioning can identify and describe objects in shelves, but they overlook that it lacks the precise localization and counting capabilities needed for inventory management.

Candidates may know YOLO is a popular object detection model and think Azure Machine Learning is the only way to use it, overlooking that Custom Vision internally uses similar deep learning models but offers a more streamlined service for this use case.

Model Interpretability explains how features influence predictions but does not evaluate bias or fairness across demographic groups.

Error Analysis focuses on identifying regions of high error in the model's predictions, not on evaluating bias against demographic groups. The question specifically asks for bias assessment, which is the role of Model Fairness Assessment.

Data Balance Analysis is used to detect imbalances in the training data (e.g., underrepresentation of a group), but the question asks about identifying bias in the model's predictions, which requires fairness assessment of the model's outputs.

A question asking which component helps understand why a model made a specific prediction for a loan applicant, such as identifying the key factors leading to a high-risk score.

A question asking: 'A data scientist wants to identify which subsets of data have the highest prediction errors to improve model accuracy. Which component of the Responsible AI dashboard should they use?' would make Error Analysis the correct answer.

A question asking: 'A data scientist wants to check if the training dataset has sufficient representation of all demographic groups before training a model. Which component should they use?'

Candidates may confuse interpretability with fairness, assuming that understanding model decisions inherently reveals bias.

Candidates may confuse 'error' with 'bias', thinking that analyzing errors will reveal unfair treatment of groups, but error analysis does not directly measure fairness or demographic disparities.

Candidates may confuse data imbalance (a cause of bias) with model fairness (the effect), assuming that analyzing data balance directly evaluates model bias.

Anomaly Detector identifies unusual patterns in time-series data, not language identification or sentiment analysis. The pipeline requires language detection and sentiment analysis, which Anomaly Detector does not provide.

Azure AI Speech is for speech-to-text and text-to-speech, not for language identification or translation. The pipeline requires language detection and translation, not speech processing.

Azure AI Vision is for image analysis, not for language identification, translation, or sentiment analysis. The question requires processing text messages, not images.

A question asking for a solution to detect anomalies in customer support ticket volumes over time, using time-series data from multiple regions, where Azure Anomaly Detector is needed alongside Azure AI Translator to preprocess multilingual text data.

If the question involved processing audio messages (e.g., customer voicemails) that need to be transcribed to text before language identification and translation, then Azure AI Speech (for transcription) combined with Azure AI Language (for sentiment) would be correct.

A question asks: 'A company wants to extract text from scanned customer feedback forms and then analyze the sentiment of the extracted text.' In that case, Azure AI Vision (OCR) followed by Azure AI Language (Text Analytics) would be correct.

Candidates may confuse 'Anomaly Detector' with a service that detects unusual content in text, or they might think it can handle language-related tasks due to the word 'detector'.

Candidates may confuse 'language' broadly with 'speech', assuming speech services handle all language tasks, or overlook that the input is already text (feedback messages), not audio.

Candidates may confuse Azure AI Vision's OCR capability as a general text processing tool, or they might think 'vision' includes language understanding, leading them to select this option without reading the question carefully.

Key phrase extraction identifies important terms but does not categorize them as product names, and language detection only identifies the language, not sentiment. The question requires extracting product names (NER) and sentiment analysis, not language detection.

Conversational language understanding (CLU) is designed for intent and entity extraction from conversational utterances, not for analyzing static support emails. Translation is irrelevant because the question does not mention multilingual needs.

Text summarization condenses content but does not extract product names, and PII detection identifies personal data like names or addresses, not product names or sentiment. Neither feature meets the requirements of extracting product names or analyzing sentiment.

A scenario where the team needs to identify the language of incoming emails and extract key terms (like product names) without needing sentiment analysis. For example, routing emails to language-specific queues and extracting main topics.

A customer support chatbot needs to understand user intents (e.g., 'reset password') and extract entities from chat messages, then translate responses for a multilingual audience. In that scenario, CLU and translation would be the correct pair.

A question requiring removal of sensitive information from documents before sharing, such as 'A legal team needs to redact personal data from contracts and generate concise summaries for review.' Then text summarization and PII detection would be correct.

Candidates may confuse key phrase extraction with named entity recognition, thinking it can extract product names, and overlook that language detection is irrelevant to sentiment analysis.

Candidates may think CLU can extract product names (entities) and translation might be needed for multilingual emails, but the question specifies no custom training and focuses on prebuilt features for email analysis.

Candidates may think 'summarization' can extract key details like product names, and 'PII detection' seems related to analyzing email content, but they confuse data extraction with content analysis.

Key phrase extraction identifies general important terms but does not specifically categorize entities like organizations or monetary values, which is required for extracting structured information from contracts.

Sentiment analysis detects positive/negative/neutral sentiment in text, not extraction of organizations or monetary values. The question requires identifying specific entities, not overall tone.

Text summarization generates concise summaries of documents, but does not extract specific entities like organization names or monetary values. The requirement is for entity extraction, not summarization.

Key phrase extraction would be correct if the question asked for extracting the main topics or important terms from documents without needing to categorize them into predefined types like person, organization, or money.

A question asking which Azure AI Language feature to use for determining customer satisfaction from product reviews or social media posts would make sentiment analysis correct.

A question asking which Azure AI Language feature to use for generating a brief overview of a long legal document, without needing to extract specific data points, would make text summarization correct.

Candidates may confuse key phrase extraction with named entity recognition because both involve extracting important words, but they overlook the need for specific entity categorization in this scenario.

Candidates may confuse 'extracting information' broadly with sentiment analysis, or think sentiment includes identifying key elements like organizations and money.

Candidates may confuse 'extracting key information' with 'summarizing', thinking that a summary would include the needed entities, but summarization produces narrative text, not structured entity lists.

OCR extracts text from images, but the task requires identifying objects (bed, desk, chair), not reading text.

Image Analysis (prebuilt) can identify objects and scenes but does not specifically detect and locate multiple instances of predefined objects like a bed, desk, and chair in a single image. Object Detection is required for that.

Handwriting OCR is designed to recognize handwritten text, not to detect objects like beds, desks, or chairs in images.

A question asking to extract printed text from images of documents or signs, using a prebuilt Azure AI service without custom training.

A question asking to 'extract descriptive tags or captions for an image' or 'determine if an image contains adult content' would make Image Analysis correct, as it provides general image categorization and content moderation without custom training.

A question asking to extract handwritten notes from scanned forms or images, where the goal is to digitize handwritten content without custom training.

Candidates may confuse 'analyzing photos' with reading text, or think OCR is the only prebuilt vision service they know.

Candidates may confuse 'Image Analysis' with 'Object Detection' because both deal with visual content, but Image Analysis is a broader service that includes object detection only as a sub-feature, not as its primary function for locating multiple specific objects.

Candidates may confuse OCR with general image analysis, assuming 'reading' text extends to identifying objects, or they may think 'handwriting' covers all visual recognition.

The situation describes a lack of explainability, not bias or discrimination. The loan rejection may be fair, but the inability to explain it violates transparency, not fairness.

The scenario describes a lack of explainability, not a failure of reliability or safety. The model works correctly but cannot provide understandable reasons.

The situation describes a lack of explainability, not a breach of data protection or unauthorized access. Privacy & Security concerns data handling, not model interpretability.

Fairness would be correct if the question described the AI system systematically denying loans to a specific demographic group (e.g., based on race or gender) without justification, indicating bias in the model's decisions.

A medical diagnosis AI system incorrectly classifies a patient's condition due to biased training data, leading to harmful treatment recommendations. This violates Reliability & Safety because the system is not robust and poses safety risks.

A question where an AI system exposes customer financial data to unauthorized third parties or fails to encrypt sensitive information would make Privacy & Security the correct answer.

Candidates may associate loan approval decisions with fairness concerns, assuming any rejection must involve bias, and overlook that the core issue here is the lack of explanation, not discrimination.

Candidates may confuse 'inability to explain' with 'unreliable or unsafe,' assuming that a complex model that cannot be explained must be unreliable.

Candidates may conflate 'explanation of decision' with 'exposure of personal data,' mistakenly thinking that providing reasons violates privacy, or they may confuse transparency with privacy.

The question focuses on detection failure in heavy rain, which is a reliability and safety issue, not fairness. Fairness addresses bias against groups, not performance degradation under environmental conditions.

The question focuses on detection failures in heavy rain, which directly impacts system reliability and safety, not privacy or security. Privacy/security concerns data protection and unauthorized access, not operational performance under adverse conditions.

An AI system for loan approvals consistently denies loans to applicants from a specific ethnic group despite equivalent financial profiles. This violates the fairness principle.

A scenario where an AI system exposes sensitive user data (e.g., facial recognition data) due to insufficient encryption or access controls would violate Privacy and security. For example, a healthcare AI that stores patient records without proper safeguards.

Candidates may confuse 'unfair outcomes' (unequal performance across conditions) with the technical definition of fairness in AI, which is about demographic parity or equal treatment across protected groups.

Candidates may confuse safety-critical failures with security vulnerabilities, or broadly associate any AI risk with privacy/security without analyzing the specific principle violated.

Temperature controls randomness of token selection, not repetition. Increasing temperature makes output more random but does not directly penalize repeated phrases.

Top-p controls the cumulative probability threshold for token selection, influencing diversity of word choices, but it does not directly penalize repetition of specific phrases or words. Frequency penalty is the parameter designed to reduce repetition by decreasing the likelihood of tokens that have already appeared.

Max tokens controls the length of the generated text, not the repetition of phrases. Increasing max tokens may allow more text but does not penalize repeated tokens.

A question asks which parameter to increase to make the model generate more creative and less deterministic responses, e.g., 'The model outputs are too predictable and safe. Which parameter should be increased?'

A question where the model generates text that is too random or incoherent, and the goal is to make the output more focused and deterministic by narrowing the set of likely tokens. For example: 'The generated text is too creative and goes off-topic. Which parameter should be decreased to make the output more focused?'

A question asks: 'The marketing team wants to ensure the generated copy does not exceed 150 words. Which parameter should they adjust?' In that case, max tokens would be the correct answer to limit output length.

Candidates may confuse 'repetition' with 'lack of creativity' and assume that increasing randomness (temperature) will reduce repetition, but temperature does not specifically target repeated tokens.

Candidates may confuse Top-p with frequency penalty because both affect output diversity, but Top-p controls the pool of candidate tokens based on probability mass, not repetition frequency.

Candidates might think that limiting the output length (max tokens) would reduce repetition by cutting off the generation early, but repetition can occur within any length, and max tokens does not address the underlying cause.

Decreasing temperature reduces randomness, which can lower creativity and make text more deterministic, but it does not specifically target repetitive phrases or loops. The question asks to reduce repetition without changing overall creativity, so temperature adjustment is not appropriate.

Increasing presence_penalty penalizes tokens based on whether they have appeared at all in the text, which reduces topic repetition but does not specifically target repetitive phrases or loops; frequency_penalty is designed for that.

Decreasing top_p reduces the set of tokens considered for sampling, which can make output less diverse but does not specifically penalize repetition; frequency_penalty directly reduces repetition by lowering the probability of tokens already generated.

In a scenario where the model generates overly random or nonsensical text and the goal is to make output more focused and predictable while preserving coherence, decreasing temperature would be correct. For example, when generating code or factual answers where creativity is not needed.

A scenario where the model keeps introducing new topics or entities that are irrelevant to the prompt, and you want to discourage the model from mentioning any topic more than once, would make presence_penalty the correct choice.

A scenario where the model's output is too random or incoherent, and the goal is to make it more focused and deterministic without completely eliminating creativity. For example, 'The generated text is too diverse and often goes off-topic; which parameter should be decreased to make it more focused?'

Candidates may confuse temperature with frequency penalty, thinking that lowering randomness will also reduce repetition, but temperature affects overall variability, not specifically penalizing repeated tokens.

Candidates may confuse 'presence' with 'frequency' or think that penalizing any repetition (presence) is equivalent to penalizing frequent repetition (frequency), but presence_penalty only cares about whether a token appears at all, not how often.

Candidates may confuse top_p (nucleus sampling) with frequency_penalty, thinking that limiting the token pool will also reduce repetition, but top_p controls diversity, not repetition frequency.

Increasing top_p (nucleus sampling) also increases randomness, but it primarily controls the cumulative probability threshold for token selection, not the overall creativity. For maximizing creativity and variety, temperature is the more direct parameter.

Increasing frequency_penalty reduces repetition by penalizing tokens that have already appeared, which can increase variety but does not directly control creativity or randomness. The question asks for the primary parameter to increase creativity and variety, which is temperature, as it directly scales the probability distribution for more random outputs.

Decreasing max_tokens limits the length of generated text, which reduces the amount of content available for creativity and variety, making stories shorter and less detailed.

A question asks: 'Which parameter should be adjusted to ensure the generated text avoids repetitive phrases while maintaining high coherence?' In that case, increasing frequency_penalty would be correct, not top_p.

A scenario where the generated text is too repetitive, with phrases or words appearing too often, and the goal is to reduce repetition without significantly increasing randomness. For example, a chatbot that keeps using the same greetings or a story generator that repeats character names excessively would benefit from increasing frequency_penalty.

A question asks: 'To reduce the cost and latency of API calls while ensuring responses are concise, which parameter should be decreased?' In that context, decreasing max_tokens limits output length, lowering token usage and cost.

Candidates may confuse top_p with temperature because both control randomness, and they might think adjusting the probability distribution is the best way to increase variety.

Candidates may confuse frequency_penalty with temperature, thinking that penalizing frequent tokens will force the model to be more creative by avoiding common patterns, but they overlook that temperature is the direct control for randomness and creativity.

Candidates may think that limiting output length forces the model to be more creative within a shorter space, or they confuse max_tokens with controlling output diversity.

The question focuses on screen reader compatibility for visual impairments, which directly relates to ensuring the assistant is usable by people with disabilities—this is the core of inclusiveness, not fairness. Fairness addresses bias and equitable treatment across groups, not accessibility features.

The question focuses on screen reader compatibility for visual impairments, which directly relates to inclusiveness (ensuring accessibility for all users). Reliability & Safety concerns system dependability and risk mitigation, not accessibility features.

Screen reader compatibility directly addresses accessibility for users with disabilities, which is the core of inclusiveness, not privacy & security. Privacy & security concerns data protection and system integrity, not assistive technology integration.

A company develops an AI hiring tool that inadvertently discriminates against female candidates. To address this, the team retrains the model with balanced data and tests for disparate impact. Which Microsoft responsible AI principle is most directly addressed?

A question asking which principle ensures an AI system performs consistently under varying conditions and avoids harmful failures, such as a self-driving car's braking system being tested for reliability in adverse weather.

A question about implementing data encryption, access controls, or anonymization techniques to protect customer data in the AI assistant would make Privacy & Security the correct answer.

Candidates may confuse 'fairness' with 'inclusiveness' because both involve equitable access, but fairness specifically targets bias and discrimination, while inclusiveness focuses on designing for diverse human abilities and needs.

Candidates may confuse 'reliability' with the general robustness of the assistant, thinking screen reader compatibility ensures the system works reliably for all users, but the principle specifically addresses accessibility, not system dependability.

Candidates may confuse inclusiveness with privacy because both involve user protection, or they might think screen readers relate to data security due to general awareness of compliance requirements.

The reporting and human review mechanism directly addresses accountability by ensuring the organization takes responsibility for AI outputs, not fairness, which focuses on avoiding bias against groups.

The reporting and human review mechanism directly addresses accountability by ensuring humans are responsible for AI outputs, not reliability and safety, which focuses on system robustness and error handling.

Transparency involves making AI systems understandable and disclosing their limitations, but the reporting and human review mechanism specifically ensures that the organization takes responsibility for the system's outputs, which is the core of accountability.

Fairness would be correct if the question described the AI chatbot providing different quality responses based on patient demographics (e.g., race, gender), and the mechanism ensured equitable treatment across groups.

This option would be correct if the question described a scenario where the chatbot was designed to fail gracefully, e.g., by detecting out-of-scope queries and providing a safe fallback response, or by undergoing rigorous testing to minimize harmful outputs.

A question asks: 'A hospital chatbot displays a disclaimer stating it is not a substitute for professional medical advice and explains how its answers are generated. Which principle does this directly implement?' The answer would be Transparency, as it involves clear communication about the system's nature and limitations.

Candidates may confuse 'accountability' with 'fairness' because both involve oversight, but fairness specifically targets bias and equitable outcomes, not the broader responsibility for AI behavior.

Candidates may confuse the human review process with ensuring system reliability, as both involve oversight, but reliability is about the AI's performance, while accountability is about human responsibility for outcomes.

Candidates may confuse transparency (disclosing information) with accountability (taking responsibility), especially when the scenario includes a disclaimer, which is a transparency action, but the reporting mechanism shifts the focus to accountability.

Image Analysis with descriptive tags and captions identifies objects and scenes but does not provide bounding box coordinates to locate objects within the frame, which is required for detecting and locating cars, pedestrians, and bicycles.

The OCR API extracts text from images, not objects like cars or pedestrians. The question requires detecting and locating objects, not reading text.

The department lacks a labeled dataset for custom training, so Custom Vision object detection cannot be used without first creating and training a custom model with labeled images.

If the traffic department only needed to generate a list of objects present in each frame (e.g., 'car, pedestrian, bicycle') without needing their positions, Image Analysis descriptive tags would be the correct choice.

A company needs to extract license plate numbers from traffic camera images to identify vehicles. They would use OCR to read the alphanumeric text on plates.

A company has a large labeled dataset of specific objects (e.g., rare bird species) and needs to detect only those objects in images. Custom Vision object detection would be correct because it allows training a custom model on their own dataset.

Candidates may confuse general image description with object detection, assuming that 'tags and captions' include location information, or they may not be aware that Image Analysis 4.0 offers a dedicated object detection feature.

Candidates may confuse OCR with object detection because both analyze images, or they might think traffic cameras primarily read license plates.

Candidates may think 'custom' implies flexibility for any scenario, overlooking the prerequisite of having a labeled dataset for training.

The Chat Completions API is used to generate responses from a model, not to estimate token counts before making a call. It does not provide a token count estimate without actually processing the prompt.

The Embeddings API converts text into vector representations for semantic similarity, not for counting tokens in a prompt. It does not provide token count estimates for API calls.

The Content Filter configuration is used to filter harmful or inappropriate content in prompts and completions, not to estimate token counts for cost management.

When the question asks which API to use to generate text completions or chat responses from a model, such as 'Which Azure OpenAI Service API should be used to generate a summary of a document?'

A question asking which Azure OpenAI Service feature to use for converting text into numerical vectors to measure semantic similarity between documents would have the Embeddings API as the correct answer.

A question asks: 'Which Azure OpenAI Service feature should be configured to prevent the model from generating offensive language in responses?' In that case, the Content Filter configuration would be the correct answer.

Candidates may confuse the API that generates completions with the tool that estimates tokens, assuming the API itself can provide token counts as part of its response.

Candidates may confuse tokenization with embeddings, thinking that embeddings involve token counting or that the API returns token usage information.

Candidates may confuse content filtering with input validation or preprocessing, mistakenly thinking it could analyze or count tokens in the prompt before the API call.

Accuracy is misleading for imbalanced datasets because a model can achieve high accuracy by simply predicting the majority class (legitimate transactions), while failing to detect the minority class (fraud). Here, 99.9% accuracy can occur even if the model never predicts fraud.

Mean Squared Error (MSE) is a regression metric, not suitable for classification tasks like fraud detection. It measures average squared difference between predicted and actual values, which doesn't apply to binary outcomes.

R-squared is a metric for regression models, measuring the proportion of variance explained by the model. It is not applicable to classification tasks like fraud detection.

When the dataset is balanced (e.g., equal numbers of positive and negative classes) and the costs of false positives and false negatives are similar, accuracy is a straightforward and appropriate metric for overall model performance.

A data scientist builds a regression model to predict house prices. The model's performance is evaluated on a test set, and the goal is to minimize large prediction errors. MSE is the appropriate metric because it penalizes larger errors more heavily.

In a regression problem where the goal is to evaluate how well a linear model fits the data, such as predicting house prices based on features, R-squared would be the appropriate metric to assess model performance.

Candidates often default to accuracy as the primary metric because it is intuitive and commonly used, without considering the impact of class imbalance on its validity.

Candidates may confuse MSE as a general error metric applicable to any model, not realizing it is specific to regression and inappropriate for classification with imbalanced classes.

Candidates may confuse R-squared with a classification metric or think it measures overall model fit regardless of task, especially if they have limited experience with evaluation metrics.

Overfitting would cause high accuracy on training data but poor generalization to new data from the same distribution. Here, the poor performance is due to a shift in class distribution (50% purchasers in training vs 5% in deployment), not overfitting.

Underfitting would cause poor performance on both training and test sets, but here the model achieved 90% accuracy on the test set, indicating it captured patterns well. The issue is a mismatch between training and deployment data distributions, not insufficient model complexity.

Data leakage would cause inflated accuracy on both training and test sets, but here the test set accuracy (90%) is consistent with the training distribution (50% purchasers), not the deployment distribution (5% purchasers). The poor performance is due to distribution shift, not leakage.

A model achieves 99% accuracy on training data but only 70% on a test set drawn from the same distribution. The model memorized noise or specific patterns in the training data, failing to generalize to unseen data from the same source.

A model trained on a dataset with complex patterns achieves low accuracy on both training and test sets (e.g., 60% on a binary classification task). This indicates the model is too simple to capture underlying relationships, making underfitting the likely cause.

A model achieves 99% accuracy on the test set but performs poorly in production. Investigation reveals that the test set contained future data (e.g., time-based leakage) or features that indirectly reveal the target (e.g., customer ID). In that scenario, data leakage is the cause.

Candidates often attribute any performance drop after deployment to overfitting, without considering that the data distribution itself has changed (covariate shift or class imbalance shift).

Candidates may assume poor deployment performance is due to the model not learning enough, especially when the training data is balanced but real-world data is imbalanced, confusing underfitting with distribution shift.

Candidates may confuse high test accuracy with overfitting or leakage, especially when the model fails in production. They might think that any gap between test and real-world performance is due to leakage, without considering distribution shift.