NCA-GENL Core Machine Learning and AI Knowledge Practice Test 6 — 15 Questions

Question 1

A data scientist is fine-tuning a pretrained large language model on a small domain-specific dataset. The model achieves high accuracy on the training set but poor performance on a held-out validation set. Which technique is most likely to improve the model's generalization?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Apply regularization techniques such as dropout or weight decay.. The described symptoms—high training accuracy but poor validation accuracy—are classic signs of overfitting. Regularization techniques such as dropout, weight decay, or early stopping are designed to reduce overfitting by penalizing complexity or adding noise during training. Among the options, applying regularization directly targets the cause, whereas the others either exacerbate overfitting or do not address generalization.

Answer

Increase the number of training epochs to further reduce training loss.

Answer

Reduce the size of the validation set to decrease evaluation variance.

Answer

Increase the learning rate to speed up convergence on the training set.

Question 2

A data scientist is pretraining a 12-layer transformer encoder on a corpus of legal contracts. To prevent the model from simply copying each token to its output during masked language modeling, the team needs a strategy that forces the model to learn bidirectional context. Which masking approach should they apply?

Accepted Answer

Mask 15% of input tokens at random, replacing 80% with [MASK], 10% with a random token, and 10% with the original token.. The standard masked language modeling objective masks a small percentage of tokens and requires the model to reconstruct them from bidirectional context. Using mostly [MASK] with some random and unchanged tokens balances learning signal and reduces pretraining-finetuning mismatch. This is the correct way to force bidirectional understanding in a transformer encoder.

Answer

Mask the final token in every sequence and train the model to predict only that token.

Answer

Replace 50% of tokens with [MASK] and leave the rest unchanged.

Answer

Randomly shuffle the order of tokens within each sequence before feeding them to the model.

Question 3

A machine learning engineer is evaluating a generative language model for a chatbot application. They notice that the model frequently generates repetitive phrases and gets stuck in loops. Which decoding strategy is most likely to reduce this repetition?

Accepted Answer

Nucleus sampling with a repetition penalty. Repetition in generated text often stems from decoding strategies that favor high-probability tokens. Nucleus sampling with a repetition penalty addresses this by dynamically truncating the probability distribution and penalizing repeated tokens, encouraging diverse and non-repetitive outputs. Greedy and beam search lack such penalties, and top-k with small k restricts diversity.

Answer

Top-k sampling with a small k

Answer

Greedy search

Answer

Beam search with a large beam width

Question 4

A research team is pretraining a transformer on a corpus of 200 billion tokens. They want the model to learn bidirectional context so each token attends to both left and right neighbors during pretraining. Which pretraining objective fits this requirement?

Accepted Answer

Masked language modeling. Bidirectional context means each token's representation is informed by both preceding and following tokens. Masked language modeling achieves this by hiding random tokens and requiring the model to reconstruct them from the full unmasked context, so attention spans the entire sequence. Autoregressive and causal-decoder objectives enforce left-to-right masking, which precludes bidirectional attention.

Answer

Contrastive next-sentence prediction

Answer

Sequence-to-sequence denoising with a causal decoder

Answer

Autoregressive next-token prediction

Question 5

A machine learning engineer is evaluating a large language model (LLM) on a text generation task. They observe that the model produces coherent and fluent sentences, but the content is factually incorrect and sometimes contradicts known facts. Which term best describes this phenomenon?

Accepted Answer

Hallucination. The phenomenon described—fluent but factually incorrect text generation—is known as hallucination. It occurs because LLMs are trained to predict likely sequences of words, not to verify facts. Hallucination is a significant challenge for deploying LLMs in applications requiring factual accuracy, and it can be mitigated with retrieval-augmented generation or factual consistency checks.

Answer

Catastrophic forgetting

Answer

Overfitting

Answer

Mode collapse

Question 6

A developer is using a pretrained large language model for a text summarization task. They want to adapt the model to a domain-specific corpus of legal documents but have limited GPU memory and a small labeled dataset. Which fine-tuning approach is most parameter-efficient and suitable for this scenario?

Accepted Answer

Low-Rank Adaptation (LoRA). LoRA is a parameter-efficient fine-tuning method that freezes the base model and trains small rank-decomposition matrices. It drastically cuts memory usage and trainable parameters, making it feasible on limited hardware and small datasets. Full fine-tuning is resource-heavy, training from scratch is impractical, and training only the head under-adapts to the specialized legal domain.

Answer

Training the model from scratch on the legal corpus

Answer

Freezing all layers and training only the output head

Answer

Full fine-tuning of all model parameters

Question 7

A machine learning engineer is preprocessing a dataset for a generative AI model and wants to ensure that the input features have a similar scale. Which technique is most appropriate?

Accepted Answer

Normalize numerical features to have zero mean and unit variance.. Normalizing numerical features to zero mean and unit variance brings them to a similar scale, which aids optimization and model performance. One-hot encoding is for categorical data, outlier removal is for data cleaning, and learning rate is unrelated to feature scaling. Thus, normalization is the correct preprocessing technique.

Answer

Increase the learning rate during training.

Answer

Remove outliers from the dataset.

Answer

Apply one-hot encoding to all categorical features.

Question 8

An ML engineer is training a transformer-based language model on a single NVIDIA A100 GPU. They observe that the training loss decreases initially but then becomes NaN after a few hundred steps. The learning rate is 1e-4, and mixed precision with FP16 is enabled. Which action is most likely to stabilize training while preserving the benefits of mixed precision?

Accepted Answer

Enable dynamic loss scaling to automatically adjust the loss scale factor during training.. Dynamic loss scaling is the standard technique to prevent underflow and overflow in FP16 mixed precision training. It scales the loss up to keep gradients in representable range and automatically backs off when NaNs occur. This stabilizes training while retaining the performance and memory advantages of mixed precision, unlike switching to FP32 or altering hyperparameters.

Answer

Switch to FP32 training for all layers.

Answer

Reduce the batch size to 1 to minimize memory usage.

Answer

Increase the learning rate to 1e-3 to escape the NaN region faster.

Question 9

A team is deploying a large language model for real-time inference on an NVIDIA GPU. They observe that the first few inference requests have high latency, but subsequent requests are much faster. What is the most likely explanation for this behavior?

Accepted Answer

CUDA kernels are being compiled and cached. The initial high latency is due to just-in-time compilation of CUDA kernels, which occurs on the first execution of each operation. Once compiled, the kernels are cached for reuse, making subsequent inferences faster. This is a well-known behavior in deep learning frameworks and explains the warm-up effect.

Answer

The model is being quantized on the fly

Answer

The GPU is warming up its clock speed

Answer

The model weights are being loaded from disk

Question 10

A machine learning engineer is fine-tuning a pre-trained language model on a small domain-specific dataset. She notices that the model quickly achieves high accuracy on the training set but performs poorly on the validation set. She wants to mitigate this overfitting without collecting more data. Which technique is most appropriate?

Accepted Answer

Apply L2 regularization. L2 regularization is a classic method to combat overfitting by penalizing large weights, which encourages the model to learn simpler patterns that generalize better. When fine-tuning on a small dataset, it is particularly effective because it constrains the model's capacity to memorize noise, improving validation performance.

Answer

Increase the learning rate

Answer

Train for more epochs

Answer

Use a larger batch size

Question 11

A team deploys a retrieval-augmented generation pipeline and observes that answers frequently cite facts not present in the retrieved passages. They want to reduce this unsupported generation behavior. (Choose two.)

Accepted Answer

Improve the retriever so that top-ranked passages are more relevant to the query. Unsupported generation in retrieval-augmented pipelines stems from two main sources: the generator lacking relevant evidence, and the generator ignoring the evidence it has. Improving retrieval precision supplies the needed facts, while explicit grounding instructions constrain the model to answer only from context and to abstain otherwise. Together they reduce fabricated claims without degrading answer quality.

Answer

Fine-tune the generator on additional unlabeled web text

Answer

Enlarge the retriever's index to include every document in the enterprise

Answer

Increase the generator's temperature to diversify its outputs

Question 12

A developer is building a text summarization assistant that must produce concise, faithful summaries of long support tickets. The team wants to fine-tune a pre-trained large language model on a small labeled dataset of ticket-summary pairs. Which training approach best matches this goal?

Accepted Answer

Supervised fine-tuning (SFT) on the ticket-summary pairs. The team has labeled input-output pairs for a specific generation task, which is exactly what supervised fine-tuning is designed for. SFT updates the pre-trained model's weights to map support tickets to concise summaries, leveraging existing language knowledge while learning the task. Other approaches either ignore the labels, require far more data and compute, or do not modify the model to perform the target task.

Answer

Unsupervised pre-training from scratch on the ticket corpus

Answer

Reinforcement learning from human feedback (RLHF) using pairwise summary preferences

Answer

Retrieval-augmented generation (RAG) with no model weight updates

Question 13

A data scientist is working with a dataset that has a highly skewed distribution, with one class representing only 2% of the samples. They are training a binary classifier and notice that the model predicts the majority class almost exclusively. Which technique is most appropriate to address this issue?

Accepted Answer

Apply class weighting in the loss function. Class imbalance causes models to favor the majority class because the loss function is dominated by those samples. Applying class weights adjusts the loss to penalize minority class errors more heavily, which balances the influence of each class during training. This is a direct and effective technique for improving minority class detection.

Answer

Remove the minority class samples

Answer

Use a simpler model architecture

Answer

Increase the learning rate

Question 14

A data scientist is preparing a dataset of 50,000 customer support chat transcripts to fine-tune an LLM for a helpdesk assistant. The raw text contains HTML tags, inconsistent whitespace, and occasional personal information such as email addresses. Which preprocessing step should be performed FIRST to prepare the text for tokenization?

Accepted Answer

Clean the text by stripping HTML, normalizing whitespace, and masking personally identifiable information.. Raw chat transcripts often contain markup and private data that degrade fine-tuning quality and raise privacy concerns. Cleaning the text by removing HTML, normalizing whitespace, and masking emails or phone numbers before tokenization ensures the tokenizer produces meaningful tokens and that sensitive data never reaches the model. This ordering also avoids retokenization work later.

Answer

Apply tokenization using the model's tokenizer.

Answer

Convert all text to lowercase and remove all punctuation.

Answer

Split the transcripts into training and validation sets.

Question 15

A team is evaluating a generative large language model for a customer support chatbot. They need to ensure the model produces factually accurate and contextually appropriate responses while avoiding harmful or biased outputs. Which two techniques are most effective for aligning the model's behavior with these safety and quality requirements? (Choose two.)

Accepted Answer

Supervised fine-tuning on curated, high-quality demonstrations. RLHF and supervised fine-tuning on curated demonstrations are central alignment techniques. RLHF uses human preferences to optimize for safety and helpfulness, while supervised fine-tuning directly teaches desired responses. Temperature, parameter count, and batch size do not specifically improve factual accuracy or reduce harmful outputs.

Answer

Using a larger batch size during pretraining

Answer

Reducing the model's parameter count

Answer

Increasing the model's temperature during inference