NCA-GENL Data Analysis and Visualization Practice Test 2 — 15 Questions

Question 1

Which THREE of the following are considered best practices for visualizing LLM evaluation results to key stakeholders?

Accepted Answer

Highlighting key performance indicators (KPIs) like accuracy and latency.. Effective stakeholder communication requires stripping away technical noise and focusing on high-level outcomes. Summarizing complex data with clear benchmarks, using intuitive comparative visuals, and highlighting business impact are essential. Stakeholders are generally not interested in individual token-level errors, but rather the overall accuracy, safety, and reliability of the model. Adhering to these practices ensures that technical progress is understood in the context of business goals, supporting informed decision-making regarding deployment and further investment.

Answer

Displaying raw gradient update matrices.

Answer

Providing the entire training log raw dump.

Question 2

A data scientist observes that the model's loss plateaus early during fine-tuning. Which visualization would best help diagnose if the model is suffering from 'catastrophic forgetting'?

Accepted Answer

A line chart comparing original and new task performance.. Catastrophic forgetting occurs when a model loses the ability to perform tasks it previously mastered while learning new ones. A line chart comparing the model's performance on the original evaluation set versus the new training set over time is the best visualization. Seeing performance on the original tasks plummet while the new task performance improves confirms the issue, allowing developers to adjust training parameters like lower learning rates or replay buffers to maintain overall performance.

Answer

A histogram of activation values.

Answer

A pie chart showing weight distribution.

Answer

A heat map of the training loss per sample.

Question 3

Refer to the exhibit. How should a data scientist interpret this evaluation result regarding the recent model update?

Accepted Answer

The update has caused a statistically significant performance drop.. The score delta is negative, and the confidence interval does not overlap zero, meaning the performance regression is statistically significant. In the context of LLM deployment, this is a clear 'red flag' suggesting that the update has degraded the model's reasoning capabilities on the MMLU benchmark. Instead of deploying, the team must investigate the cause, such as data contamination or poor fine-tuning data, to prevent releasing a model that performs worse than the current production baseline.

Answer

The performance change is statistically insignificant.

Answer

The update has significantly improved model accuracy.

Answer

The results are inconclusive due to the small sample size.

Question 4

Which approach is most effective for visualizing 'attention heads' in a Transformer model to debug why the model ignores specific information?

Accepted Answer

Visualization of attention head weights as heat maps.. Attention maps (often visualized as heat maps or dependency graphs) allow researchers to see which tokens the model focuses on during inference. By visualizing these weights, developers can identify if the model is failing to attend to critical context, which explains why it might ignore provided information in a RAG pipeline. This visibility is essential for understanding the internal logic of the model and fixing grounding issues in generative AI workflows.

Answer

Global average loss across the training run.

Answer

A bar chart of token frequencies in the dataset.

Answer

A histogram of the output sequence length.

Question 5

When evaluating a generative model, why is it important to visualize the distribution of output sequence lengths?

Accepted Answer

To identify issues like repetition or infinite generation loops.. Analyzing sequence length distribution is key to detecting issues like 'infinite loops' or 'verbosity bias', where a model generates unnecessarily long or repetitive outputs. Unexpected peaks in the distribution often point to failure modes where the model struggles to reach a coherent stopping point. Visualizing this allows developers to refine stopping criteria and penalize excessive verbosity, ensuring that the final output is concise, relevant, and efficient for end-users in real-time applications.

Answer

To determine the optimal GPU batch size.

Answer

To measure the training loss convergence rate.

Answer

To visualize the internal weight distribution.

Question 6

You are performing exploratory data analysis on a massive dataset for an LLM training pipeline. You need to visualize the distribution of token frequencies in a corpus of 10 billion tokens. Which visualization technique is most effective for identifying long-tail patterns in power-law distributions typical of natural language data?

Accepted Answer

A log-log scale scatter plot of frequency versus rank. A log-log scale plot is the standard for analyzing power-law distributions. In LLM tokenization, the Zipfian distribution means a few tokens appear very frequently while most appear rarely. By plotting frequency versus rank on both logarithmic axes, you transform the curved power-law distribution into a linear relationship, making it significantly easier to identify outliers, detect artifacts in the tokenization process, and validate the model's expected vocabulary coverage.

Answer

A standard linear scale histogram

Answer

A pie chart showing top 20 token percentages

Answer

A box plot summarizing token length statistics

Question 7

You are refining a dataset for a domain-specific LLM using NVIDIA NeMo. You want to visualize the similarity of documents to ensure your training set covers the required technical domains effectively. Which tool and visualization combination is best suited for this?

Accepted Answer

UMAP projection of document embeddings. Using Sentence-BERT embeddings combined with UMAP (Uniform Manifold Approximation and Projection) is the standard for high-dimensional document visualization. UMAP preserves both local and global data structures better than other methods. By plotting these embeddings, you can visually confirm if your training data covers the entire technical scope required and identify any 'holes' or clusters of off-topic data that need to be removed or augmented.

Answer

K-means clustering represented by simple bar charts

Answer

A word cloud generated from the top 1000 terms

Answer

A heatmap of raw document-term matrices

Question 8

Refer to the exhibit. The monitoring JSON indicates high KV cache fragmentation. Which visualization best helps developers diagnose if this is caused by heterogeneous request lengths in the workload?

Accepted Answer

A histogram of request input/output sequence lengths. KV cache fragmentation occurs when varying sequence lengths lead to non-contiguous memory allocations. By plotting a histogram of 'input sequence lengths' versus 'output sequence lengths', developers can see the variance in request sizes. High variance indicates a need for paged attention or continuous batching optimization, which allows the engine to handle variable lengths efficiently without wasting memory on fragmented cache blocks, directly addressing the performance degradation.

Answer

A line chart of GPU memory temperature

Answer

A heat map of individual GPU core activity

Answer

A scatter plot of token generation probabilities

Question 9

During the evaluation of a Large Language Model, you notice that the model consistently predicts the most frequent tokens regardless of the context. Which visualization would most clearly illustrate this phenomenon of 'probability collapse'?

Accepted Answer

A histogram of token probability distributions. Probability collapse occurs when the model's output distribution becomes overly concentrated on a few high-probability tokens, ignoring the diversity of the context. A probability distribution histogram of the model's top-k predictions shows a sharp peak at the most likely token, with near-zero probability for others. Visualizing this for various prompts demonstrates the lack of entropy, signaling that the model is failing to utilize its full vocabulary effectively.

Answer

A scatter plot of average response length

Answer

A line chart of training loss over time

Answer

A bar chart showing total inference time

Question 10

You are conducting an error analysis on an LLM's performance. Which THREE visualizations are most effective for identifying where the model struggles with factual accuracy in a RAG (Retrieval-Augmented Generation) pipeline?

Accepted Answer

Cosine similarity distribution of retrieved documents. In a RAG pipeline, failure often stems from the retrieval stage (fetching irrelevant context) or the generation stage (hallucination). Visualizing retrieval precision, cosine similarity between retrieved chunks and the query, and the correlation between context relevance and final answer accuracy allows engineers to isolate the failure point. These insights are essential for tuning the retriever, improving document indexing, and refining the prompt engineering for better grounded output.

Answer

A pie chart of the model's total word count

Answer

The memory usage of the vector database

Question 11

You need to compare the performance of two different LLMs on a set of benchmark tasks. Which visualization technique is most appropriate for a side-by-side comparison of multiple performance metrics (e.g., accuracy, latency, and truthfulness)?

Accepted Answer

A radar chart showing normalized metrics. Radar charts (or spider plots) are ideal for comparing models across multiple distinct, normalized metrics. They allow for a comprehensive view of a model's strengths and weaknesses in a single plot. For instance, you can easily see if Model A excels in accuracy while Model B outperforms in latency, providing a clear visual basis for selecting the correct model for specific production use cases and requirements.

Answer

A standard line graph

Answer

A scatter plot with only two axes

Answer

A simple histogram of total parameter count

Question 12

Refer to the exhibit. The model is failing with an OOM at layer 42 during training. What visualization would most likely point to the cause of the memory fragmentation?

Accepted Answer

A memory allocation timeline plot per layer. The OOM and high fragmentation are caused by the interaction of dense-attention mechanisms and large sequence lengths (32k). Dense attention scales quadratically with sequence length, consuming massive memory. A heatmap of 'memory allocation per layer' or a 'memory timeline plot' would reveal the memory spikes during the attention computation stage, confirming that the current architecture requires FlashAttention or sequence parallelization to manage memory more efficiently.

Answer

A histogram of training loss values

Answer

A 3D surface plot of GPU core clock speeds

Answer

A line chart showing model weight distribution

Question 13

You are performing a bias audit on a fine-tuned chat model. You need to visualize the model's responses to sensitive prompts across various demographic categories. Which visualization is most effective for identifying systemic bias?

Accepted Answer

Grouped violin plots of toxicity scores by demographic. Grouped box plots or violin plots of sentiment or 'toxicity' scores are the standard for bias audits. By plotting these scores for different demographic categories (e.g., gender, ethnicity), you can easily identify statistical shifts in model response patterns. These visualizations make it clear if the model is behaving differently toward specific groups, which is critical for meeting ethical safety standards in production-ready generative AI.

Answer

A scatter plot of token generation speed

Answer

A simple bar chart of total word counts

Answer

A matrix plot of model layer activations

Question 14

You are analyzing the quality of a synthetic data generation pipeline for an LLM. You want to ensure the synthetic data does not suffer from 'mode collapse' compared to the real-world dataset. Which visualization technique is most effective for comparing the diversity of the two datasets?

Accepted Answer

A scatter plot of embedding density. Mode collapse is the phenomenon where a generative model produces a limited subset of variations. To detect this, you can compute embeddings for both real and synthetic data and plot them using a density-based approach. If the synthetic density plot is concentrated in small areas compared to the broad coverage of the real data, it indicates mode collapse. This comparison is vital for validating that synthetic data preserves the distribution of the original corpus.

Answer

A bar chart of the number of documents

Answer

A line chart of training loss

Answer

A histogram of average word count

Question 15

You are preparing a quarterly report on an LLM's inference latency for stakeholders. The raw data contains 50,000 individual request latencies in milliseconds. You need a single visualization that shows the full distribution shape, including any long tail of slow requests, without losing information to binning. Which visualization should you use?

Accepted Answer

An empirical cumulative distribution function (ECDF) plot of the latency values. An ECDF plot is ideal when the goal is to preserve all information about a distribution, including its tail, without binning or smoothing. It plots the proportion of requests at or below each latency value, so stakeholders can directly read percentiles and see the exact shape of the slow-request tail in an LLM serving workload.

Answer

A histogram with 20 bins of the latency values

Answer

A box plot of the latency values

Answer

A violin plot of the latency values