NCP-GENL Evaluation — Timed 10 Questions

Question 1

A financial services company has fine-tuned a Llama-3-70B model using NVIDIA NeMo to generate investment summaries. They need to evaluate the model's output for factual consistency against a set of gold-standard analyst reports. The team wants a metric that measures the longest common subsequence between the generated summary and the reference summary, ignoring word order. Which evaluation metric should they use?

10 minute time limit — choose an answer to begin.

Accepted Answer

ROUGE-L. ROUGE-L is designed for summarization evaluation and uses the longest common subsequence to capture in-order matching without requiring contiguous matches. It balances precision and recall via an F-measure, making it suitable for comparing generated summaries against gold references. The other metrics either focus on n-gram precision, semantic similarity, or unigram alignment, and none directly compute LCS as required.

Answer

BERTScore

Answer

METEOR

Answer

BLEU

Question 2

A team is evaluating a fine-tuned GPT-style model using NVIDIA NeMo. They want to assess the model's performance on a question-answering task. Which two metrics are most appropriate for measuring exact match and semantic similarity between the model's answers and reference answers? (Choose two.)

Accepted Answer

Exact Match (EM). Exact Match and F1 score are the standard metrics for question answering. Exact Match checks if the predicted answer exactly matches the reference after normalization, while F1 score computes token-level overlap, capturing partial correctness and semantic similarity. Other metrics like BLEU, ROUGE-L, and perplexity are not designed for QA and do not provide the required measurements.

Answer

BLEU

Answer

Perplexity

Answer

ROUGE-L

Question 3

You are evaluating a fine-tuned NVIDIA Nemotron-4 15B model for a customer-facing summarization task. The model was fine-tuned using NVIDIA NeMo on a domain-specific dataset. During evaluation, you notice that the model produces summaries that are fluent but frequently omit critical numerical data present in the source documents. You need to quantify this omission issue. Which evaluation metric should you prioritize?

Accepted Answer

ROUGE-1 with entity masking for numbers. The model's fluent but incomplete summaries omit numerical data, so the evaluation must specifically detect missing numbers. Standard metrics like ROUGE-L or BERTScore may not penalize numerical omissions sufficiently. FactCC checks entailment but does not isolate numbers. Masking numbers as entities and using ROUGE-1 recall ensures each critical number contributes to the score, making this the most appropriate metric.

Answer

FactCC

Answer

BERTScore

Answer

ROUGE-L

Question 4

You are evaluating a large language model (LLM) deployed with NVIDIA Triton Inference Server for a customer service chatbot. The model's responses are sometimes irrelevant to the user's query, and you need to set up an automated evaluation pipeline to detect this. Which two metrics are most appropriate for this task? (Choose two.)

Accepted Answer

METEOR. Detecting irrelevance requires metrics that assess semantic similarity between the response and the expected answer. BERTScore uses contextual embeddings to capture meaning, and METEOR aligns synonyms and paraphrases, both making them robust to lexical variations. Perplexity, Exact Match, and ROUGE-L are less effective: perplexity ignores query-response relation, Exact Match is too strict, and ROUGE-L relies on surface overlap.

Answer

ROUGE-L

Answer

Exact Match (EM)

Answer

Perplexity

Question 5

An enterprise fine-tunes a Llama-3-70B model using NVIDIA NeMo for automated technical support ticketing. The development team needs an automated evaluation pipeline that measures semantic similarity against human-curated reference answers without relying on costly human annotators. Which metric provides the most robust embedding-based semantic similarity assessment for this scenario?

Accepted Answer

BERTScore computes token similarity matrices using contextual embeddings to capture deep semantic meaning regardless of surface-level phrasing variations.. BERTScore leverages contextual embeddings from transformer models to evaluate token-level semantic overlap rather than exact string matching, making it ideal for technical support text where phrasing varies. This automated metric correlates strongly with human judgment, significantly accelerating iteration cycles during enterprise model development workflows on NVIDIA infrastructure.

Answer

ROUGE-1 measures unigram overlap but fails to capture semantic synonyms and contextual nuances common in enterprise technical support documentation.

Answer

BLEU evaluates n-gram precision with a brevity penalty, heavily penalizing valid creative paraphrasing typically found in conversational AI support responses.

Answer

Perplexity measures how well a probability distribution predicts a sample, reflecting language fluency rather than semantic alignment with a specific reference answer.

Question 6

A financial services company has fine-tuned a Llama-3-70B model using NVIDIA NeMo to generate investment summaries. They need to evaluate the model's output for factual consistency against a set of gold-standard analyst reports. The team wants a metric that measures the longest common subsequence between the generated summary and the reference summary, ignoring word order. Which evaluation metric should they use?

10 minute time limit — choose an answer to begin.

Accepted Answer

Semantic similarity between outputs for paraphrased inputs. The team needs to measure output consistency across semantically equivalent inputs. Semantic similarity between responses to paraphrased queries directly captures this variance, unlike lexical overlap metrics such as BLEU or ROUGE, which compare to references. Perplexity reflects language modeling confidence, not answer stability. Therefore, semantic similarity is the correct choice for quantifying inconsistency in the fine-tuned model's responses.

Answer

ROUGE score

Answer

BLEU score

Answer

Perplexity

Question 7

A healthcare AI team is evaluating a fine-tuned GPT-based model for clinical note summarization using NVIDIA NeMo. They need to assess whether the model's summaries contain no fabricated medical facts. Which evaluation approach is most appropriate?

Accepted Answer

Use a factual consistency metric such as SummaC or Q2. The critical requirement is detecting fabricated medical facts, which demands checking entailment between the summary and the source clinical note. Factual consistency metrics such as SummaC or Q2 are specifically designed for this purpose, unlike similarity-based metrics (BERTScore, ROUGE-L) that compare to references and may miss hallucinations. Perplexity assesses fluency, not factuality. Therefore, a factual consistency metric is the correct evaluation approach.

Answer

Measure BERTScore against reference summaries

Answer

Compute perplexity of the summaries

Answer

Calculate ROUGE-L scores

Question 8

You are evaluating a fine-tuned Llama-3-70B model for a customer service chatbot using NVIDIA NeMo Evaluation. The model's outputs are factually correct but often verbose, exceeding the desired response length. Which metric should you prioritize to quantify this issue?

Accepted Answer

Average response length. The core issue is that responses are too long, so the evaluation must quantify length. Average response length provides a direct, interpretable measure of verbosity. Other metrics like ROUGE-L, BLEU, and perplexity assess content overlap or fluency, not length, and thus cannot reliably diagnose the problem.

Answer

BLEU

Answer

Perplexity

Answer

ROUGE-L

Question 9

A team is evaluating a retrieval-augmented generation (RAG) pipeline using NVIDIA NeMo Evaluation. They notice that the generated answers are fluent but sometimes contradict the retrieved documents. Which evaluation approach best identifies this issue?

Accepted Answer

Use a natural language inference (NLI) model to check entailment between retrieved documents and generated answers.. The problem is factual inconsistency between retrieved documents and generated answers. Natural language inference directly evaluates entailment, identifying contradictions. Other metrics like BLEU, perplexity, or document length do not measure this relationship, making NLI the appropriate choice for this RAG evaluation scenario.

Answer

Compute BLEU score against reference answers.

Answer

Calculate perplexity of the generated answers.

Answer

Measure the average length of retrieved documents.

Question 10

You are using NVIDIA NeMo Evaluation to assess a summarization model. The model produces summaries that are grammatically correct but omit key information from the source. Which metric should you use to quantify the amount of missing content?

Accepted Answer

ROUGE-1 recall. The problem is omission of key information from summaries. ROUGE-1 recall directly measures how much of the reference unigrams are present in the generated summary. Low recall indicates missing content. Precision, BLEU, and perplexity focus on relevance, precision, or fluency, not on capturing all reference content, making recall the correct choice.

Answer

Perplexity

Answer

BLEU score

Answer

ROUGE-1 precision