NCP-GENL Evaluation — Timed 15 Questions

Question 1

A financial services company has fine-tuned a Llama-3-70B model using NVIDIA NeMo to generate investment summaries. They need to evaluate the model's output for factual consistency against a set of gold-standard analyst reports. The team wants a metric that measures the longest common subsequence between the generated summary and the reference summary, ignoring word order. Which evaluation metric should they use?

15 minute time limit — choose an answer to begin.

Accepted Answer

ROUGE-L. ROUGE-L is designed for summarization evaluation and uses the longest common subsequence to capture in-order matching without requiring contiguous matches. It balances precision and recall via an F-measure, making it suitable for comparing generated summaries against gold references. The other metrics either focus on n-gram precision, semantic similarity, or unigram alignment, and none directly compute LCS as required.

Answer

BERTScore

Answer

METEOR

Answer

BLEU

Question 2

A team is evaluating a fine-tuned GPT-style model using NVIDIA NeMo. They want to assess the model's performance on a question-answering task. Which two metrics are most appropriate for measuring exact match and semantic similarity between the model's answers and reference answers? (Choose two.)

Accepted Answer

Exact Match (EM). Exact Match and F1 score are the standard metrics for question answering. Exact Match checks if the predicted answer exactly matches the reference after normalization, while F1 score computes token-level overlap, capturing partial correctness and semantic similarity. Other metrics like BLEU, ROUGE-L, and perplexity are not designed for QA and do not provide the required measurements.

Answer

BLEU

Answer

Perplexity

Answer

ROUGE-L

Question 3

You are evaluating a fine-tuned NVIDIA Nemotron-4 15B model for a customer-facing summarization task. The model was fine-tuned using NVIDIA NeMo on a domain-specific dataset. During evaluation, you notice that the model produces summaries that are fluent but frequently omit critical numerical data present in the source documents. You need to quantify this omission issue. Which evaluation metric should you prioritize?

Accepted Answer

ROUGE-1 with entity masking for numbers. The model's fluent but incomplete summaries omit numerical data, so the evaluation must specifically detect missing numbers. Standard metrics like ROUGE-L or BERTScore may not penalize numerical omissions sufficiently. FactCC checks entailment but does not isolate numbers. Masking numbers as entities and using ROUGE-1 recall ensures each critical number contributes to the score, making this the most appropriate metric.

Answer

FactCC

Answer

BERTScore

Answer

ROUGE-L

Question 4

You are evaluating a large language model (LLM) deployed with NVIDIA Triton Inference Server for a customer service chatbot. The model's responses are sometimes irrelevant to the user's query, and you need to set up an automated evaluation pipeline to detect this. Which two metrics are most appropriate for this task? (Choose two.)

Accepted Answer

METEOR. Detecting irrelevance requires metrics that assess semantic similarity between the response and the expected answer. BERTScore uses contextual embeddings to capture meaning, and METEOR aligns synonyms and paraphrases, both making them robust to lexical variations. Perplexity, Exact Match, and ROUGE-L are less effective: perplexity ignores query-response relation, Exact Match is too strict, and ROUGE-L relies on surface overlap.

Answer

ROUGE-L

Answer

Exact Match (EM)

Answer

Perplexity

Question 5

An enterprise fine-tunes a Llama-3-70B model using NVIDIA NeMo for automated technical support ticketing. The development team needs an automated evaluation pipeline that measures semantic similarity against human-curated reference answers without relying on costly human annotators. Which metric provides the most robust embedding-based semantic similarity assessment for this scenario?

Accepted Answer

BERTScore computes token similarity matrices using contextual embeddings to capture deep semantic meaning regardless of surface-level phrasing variations.. BERTScore leverages contextual embeddings from transformer models to evaluate token-level semantic overlap rather than exact string matching, making it ideal for technical support text where phrasing varies. This automated metric correlates strongly with human judgment, significantly accelerating iteration cycles during enterprise model development workflows on NVIDIA infrastructure.

Answer

ROUGE-1 measures unigram overlap but fails to capture semantic synonyms and contextual nuances common in enterprise technical support documentation.

Answer

BLEU evaluates n-gram precision with a brevity penalty, heavily penalizing valid creative paraphrasing typically found in conversational AI support responses.

Answer

Perplexity measures how well a probability distribution predicts a sample, reflecting language fluency rather than semantic alignment with a specific reference answer.

Question 6

A financial services company has fine-tuned a Llama-3-70B model using NVIDIA NeMo to generate investment summaries. They need to evaluate the model's output for factual consistency against a set of gold-standard analyst reports. The team wants a metric that measures the longest common subsequence between the generated summary and the reference summary, ignoring word order. Which evaluation metric should they use?

15 minute time limit — choose an answer to begin.

Accepted Answer

Semantic similarity between outputs for paraphrased inputs. The team needs to measure output consistency across semantically equivalent inputs. Semantic similarity between responses to paraphrased queries directly captures this variance, unlike lexical overlap metrics such as BLEU or ROUGE, which compare to references. Perplexity reflects language modeling confidence, not answer stability. Therefore, semantic similarity is the correct choice for quantifying inconsistency in the fine-tuned model's responses.

Answer

ROUGE score

Answer

BLEU score

Answer

Perplexity

Question 7

A healthcare AI team is evaluating a fine-tuned GPT-based model for clinical note summarization using NVIDIA NeMo. They need to assess whether the model's summaries contain no fabricated medical facts. Which evaluation approach is most appropriate?

Accepted Answer

Use a factual consistency metric such as SummaC or Q2. The critical requirement is detecting fabricated medical facts, which demands checking entailment between the summary and the source clinical note. Factual consistency metrics such as SummaC or Q2 are specifically designed for this purpose, unlike similarity-based metrics (BERTScore, ROUGE-L) that compare to references and may miss hallucinations. Perplexity assesses fluency, not factuality. Therefore, a factual consistency metric is the correct evaluation approach.

Answer

Measure BERTScore against reference summaries

Answer

Compute perplexity of the summaries

Answer

Calculate ROUGE-L scores

Question 8

You are evaluating a fine-tuned Llama-3-70B model for a customer service chatbot using NVIDIA NeMo Evaluation. The model's outputs are factually correct but often verbose, exceeding the desired response length. Which metric should you prioritize to quantify this issue?

Accepted Answer

Average response length. The core issue is that responses are too long, so the evaluation must quantify length. Average response length provides a direct, interpretable measure of verbosity. Other metrics like ROUGE-L, BLEU, and perplexity assess content overlap or fluency, not length, and thus cannot reliably diagnose the problem.

Answer

BLEU

Answer

Perplexity

Answer

ROUGE-L

Question 9

A team is evaluating a retrieval-augmented generation (RAG) pipeline using NVIDIA NeMo Evaluation. They notice that the generated answers are fluent but sometimes contradict the retrieved documents. Which evaluation approach best identifies this issue?

Accepted Answer

Use a natural language inference (NLI) model to check entailment between retrieved documents and generated answers.. The problem is factual inconsistency between retrieved documents and generated answers. Natural language inference directly evaluates entailment, identifying contradictions. Other metrics like BLEU, perplexity, or document length do not measure this relationship, making NLI the appropriate choice for this RAG evaluation scenario.

Answer

Compute BLEU score against reference answers.

Answer

Calculate perplexity of the generated answers.

Answer

Measure the average length of retrieved documents.

Question 10

You are using NVIDIA NeMo Evaluation to assess a summarization model. The model produces summaries that are grammatically correct but omit key information from the source. Which metric should you use to quantify the amount of missing content?

Accepted Answer

ROUGE-1 recall. The problem is omission of key information from summaries. ROUGE-1 recall directly measures how much of the reference unigrams are present in the generated summary. Low recall indicates missing content. Precision, BLEU, and perplexity focus on relevance, precision, or fluency, not on capturing all reference content, making recall the correct choice.

Answer

Perplexity

Answer

BLEU score

Answer

ROUGE-1 precision

Question 11

A healthcare AI team is evaluating a large language model for clinical note summarization. They want to measure whether the generated summaries contain fabricated information not present in the source notes. Which evaluation approach is most appropriate for detecting hallucinated content?

Accepted Answer

Use a natural language inference (NLI) model to check entailment between the source note and the generated summary.. Natural language inference (NLI) is effective for hallucination detection because it evaluates whether the generated summary is logically entailed by the source document. If the summary introduces unsupported information, the NLI model will not predict entailment. Other metrics like perplexity, BLEU, and ROUGE-L focus on fluency or surface overlap and cannot reliably identify fabricated content. Therefore, NLI-based checking is the most appropriate approach.

Answer

Compute the perplexity of the generated summaries on a held-out set of clinical notes.

Answer

Calculate the BLEU score between the generated summary and a reference summary written by a clinician.

Answer

Measure the ROUGE-L score between the generated summary and the source note.

Question 12

A team is evaluating a large language model for a question-answering system using NVIDIA NeMo Evaluation. They need to assess both the relevance of the answer to the question and its factual correctness. (Choose two.)

Accepted Answer

Factual consistency score using an NLI model. The team must evaluate relevance and factual correctness. Answer relevance score directly measures how well the answer addresses the question, while factual consistency score via NLI checks whether the answer is factually supported. Exact Match, token F1, and BLEU focus on string overlap and do not separately assess relevance and factual correctness, making them less suitable for this dual requirement.

Answer

F1 score over tokens

Answer

BLEU score

Answer

Exact Match (EM)

Question 13

A team has deployed a Llama-3-8B model with NVIDIA Triton Inference Server for real-time chatbot responses. They need to evaluate the model's generation quality on a held-out test set of 500 prompts, focusing on semantic similarity to reference answers while ensuring low latency. Which evaluation approach is most appropriate to use with NVIDIA NeMo Evaluator?

Accepted Answer

Use BERTScore or a similar embedding-based metric via NeMo Evaluator's custom metric interface.. For evaluating semantic similarity in generated text, embedding-based metrics like BERTScore are preferred because they capture meaning beyond exact wording. NeMo Evaluator's extensibility allows incorporating such metrics, providing a more human-aligned assessment for chatbot responses. Perplexity and n-gram overlap metrics like BLEU and ROUGE-L do not adequately measure semantic equivalence, especially when paraphrasing is expected.

Answer

Calculate perplexity on the test set using the model's own logits.

Answer

Compute BLEU score using the sacreBLEU library integrated into NeMo Evaluator.

Answer

Run ROUGE-L scoring on the generated outputs and references.

Question 14

You are evaluating a text generation model using NVIDIA NeMo Evaluation and want to measure how well the generated text matches a reference translation. Which metric is specifically designed for this purpose?

Accepted Answer

BLEU. BLEU is the standard metric for machine translation, measuring n-gram precision with a brevity penalty. It directly compares generated text to reference translations. Perplexity measures fluency without references, ROUGE is for summarization, and METEOR, while translation-oriented, is less commonly used in NeMo Evaluation. Thus, BLEU is the correct metric for this scenario.

Answer

METEOR

Answer

Perplexity

Answer

ROUGE

Question 15

A financial institution is evaluating a fine-tuned GPT-3 model for generating investment advice summaries. They must ensure the model does not produce harmful or biased recommendations. Which evaluation methodology should they implement using NVIDIA NeMo Guardrails and NeMo Evaluator to systematically detect and quantify such issues?

Accepted Answer

Use NeMo Guardrails to define a set of safety policies, then run the model's outputs through the guardrails and measure the percentage of violations flagged.. To systematically detect and quantify harmful or biased outputs, the team should leverage NeMo Guardrails to encode safety policies and measure violation rates. This provides a direct, quantifiable assessment of safety compliance. Other metrics like perplexity, BLEU, or F1 do not target harmful content and therefore cannot fulfill the evaluation requirement.

Answer

Use NeMo Evaluator to compute the F1 score of the model's outputs against a set of correct summaries.

Answer

Compute the model's perplexity on a validation set of financial texts.

Answer

Fine-tune the model further on a dataset of unbiased financial advice and then evaluate with BLEU score.