Evaluation practice questions

A financial services company has fine-tuned a Llama-3-70B model using NVIDIA NeMo to generate investment summaries. They need to evaluate the model's output for factual consistency against a set of gold-standard analyst reports. The team wants a metric that measures the longest common subsequence between the generated summary and the reference summary, ignoring word order. Which evaluation metric should they use?

Trap 1: BERTScore

BERTScore uses contextual embeddings to compute similarity between tokens, capturing semantic equivalence rather than surface-level n-gram overlap. It does not measure longest common subsequence; instead, it computes cosine similarities and aggregates them. While useful for semantic evaluation, it does not align with the requirement to measure LCS between generated and reference summaries.

Trap 2: METEOR

METEOR computes a harmonic mean of precision and recall based on unigram matches, incorporating synonyms and stemming. While it allows some word order flexibility, it does not explicitly compute the longest common subsequence. METEOR aligns words and computes chunk penalties, but the metric is not defined by LCS, so it does not meet the specific requirement.

Trap 3: BLEU

BLEU measures precision of n-grams, typically up to 4-grams, and includes a brevity penalty. It focuses on exact word order and does not use longest common subsequence, making it less appropriate for summarization where flexible ordering is acceptable. BLEU is more common in machine translation and would not satisfy the requirement of ignoring word order.

A
ROUGE-L
ROUGE-L computes the longest common subsequence between the generated and reference summaries, which captures fluency and in-order matching without requiring contiguous n-grams. It is well-suited for summarization tasks where word order may vary but key content overlap matters, matching the requirement to ignore strict word order and measure factual consistency via LCS.
B
BERTScore
Why it fails: BERTScore uses contextual embeddings to compute similarity between tokens, capturing semantic equivalence rather than surface-level n-gram overlap. It does not measure longest common subsequence; instead, it computes cosine similarities and aggregates them. While useful for semantic evaluation, it does not align with the requirement to measure LCS between generated and reference summaries.
C
METEOR
Why it fails: METEOR computes a harmonic mean of precision and recall based on unigram matches, incorporating synonyms and stemming. While it allows some word order flexibility, it does not explicitly compute the longest common subsequence. METEOR aligns words and computes chunk penalties, but the metric is not defined by LCS, so it does not meet the specific requirement.
D
BLEU
Why it fails: BLEU measures precision of n-grams, typically up to 4-grams, and includes a brevity penalty. It focuses on exact word order and does not use longest common subsequence, making it less appropriate for summarization where flexible ordering is acceptable. BLEU is more common in machine translation and would not satisfy the requirement of ignoring word order.

Question 2mediummulti select

A team is evaluating a fine-tuned GPT-style model using NVIDIA NeMo. They want to assess the model's performance on a question-answering task. Which two metrics are most appropriate for measuring exact match and semantic similarity between the model's answers and reference answers? (Choose two.)

Trap 1: BLEU

BLEU is designed for machine translation and measures n-gram precision with a brevity penalty. While it can be applied to QA, it is not standard and does not directly measure exact match or token-level F1. It focuses on precision of n-grams and may not correlate well with answer correctness for short spans. Thus, it is less appropriate than EM and F1.

Trap 2: Perplexity

Perplexity measures how well a language model predicts a sample, reflecting fluency rather than answer correctness. It does not compare against reference answers and cannot measure exact match or semantic similarity. Therefore, it is not suitable for evaluating QA performance in this context.

Trap 3: ROUGE-L

ROUGE-L is primarily used for summarization and measures longest common subsequence. It does not directly compute exact match or token-level F1, which are the de facto standards for QA. While it can assess overlap, it is not the most appropriate metric for measuring exact match and semantic similarity in QA tasks.

A
Exact Match (EM)
Exact Match measures the percentage of predictions that exactly match the reference answer after normalization. It is a standard metric for question answering, especially on datasets like SQuAD, where answers are spans. It directly addresses the need to measure exact match, making it appropriate for this scenario.
B
BLEU
Why it fails: BLEU is designed for machine translation and measures n-gram precision with a brevity penalty. While it can be applied to QA, it is not standard and does not directly measure exact match or token-level F1. It focuses on precision of n-grams and may not correlate well with answer correctness for short spans. Thus, it is less appropriate than EM and F1.
C
Perplexity
Why it fails: Perplexity measures how well a language model predicts a sample, reflecting fluency rather than answer correctness. It does not compare against reference answers and cannot measure exact match or semantic similarity. Therefore, it is not suitable for evaluating QA performance in this context.
D
F1 score
F1 score computes the harmonic mean of precision and recall based on token overlap between predicted and reference answers. It captures partial matches and is widely used alongside Exact Match for QA evaluation. It provides a measure of semantic similarity at the token level, fulfilling the requirement for semantic similarity assessment.
E
ROUGE-L
Why it fails: ROUGE-L is primarily used for summarization and measures longest common subsequence. It does not directly compute exact match or token-level F1, which are the de facto standards for QA. While it can assess overlap, it is not the most appropriate metric for measuring exact match and semantic similarity in QA tasks.

Question 3mediummultiple choice

You are evaluating a fine-tuned NVIDIA Nemotron-4 15B model for a customer-facing summarization task. The model was fine-tuned using NVIDIA NeMo on a domain-specific dataset. During evaluation, you notice that the model produces summaries that are fluent but frequently omit critical numerical data present in the source documents. You need to quantify this omission issue. Which evaluation metric should you prioritize?

Trap 1: FactCC

FactCC is a factuality metric that checks whether generated text is entailed by the source. It can detect factual inconsistencies, but it does not specifically focus on numerical omissions; it treats all facts equally. A missing number might not trigger a low FactCC score if the remaining text is entailed. Thus it is not the best choice for quantifying numerical omission.

Trap 2: BERTScore

BERTScore uses contextual embeddings to compute semantic similarity, which can capture meaning beyond exact matches. However, it is not designed to detect the absence of specific numerical facts; a summary missing a number may still be semantically close. Therefore, BERTScore would not reliably highlight the omission of critical numerical data in this evaluation.

Trap 3: ROUGE-L

ROUGE-L measures longest common subsequence between generated and reference summaries, capturing fluency and some content overlap. However, it does not specifically penalize missing numerical values; a summary omitting a critical number might still have high ROUGE-L if other words overlap. Thus it fails to directly quantify the omission of key numerical data, making it less suitable for this scenario.

A
FactCC
Why it fails: FactCC is a factuality metric that checks whether generated text is entailed by the source. It can detect factual inconsistencies, but it does not specifically focus on numerical omissions; it treats all facts equally. A missing number might not trigger a low FactCC score if the remaining text is entailed. Thus it is not the best choice for quantifying numerical omission.
B
BERTScore
Why it fails: BERTScore uses contextual embeddings to compute semantic similarity, which can capture meaning beyond exact matches. However, it is not designed to detect the absence of specific numerical facts; a summary missing a number may still be semantically close. Therefore, BERTScore would not reliably highlight the omission of critical numerical data in this evaluation.
C
ROUGE-1 with entity masking for numbers
ROUGE-1 with entity masking for numbers computes unigram overlap while explicitly treating numerical entities as separate tokens, ensuring that missing numbers are penalized. This directly addresses the omission of critical numerical data by making each number count in the recall calculation. It is a targeted adaptation of a standard metric to the scenario's specific failure mode.
D
ROUGE-L
Why it fails: ROUGE-L measures longest common subsequence between generated and reference summaries, capturing fluency and some content overlap. However, it does not specifically penalize missing numerical values; a summary omitting a critical number might still have high ROUGE-L if other words overlap. Thus it fails to directly quantify the omission of key numerical data, making it less suitable for this scenario.

Question 4hardmulti select

You are evaluating a large language model (LLM) deployed with NVIDIA Triton Inference Server for a customer service chatbot. The model's responses are sometimes irrelevant to the user's query, and you need to set up an automated evaluation pipeline to detect this. Which two metrics are most appropriate for this task? (Choose two.)

Trap 1: ROUGE-L

ROUGE-L measures longest common subsequence overlap with a reference, focusing on lexical similarity. It may not capture semantic relevance; a response with different wording but correct meaning could score low. Thus, it is less robust for detecting irrelevance in conversational responses where paraphrasing is common. It is not among the most appropriate metrics for this task.

Trap 2: Exact Match (EM)

Exact Match requires the generated response to exactly match a reference answer. In a chatbot scenario, responses are often paraphrased, so exact match would incorrectly flag relevant responses as irrelevant. It is too strict and not suitable for evaluating relevance in open-ended dialogue, where multiple valid phrasings exist.

Trap 3: Perplexity

Perplexity measures how well the model predicts the next token in a sequence, reflecting fluency and confidence. It does not assess whether the response is relevant to the query. A fluent but irrelevant response can have low perplexity. Therefore, perplexity is not appropriate for detecting irrelevance, as it ignores the relationship between query and response.

A
ROUGE-L
Why it fails: ROUGE-L measures longest common subsequence overlap with a reference, focusing on lexical similarity. It may not capture semantic relevance; a response with different wording but correct meaning could score low. Thus, it is less robust for detecting irrelevance in conversational responses where paraphrasing is common. It is not among the most appropriate metrics for this task.
B
Exact Match (EM)
Why it fails: Exact Match requires the generated response to exactly match a reference answer. In a chatbot scenario, responses are often paraphrased, so exact match would incorrectly flag relevant responses as irrelevant. It is too strict and not suitable for evaluating relevance in open-ended dialogue, where multiple valid phrasings exist.
C
METEOR
METEOR computes alignment between generated and reference text based on exact matches, synonyms, and paraphrases, and it incorporates precision and recall. This makes it more sensitive to semantic equivalence than pure n-gram overlap, allowing it to better detect irrelevant responses that lack meaningful overlap with the expected answer. It is suitable for automated relevance evaluation in chatbot scenarios.
D
BERTScore
BERTScore computes semantic similarity between the generated response and a reference answer using contextual embeddings. It can detect irrelevance by assigning low scores when the response is semantically distant from the expected answer. This makes it suitable for identifying irrelevant responses in an automated evaluation pipeline, as it captures meaning beyond exact wording.
E
Perplexity
Why it fails: Perplexity measures how well the model predicts the next token in a sequence, reflecting fluency and confidence. It does not assess whether the response is relevant to the query. A fluent but irrelevant response can have low perplexity. Therefore, perplexity is not appropriate for detecting irrelevance, as it ignores the relationship between query and response.

Question 5mediummultiple choice

An enterprise fine-tunes a Llama-3-70B model using NVIDIA NeMo for automated technical support ticketing. The development team needs an automated evaluation pipeline that measures semantic similarity against human-curated reference answers without relying on costly human annotators. Which metric provides the most robust embedding-based semantic similarity assessment for this scenario?

Trap 1: ROUGE-1 measures unigram overlap but fails to capture semantic…

ROUGE-1 counts exact unigram matches, so paraphrases and synonyms in support answers score poorly despite identical meaning, making it unsuitable for embedding-based semantic assessment. It is tempting because it is a cheap, widely used reference-based metric, and it would be correct where surface lexical overlap with the reference genuinely reflects answer quality.

Trap 2: BLEU evaluates n-gram precision with a brevity penalty, heavily…

BLEU was originally designed for rigid machine translation tasks and focuses heavily on exact n-gram precision. It lacks semantic understanding and penalizes valid alternative phrasings, making it a poor choice for evaluating modern generative support assistants.

Trap 3: Perplexity measures how well a probability distribution predicts a…

Perplexity scores how fluently a model predicts token sequences, with no reference answer involved, so it cannot measure alignment with curated responses. It is tempting because it is a standard intrinsic language-model metric requiring no labelled data, and it would be correct for comparing base and fine-tuned model fluency, not semantic similarity evaluation.

A
ROUGE-1 measures unigram overlap but fails to capture semantic synonyms and contextual nuances common in enterprise technical support documentation.
Why it fails: ROUGE-1 counts exact unigram matches, so paraphrases and synonyms in support answers score poorly despite identical meaning, making it unsuitable for embedding-based semantic assessment. It is tempting because it is a cheap, widely used reference-based metric, and it would be correct where surface lexical overlap with the reference genuinely reflects answer quality.
B
BLEU evaluates n-gram precision with a brevity penalty, heavily penalizing valid creative paraphrasing typically found in conversational AI support responses.
Why it fails: BLEU was originally designed for rigid machine translation tasks and focuses heavily on exact n-gram precision. It lacks semantic understanding and penalizes valid alternative phrasings, making it a poor choice for evaluating modern generative support assistants.
C
BERTScore computes token similarity matrices using contextual embeddings to capture deep semantic meaning regardless of surface-level phrasing variations.
BERTScore aligns tokens between candidate and reference using contextual embeddings, computing precision, recall and F1 over cosine similarity. This satisfies the stem's requirement for embedding-based semantic assessment that captures meaning despite surface-level phrasing variation, without human annotators.
D
Perplexity measures how well a probability distribution predicts a sample, reflecting language fluency rather than semantic alignment with a specific reference answer.
Why it fails: Perplexity scores how fluently a model predicts token sequences, with no reference answer involved, so it cannot measure alignment with curated responses. It is tempting because it is a standard intrinsic language-model metric requiring no labelled data, and it would be correct for comparing base and fine-tuned model fluency, not semantic similarity evaluation.

Question 6mediummultiple choice

A financial services company has fine-tuned a Llama-3-70B model using NVIDIA NeMo for automated loan risk assessment. During evaluation, they observe that the model achieves high scores on standard accuracy metrics but produces inconsistent responses when the same query is phrased slightly differently. They need a metric that quantifies this inconsistency. Which evaluation metric should they use?

Trap 1: ROUGE score

ROUGE evaluates recall-oriented overlap with reference summaries, common in summarization. It cannot detect whether the model yields consistent answers to semantically equivalent queries. Using ROUGE here would measure similarity to a reference, not the variance across multiple phrasings of the same question. Therefore, it does not address the inconsistency issue.

Trap 2: BLEU score

BLEU measures n-gram overlap between generated and reference texts, typically used in translation. It does not assess response consistency across paraphrased inputs. In this scenario, BLEU would penalize valid paraphrases that differ lexically from a reference, failing to capture the semantic stability the team needs. Thus, it is unsuitable for quantifying inconsistency.

Trap 3: Perplexity

Perplexity quantifies how well a language model predicts a sample, reflecting fluency and confidence. It does not compare outputs for the same semantic query. A model could have low perplexity yet produce divergent answers to paraphrases. In this scenario, perplexity would not reveal the inconsistency the team observes, making it the wrong metric.

A
Semantic similarity between outputs for paraphrased inputs
Semantic similarity measures how closely the model's responses align in meaning when the same question is asked with different wording. High similarity indicates consistent understanding and stable generation. In this scenario, computing similarity across paraphrased loan queries directly quantifies the observed inconsistency. This metric is appropriate for detecting and reducing output variance.
B
ROUGE score
Why it fails: ROUGE evaluates recall-oriented overlap with reference summaries, common in summarization. It cannot detect whether the model yields consistent answers to semantically equivalent queries. Using ROUGE here would measure similarity to a reference, not the variance across multiple phrasings of the same question. Therefore, it does not address the inconsistency issue.
C
BLEU score
Why it fails: BLEU measures n-gram overlap between generated and reference texts, typically used in translation. It does not assess response consistency across paraphrased inputs. In this scenario, BLEU would penalize valid paraphrases that differ lexically from a reference, failing to capture the semantic stability the team needs. Thus, it is unsuitable for quantifying inconsistency.
D
Perplexity
Why it fails: Perplexity quantifies how well a language model predicts a sample, reflecting fluency and confidence. It does not compare outputs for the same semantic query. A model could have low perplexity yet produce divergent answers to paraphrases. In this scenario, perplexity would not reveal the inconsistency the team observes, making it the wrong metric.

Question 7hardmultiple choice

A healthcare AI team is evaluating a fine-tuned GPT-based model for clinical note summarization using NVIDIA NeMo. They need to assess whether the model's summaries contain no fabricated medical facts. Which evaluation approach is most appropriate?

Trap 1: Measure BERTScore against reference summaries

BERTScore computes semantic similarity between generated and reference texts using contextual embeddings. While it captures semantic overlap, it does not verify factual accuracy against source documents. A summary could be semantically similar to a reference yet contain fabricated facts not present in the source. In this clinical scenario, BERTScore alone would not detect hallucinations, so it is insufficient.

Trap 2: Compute perplexity of the summaries

Perplexity measures how well the language model predicts the summary text, reflecting fluency. It does not compare the summary to the source document for factual accuracy. A fluent but hallucinated summary could have low perplexity. In clinical note summarization, perplexity would not flag fabricated facts, so it is not a valid evaluation method for this requirement.

Trap 3: Calculate ROUGE-L scores

ROUGE-L measures longest common subsequence overlap with references, focusing on recall. It is commonly used in summarization but does not check factual consistency with the source. A model could score high by copying phrases while introducing false statements. For clinical notes, where fabricated facts are dangerous, ROUGE-L fails to provide the necessary factual verification, making it inappropriate.

A
Use a factual consistency metric such as SummaC or Q2
Factual consistency metrics like SummaC or Q2 evaluate whether generated summaries are entailed by the source document, detecting hallucinations. They are designed to catch fabricated facts by comparing summary claims against source content. In this healthcare scenario, applying such a metric directly addresses the need to ensure no invented medical facts, making it the most suitable approach.
B
Measure BERTScore against reference summaries
Why it fails: BERTScore computes semantic similarity between generated and reference texts using contextual embeddings. While it captures semantic overlap, it does not verify factual accuracy against source documents. A summary could be semantically similar to a reference yet contain fabricated facts not present in the source. In this clinical scenario, BERTScore alone would not detect hallucinations, so it is insufficient.
C
Compute perplexity of the summaries
Why it fails: Perplexity measures how well the language model predicts the summary text, reflecting fluency. It does not compare the summary to the source document for factual accuracy. A fluent but hallucinated summary could have low perplexity. In clinical note summarization, perplexity would not flag fabricated facts, so it is not a valid evaluation method for this requirement.
D
Calculate ROUGE-L scores
Why it fails: ROUGE-L measures longest common subsequence overlap with references, focusing on recall. It is commonly used in summarization but does not check factual consistency with the source. A model could score high by copying phrases while introducing false statements. For clinical notes, where fabricated facts are dangerous, ROUGE-L fails to provide the necessary factual verification, making it inappropriate.

Question 8mediummultiple choice

You are evaluating a fine-tuned Llama-3-70B model for a customer service chatbot using NVIDIA NeMo Evaluation. The model's outputs are factually correct but often verbose, exceeding the desired response length. Which metric should you prioritize to quantify this issue?

Trap 1: BLEU

BLEU computes precision of n-grams with a brevity penalty, but it does not explicitly measure output length. A verbose response may still achieve high BLEU if its n-grams match the reference. Therefore, BLEU is not the right metric to assess verbosity here.

Trap 2: Perplexity

Perplexity measures how well a language model predicts a sample, reflecting fluency and confidence. It does not capture output length; a verbose but fluent response can have low perplexity. Thus, perplexity is irrelevant for diagnosing verbosity in this customer service chatbot.

Trap 3: ROUGE-L

ROUGE-L measures longest common subsequence between generated and reference texts, focusing on recall of n-grams. It does not directly penalize verbosity; a longer response can still have high ROUGE-L if it contains the reference content. Thus, it fails to quantify excessive length in this scenario.

A
Average response length
Average response length directly measures the mean number of tokens or words in generated responses. Since the issue is verbosity, this metric quantifies the problem precisely. It can be computed easily within NeMo Evaluation and compared against a target threshold to guide fine-tuning adjustments.
B
BLEU
Why it fails: BLEU computes precision of n-grams with a brevity penalty, but it does not explicitly measure output length. A verbose response may still achieve high BLEU if its n-grams match the reference. Therefore, BLEU is not the right metric to assess verbosity here.
C
Perplexity
Why it fails: Perplexity measures how well a language model predicts a sample, reflecting fluency and confidence. It does not capture output length; a verbose but fluent response can have low perplexity. Thus, perplexity is irrelevant for diagnosing verbosity in this customer service chatbot.
D
ROUGE-L
Why it fails: ROUGE-L measures longest common subsequence between generated and reference texts, focusing on recall of n-grams. It does not directly penalize verbosity; a longer response can still have high ROUGE-L if it contains the reference content. Thus, it fails to quantify excessive length in this scenario.

Question 9hardmultiple choice

A team is evaluating a retrieval-augmented generation (RAG) pipeline using NVIDIA NeMo Evaluation. They notice that the generated answers are fluent but sometimes contradict the retrieved documents. Which evaluation approach best identifies this issue?

Trap 1: Compute BLEU score against reference answers.

BLEU compares n-gram overlap with reference answers, but references may not exist or may not reflect retrieved documents. Contradictions with retrieved context are not captured by BLEU, as it focuses on surface similarity to references, not factual consistency with source documents. Thus, it fails to detect the contradiction problem.

Trap 2: Calculate perplexity of the generated answers.

Perplexity measures language model confidence and fluency, not factual consistency with retrieved documents. A fluent but contradictory answer can have low perplexity. Therefore, perplexity cannot identify contradictions; it only indicates how predictable the text is under the model, which is unrelated to the retrieval grounding issue.

Trap 3: Measure the average length of retrieved documents.

Average document length is a retrieval statistic and does not assess the generated answer's fidelity to the documents. Contradictions can occur regardless of document length. Thus, this metric is irrelevant for detecting whether the generated answer contradicts the retrieved content, and it provides no insight into factual consistency.

A
Compute BLEU score against reference answers.
Why it fails: BLEU compares n-gram overlap with reference answers, but references may not exist or may not reflect retrieved documents. Contradictions with retrieved context are not captured by BLEU, as it focuses on surface similarity to references, not factual consistency with source documents. Thus, it fails to detect the contradiction problem.
B
Calculate perplexity of the generated answers.
Why it fails: Perplexity measures language model confidence and fluency, not factual consistency with retrieved documents. A fluent but contradictory answer can have low perplexity. Therefore, perplexity cannot identify contradictions; it only indicates how predictable the text is under the model, which is unrelated to the retrieval grounding issue.
C
Measure the average length of retrieved documents.
Why it fails: Average document length is a retrieval statistic and does not assess the generated answer's fidelity to the documents. Contradictions can occur regardless of document length. Thus, this metric is irrelevant for detecting whether the generated answer contradicts the retrieved content, and it provides no insight into factual consistency.
D
Use a natural language inference (NLI) model to check entailment between retrieved documents and generated answers.
NLI models determine whether a hypothesis (generated answer) is entailed by, neutral to, or contradicts a premise (retrieved document). By running an NLI model, you can flag contradictions directly. This approach is robust for RAG evaluation because it assesses factual consistency without requiring reference answers, aligning with the goal of detecting contradictions.

Question 10mediummultiple choice

You are using NVIDIA NeMo Evaluation to assess a summarization model. The model produces summaries that are grammatically correct but omit key information from the source. Which metric should you use to quantify the amount of missing content?

Trap 1: Perplexity

Perplexity measures the fluency and predictability of the generated text under a language model. It does not compare against a reference summary and cannot indicate whether key information is missing. A fluent but incomplete summary can have low perplexity, so perplexity is not useful for this evaluation.

Trap 2: BLEU score

BLEU is a precision-oriented metric with a brevity penalty. It does not directly measure recall of reference content. While the brevity penalty may indirectly penalize short summaries, it does not quantify missing information in a granular way. Therefore, BLEU is less appropriate than recall for assessing omissions.

Trap 3: ROUGE-1 precision

ROUGE-1 precision measures how much of the generated summary overlaps with the reference, relative to the generated summary. High precision means the generated content is relevant, but it does not penalize omission; a short summary with high precision can still miss key information. Thus, precision is not suitable for quantifying missing content.

A
Perplexity
Why it fails: Perplexity measures the fluency and predictability of the generated text under a language model. It does not compare against a reference summary and cannot indicate whether key information is missing. A fluent but incomplete summary can have low perplexity, so perplexity is not useful for this evaluation.
B
BLEU score
Why it fails: BLEU is a precision-oriented metric with a brevity penalty. It does not directly measure recall of reference content. While the brevity penalty may indirectly penalize short summaries, it does not quantify missing information in a granular way. Therefore, BLEU is less appropriate than recall for assessing omissions.
C
ROUGE-1 recall
ROUGE-1 recall measures the proportion of unigram overlaps between the generated summary and the reference summary, relative to the reference. Low recall indicates missing content. Since the issue is omission of key information, recall is the appropriate metric to quantify missing content, as it directly reflects how much of the reference is captured.
D
ROUGE-1 precision
Why it fails: ROUGE-1 precision measures how much of the generated summary overlaps with the reference, relative to the generated summary. High precision means the generated content is relevant, but it does not penalize omission; a short summary with high precision can still miss key information. Thus, precision is not suitable for quantifying missing content.

Question 11hardmultiple choice

A healthcare AI team is evaluating a large language model for clinical note summarization. They want to measure whether the generated summaries contain fabricated information not present in the source notes. Which evaluation approach is most appropriate for detecting hallucinated content?

Trap 1: Compute the perplexity of the generated summaries on a held-out set…

Perplexity measures how well a language model predicts a sample, reflecting fluency and likelihood. It does not compare the generated summary against the source note to detect fabricated content. A low perplexity indicates the model finds the text probable, but the text could still contain hallucinations. Thus, perplexity is not suitable for detecting factual inconsistencies or fabricated information.

Trap 2: Calculate the BLEU score between the generated summary and a…

BLEU compares n-gram overlap between generated and reference summaries. While it can indicate similarity, it does not verify factual consistency against the source note. A summary could have high BLEU by copying phrases from the reference but still include hallucinations not in the source. BLEU is not designed to detect fabricated content, so it is inadequate for this purpose.

Trap 3: Measure the ROUGE-L score between the generated summary and the…

ROUGE-L computes longest common subsequence between generated and reference texts, typically used for summarization quality. If the reference is the source note, it measures overlap but does not assess whether the summary is factually supported. A summary could share many sequences with the source yet still introduce fabricated details. ROUGE-L does not detect hallucinations.

A
Use a natural language inference (NLI) model to check entailment between the source note and the generated summary.
NLI models determine whether a hypothesis (the summary) is entailed by a premise (the source note). If the summary contains fabricated information, the NLI model would predict contradiction or neutral rather than entailment. This approach directly assesses factual consistency and is widely used for hallucination detection in summarization, making it the most appropriate method here.
B
Compute the perplexity of the generated summaries on a held-out set of clinical notes.
Why it fails: Perplexity measures how well a language model predicts a sample, reflecting fluency and likelihood. It does not compare the generated summary against the source note to detect fabricated content. A low perplexity indicates the model finds the text probable, but the text could still contain hallucinations. Thus, perplexity is not suitable for detecting factual inconsistencies or fabricated information.
C
Calculate the BLEU score between the generated summary and a reference summary written by a clinician.
Why it fails: BLEU compares n-gram overlap between generated and reference summaries. While it can indicate similarity, it does not verify factual consistency against the source note. A summary could have high BLEU by copying phrases from the reference but still include hallucinations not in the source. BLEU is not designed to detect fabricated content, so it is inadequate for this purpose.
D
Measure the ROUGE-L score between the generated summary and the source note.
Why it fails: ROUGE-L computes longest common subsequence between generated and reference texts, typically used for summarization quality. If the reference is the source note, it measures overlap but does not assess whether the summary is factually supported. A summary could share many sequences with the source yet still introduce fabricated details. ROUGE-L does not detect hallucinations.

Question 12hardmulti select

A team is evaluating a large language model for a question-answering system using NVIDIA NeMo Evaluation. They need to assess both the relevance of the answer to the question and its factual correctness. (Choose two.)

Trap 1: F1 score over tokens

Token-level F1 measures the overlap between generated and reference answers, balancing precision and recall. While it captures partial correctness, it does not evaluate whether the answer is relevant to the question; an answer could overlap with the reference but be irrelevant if the reference is misaligned. Thus, it does not fully address both criteria.

Trap 2: BLEU score

BLEU measures n-gram precision against reference answers, primarily used in machine translation. It does not assess relevance to the question or factual correctness; a high BLEU score can be achieved with fluent but irrelevant or incorrect answers. Therefore, BLEU is not suitable for evaluating the two specified aspects.

Trap 3: Exact Match (EM)

Exact Match measures whether the generated answer exactly matches the reference answer after normalization. It is a strict metric that does not assess relevance to the question or factual correctness beyond exact string match. In many QA scenarios, answers can be phrased differently but still correct, so EM alone is insufficient for evaluating both relevance and correctness.

A
F1 score over tokens
Why it fails: Token-level F1 measures the overlap between generated and reference answers, balancing precision and recall. While it captures partial correctness, it does not evaluate whether the answer is relevant to the question; an answer could overlap with the reference but be irrelevant if the reference is misaligned. Thus, it does not fully address both criteria.
B
BLEU score
Why it fails: BLEU measures n-gram precision against reference answers, primarily used in machine translation. It does not assess relevance to the question or factual correctness; a high BLEU score can be achieved with fluent but irrelevant or incorrect answers. Therefore, BLEU is not suitable for evaluating the two specified aspects.
C
Exact Match (EM)
Why it fails: Exact Match measures whether the generated answer exactly matches the reference answer after normalization. It is a strict metric that does not assess relevance to the question or factual correctness beyond exact string match. In many QA scenarios, answers can be phrased differently but still correct, so EM alone is insufficient for evaluating both relevance and correctness.
D
Factual consistency score using an NLI model
A factual consistency score uses an NLI model to check whether the generated answer is entailed by a trusted knowledge source (e.g., a reference document). This directly measures factual correctness, the second required criterion. It is robust to paraphrasing and can be automated within NeMo Evaluation to flag hallucinations or contradictions.
E
Answer relevance score from a QA evaluation model
An answer relevance score, often computed by a model like a cross-encoder or a specialized QA evaluator, assesses how well the answer addresses the question. This directly measures relevance, which is one of the two required criteria. It can be integrated into NeMo Evaluation to provide a quantitative relevance metric for each generated answer.

Question 13mediummultiple choice

A team has deployed a Llama-3-8B model with NVIDIA Triton Inference Server for real-time chatbot responses. They need to evaluate the model's generation quality on a held-out test set of 500 prompts, focusing on semantic similarity to reference answers while ensuring low latency. Which evaluation approach is most appropriate to use with NVIDIA NeMo Evaluator?

Trap 1: Calculate perplexity on the test set using the model's own logits.

Perplexity measures how well a language model predicts a sample and is useful for intrinsic evaluation, but it does not directly assess semantic similarity to reference answers. A model can have low perplexity yet generate irrelevant or off-topic responses. Thus, perplexity alone would not satisfy the need to evaluate generation quality against references.

Trap 2: Compute BLEU score using the sacreBLEU library integrated into NeMo…

BLEU measures n-gram overlap and is primarily designed for machine translation. For semantic similarity in open-ended chatbot responses, BLEU often correlates poorly with human judgment because it ignores meaning and penalizes valid paraphrases. While NeMo Evaluator can compute BLEU, it is not the most appropriate metric for this scenario's focus on semantic similarity.

Trap 3: Run ROUGE-L scoring on the generated outputs and references.

ROUGE-L focuses on longest common subsequence and recall, often used in summarization. It captures some overlap but still relies on lexical matching, so it may miss semantic equivalence. For a chatbot where paraphrasing is common, ROUGE-L would undervalue correct answers that use different wording, making it less suitable than embedding-based methods.

A
Calculate perplexity on the test set using the model's own logits.
Why it fails: Perplexity measures how well a language model predicts a sample and is useful for intrinsic evaluation, but it does not directly assess semantic similarity to reference answers. A model can have low perplexity yet generate irrelevant or off-topic responses. Thus, perplexity alone would not satisfy the need to evaluate generation quality against references.
B
Use BERTScore or a similar embedding-based metric via NeMo Evaluator's custom metric interface.
BERTScore leverages contextual embeddings to compute semantic similarity between generated and reference texts, aligning with the requirement to assess meaning rather than surface form. NeMo Evaluator supports custom metrics, allowing integration of BERTScore. This approach handles paraphrases well and is efficient for 500 prompts, meeting the low-latency evaluation goal.
C
Compute BLEU score using the sacreBLEU library integrated into NeMo Evaluator.
Why it fails: BLEU measures n-gram overlap and is primarily designed for machine translation. For semantic similarity in open-ended chatbot responses, BLEU often correlates poorly with human judgment because it ignores meaning and penalizes valid paraphrases. While NeMo Evaluator can compute BLEU, it is not the most appropriate metric for this scenario's focus on semantic similarity.
D
Run ROUGE-L scoring on the generated outputs and references.
Why it fails: ROUGE-L focuses on longest common subsequence and recall, often used in summarization. It captures some overlap but still relies on lexical matching, so it may miss semantic equivalence. For a chatbot where paraphrasing is common, ROUGE-L would undervalue correct answers that use different wording, making it less suitable than embedding-based methods.

Question 14easymultiple choice

You are evaluating a text generation model using NVIDIA NeMo Evaluation and want to measure how well the generated text matches a reference translation. Which metric is specifically designed for this purpose?

Trap 1: METEOR

METEOR is a translation evaluation metric that considers synonyms and stemming, aiming to improve correlation with human judgment. However, it is not the standard metric in NVIDIA NeMo Evaluation for translation; BLEU is more commonly used and supported. While METEOR is designed for translation, the question asks for the metric specifically designed for this purpose, which is BLEU. In practice, METEOR could be used, but BLEU is the canonical choice.

Trap 2: Perplexity

Perplexity measures the fluency of a language model by calculating the exponentiated average negative log-likelihood of a sequence. It does not compare against a reference translation and thus cannot assess translation accuracy. Therefore, perplexity is not designed for evaluating translation match, and it is unsuitable here.

Trap 3: ROUGE

ROUGE is primarily used for summarization evaluation, focusing on recall of n-grams or longest common subsequences against reference summaries. While it can be applied to translation, it is not specifically designed for that task and lacks the precision-oriented brevity penalty of BLEU. Thus, it is not the best choice for translation evaluation.

A
BLEU
BLEU (Bilingual Evaluation Understudy) is designed for machine translation evaluation. It computes n-gram precision between generated and reference translations, with a brevity penalty. It is the standard metric for translation quality and is directly applicable when a reference translation exists, making it the correct choice for this scenario.
B
METEOR
Why it fails: METEOR is a translation evaluation metric that considers synonyms and stemming, aiming to improve correlation with human judgment. However, it is not the standard metric in NVIDIA NeMo Evaluation for translation; BLEU is more commonly used and supported. While METEOR is designed for translation, the question asks for the metric specifically designed for this purpose, which is BLEU. In practice, METEOR could be used, but BLEU is the canonical choice.
C
Perplexity
Why it fails: Perplexity measures the fluency of a language model by calculating the exponentiated average negative log-likelihood of a sequence. It does not compare against a reference translation and thus cannot assess translation accuracy. Therefore, perplexity is not designed for evaluating translation match, and it is unsuitable here.
D
ROUGE
Why it fails: ROUGE is primarily used for summarization evaluation, focusing on recall of n-grams or longest common subsequences against reference summaries. While it can be applied to translation, it is not specifically designed for that task and lacks the precision-oriented brevity penalty of BLEU. Thus, it is not the best choice for translation evaluation.

Question 15hardmultiple choice

A financial institution is evaluating a fine-tuned GPT-3 model for generating investment advice summaries. They must ensure the model does not produce harmful or biased recommendations. Which evaluation methodology should they implement using NVIDIA NeMo Guardrails and NeMo Evaluator to systematically detect and quantify such issues?

Trap 1: Use NeMo Evaluator to compute the F1 score of the model's outputs…

F1 score evaluates classification or extraction accuracy against ground truth, but harmful or biased content is not necessarily captured by correctness metrics. A summary could be factually correct yet contain biased language. Thus, F1 score would not reliably detect safety issues, making it inadequate for this scenario.

Trap 2: Compute the model's perplexity on a validation set of financial…

Perplexity measures language modeling confidence, not the presence of harmful or biased content. A model can have low perplexity yet still generate biased advice. Therefore, perplexity does not address the requirement to detect and quantify harmful recommendations, making it unsuitable for this safety evaluation.

Trap 3: Fine-tune the model further on a dataset of unbiased financial…

Fine-tuning may reduce bias but does not guarantee its elimination, and BLEU score only measures n-gram overlap with references. BLEU cannot detect subtle harmful content or bias. This approach neither systematically detects violations nor quantifies them, so it fails to meet the evaluation objective.

A
Use NeMo Evaluator to compute the F1 score of the model's outputs against a set of correct summaries.
Why it fails: F1 score evaluates classification or extraction accuracy against ground truth, but harmful or biased content is not necessarily captured by correctness metrics. A summary could be factually correct yet contain biased language. Thus, F1 score would not reliably detect safety issues, making it inadequate for this scenario.
B
Use NeMo Guardrails to define a set of safety policies, then run the model's outputs through the guardrails and measure the percentage of violations flagged.
NeMo Guardrails allows defining programmable rules (e.g., using Colang) to detect and block harmful or biased content. By applying these guardrails to model outputs and quantifying violation rates, the team can systematically evaluate safety and bias. This approach integrates with NeMo Evaluator to log and analyze flagged instances, providing a quantifiable metric for compliance.
C
Compute the model's perplexity on a validation set of financial texts.
Why it fails: Perplexity measures language modeling confidence, not the presence of harmful or biased content. A model can have low perplexity yet still generate biased advice. Therefore, perplexity does not address the requirement to detect and quantify harmful recommendations, making it unsuitable for this safety evaluation.
D
Fine-tune the model further on a dataset of unbiased financial advice and then evaluate with BLEU score.
Why it fails: Fine-tuning may reduce bias but does not guarantee its elimination, and BLEU score only measures n-gram overlap with references. BLEU cannot detect subtle harmful content or bias. This approach neither systematically detects violations nor quantifies them, so it fails to meet the evaluation objective.

Question 16mediummultiple choice

A media company is using a large language model to generate news article headlines. They want to evaluate the diversity of the generated headlines to ensure they are not overly repetitive. Which metric should they use to quantify the lexical diversity of the generated headlines?

Trap 1: Perplexity

Perplexity measures the model's uncertainty in predicting text, reflecting fluency. Lower perplexity means the model is more confident, but it does not indicate diversity of generated outputs. A model can have low perplexity yet produce repetitive headlines. Hence, perplexity is not a direct measure of lexical diversity.

Trap 2: ROUGE-L

ROUGE-L evaluates longest common subsequence with a reference, focusing on recall of content. It does not measure diversity among multiple generated outputs. It is used for summarization quality, not for assessing repetition or lexical variety. Therefore, it is not appropriate for this scenario.

Trap 3: BLEU

BLEU measures n-gram precision between generated and reference texts, often used in translation. It does not assess diversity within the generated set. A model could produce diverse headlines but low BLEU if references differ, or repetitive headlines with high BLEU. Thus, BLEU is not suitable for quantifying lexical diversity.

A
Perplexity
Why it fails: Perplexity measures the model's uncertainty in predicting text, reflecting fluency. Lower perplexity means the model is more confident, but it does not indicate diversity of generated outputs. A model can have low perplexity yet produce repetitive headlines. Hence, perplexity is not a direct measure of lexical diversity.
B
ROUGE-L
Why it fails: ROUGE-L evaluates longest common subsequence with a reference, focusing on recall of content. It does not measure diversity among multiple generated outputs. It is used for summarization quality, not for assessing repetition or lexical variety. Therefore, it is not appropriate for this scenario.
C
BLEU
Why it fails: BLEU measures n-gram precision between generated and reference texts, often used in translation. It does not assess diversity within the generated set. A model could produce diverse headlines but low BLEU if references differ, or repetitive headlines with high BLEU. Thus, BLEU is not suitable for quantifying lexical diversity.
D
Distinct-n
Distinct-n measures the number of unique n-grams divided by the total number of n-grams in the generated text. It is specifically designed to quantify lexical diversity and repetition. For evaluating headline diversity, Distinct-n provides a direct measure of how varied the word choices are, making it the appropriate metric.

Question 17mediummulti select

A team is using NVIDIA NeMo Evaluator to assess a Llama-3-70B model fine-tuned for medical question answering. They want to evaluate both the correctness of answers and the model's ability to avoid hallucinating unsupported facts. Which two evaluation strategies should they implement? (Choose two.)

Trap 1: Compute BLEU score on the generated answers.

BLEU score measures n-gram overlap and is designed for machine translation. In medical QA, correct answers may use different phrasing, so BLEU would unfairly penalize valid responses. It also does not assess factual correctness or hallucination. Thus, BLEU is not suitable for this evaluation goal.

Trap 2: Calculate perplexity of the model on a held-out set of medical…

Perplexity measures how well the model predicts the next token and is not a direct measure of factual correctness or hallucination. A model can have low perplexity yet still generate false statements. Therefore, perplexity alone does not satisfy the need to evaluate correctness or detect hallucinations.

Trap 3: Run the model through NeMo Guardrails with a policy that blocks any…

Blocking outputs with numbers is an arbitrary rule that would not effectively detect hallucinations; many correct medical answers include numbers (e.g., dosages). This approach would introduce false positives and fail to assess factual accuracy. It does not address the core evaluation goals of correctness and hallucination detection.

A
Compute BLEU score on the generated answers.
Why it fails: BLEU score measures n-gram overlap and is designed for machine translation. In medical QA, correct answers may use different phrasing, so BLEU would unfairly penalize valid responses. It also does not assess factual correctness or hallucination. Thus, BLEU is not suitable for this evaluation goal.
B
Calculate perplexity of the model on a held-out set of medical questions.
Why it fails: Perplexity measures how well the model predicts the next token and is not a direct measure of factual correctness or hallucination. A model can have low perplexity yet still generate false statements. Therefore, perplexity alone does not satisfy the need to evaluate correctness or detect hallucinations.
C
Use a fact-checking module that cross-references generated statements with a trusted medical knowledge base.
A fact-checking module can verify each claim in the model's output against a curated medical knowledge base, directly detecting hallucinations or unsupported facts. This approach provides a granular, evidence-based assessment of factual accuracy, which is critical in healthcare. Integrating such a module with NeMo Evaluator allows automated flagging of unsupported statements.
D
Compare model outputs against gold-standard answers using exact match and F1 score.
Exact match and F1 score provide a quantitative measure of answer correctness relative to ground truth. In medical QA, where precise terminology matters, these metrics can indicate how often the model produces the expected answer. While they may not capture semantic equivalence perfectly, they are standard and efficient for large-scale evaluation, making them a valid component of the assessment.
E
Run the model through NeMo Guardrails with a policy that blocks any output containing numbers.
Why it fails: Blocking outputs with numbers is an arbitrary rule that would not effectively detect hallucinations; many correct medical answers include numbers (e.g., dosages). This approach would introduce false positives and fail to assess factual accuracy. It does not address the core evaluation goals of correctness and hallucination detection.

Question 18hardmultiple choice

A research team is evaluating a large language model's ability to follow instructions. They have a dataset of prompts with corresponding reference outputs. They want to use an automated metric that correlates well with human judgments of instruction-following quality. Which evaluation method is most suitable?

Trap 1: BLEU score against reference outputs

BLEU measures n-gram precision and is designed for translation. It does not capture instruction-following quality, as it ignores semantics and can penalize valid paraphrases. A model could follow instructions perfectly but use different wording, resulting in low BLEU. Thus, BLEU correlates poorly with human judgments of instruction adherence.

Trap 2: Perplexity of the model on the reference outputs

Perplexity measures how well the model predicts the reference outputs, reflecting fluency. It does not evaluate whether the model's own generations follow instructions. A model could assign high probability to reference outputs but still fail to follow instructions when generating. Thus, perplexity is not appropriate for assessing instruction-following quality.

Trap 3: ROUGE-L score against reference outputs

ROUGE-L measures longest common subsequence and is used for summarization. It focuses on recall of content and does not assess whether the model followed specific instructions like format or constraints. It may reward copying reference text without actually following instructions. Therefore, it is not suitable for evaluating instruction-following.

A
BLEU score against reference outputs
Why it fails: BLEU measures n-gram precision and is designed for translation. It does not capture instruction-following quality, as it ignores semantics and can penalize valid paraphrases. A model could follow instructions perfectly but use different wording, resulting in low BLEU. Thus, BLEU correlates poorly with human judgments of instruction adherence.
B
GPT-4-based evaluation with a detailed rubric
Using a strong LLM like GPT-4 as a judge with a detailed rubric has been shown to correlate well with human judgments for instruction-following tasks. The rubric can specify criteria such as adherence to format, constraints, and correctness. This method captures nuanced aspects that n-gram metrics miss, making it the most suitable for this scenario.
C
Perplexity of the model on the reference outputs
Why it fails: Perplexity measures how well the model predicts the reference outputs, reflecting fluency. It does not evaluate whether the model's own generations follow instructions. A model could assign high probability to reference outputs but still fail to follow instructions when generating. Thus, perplexity is not appropriate for assessing instruction-following quality.
D
ROUGE-L score against reference outputs
Why it fails: ROUGE-L measures longest common subsequence and is used for summarization. It focuses on recall of content and does not assess whether the model followed specific instructions like format or constraints. It may reward copying reference text without actually following instructions. Therefore, it is not suitable for evaluating instruction-following.

Question 19hardmultiple choice

A research team is evaluating a large language model's robustness to adversarial attacks. They want to use NVIDIA NeMo Evaluator to measure how often the model's output changes when small, semantically preserving perturbations are applied to input prompts. Which evaluation metric or method should they implement?

Trap 1: Use exact match to compare the outputs from original and perturbed…

Exact match would only count identical strings, which is too strict; semantically equivalent outputs with different wording would be marked as inconsistent. This would underestimate robustness. A semantic similarity metric is needed to properly assess meaning preservation.

Trap 2: Measure the perplexity of the model on the perturbed prompts.

Perplexity measures how well the model predicts the perturbed input itself, not the consistency of its generated outputs. A model could have low perplexity on perturbed prompts yet produce different answers. Therefore, perplexity does not measure robustness as defined here.

Trap 3: Compute the BLEU score between outputs from original and perturbed…

BLEU measures n-gram overlap between a candidate and reference, not the consistency between two model outputs. Using BLEU to compare outputs from original and perturbed inputs would not directly quantify robustness; it would only show lexical similarity, which may be high even if the meaning changes. Thus, it is not the appropriate metric.

A
Use exact match to compare the outputs from original and perturbed prompts.
Why it fails: Exact match would only count identical strings, which is too strict; semantically equivalent outputs with different wording would be marked as inconsistent. This would underestimate robustness. A semantic similarity metric is needed to properly assess meaning preservation.
B
Measure the perplexity of the model on the perturbed prompts.
Why it fails: Perplexity measures how well the model predicts the perturbed input itself, not the consistency of its generated outputs. A model could have low perplexity on perturbed prompts yet produce different answers. Therefore, perplexity does not measure robustness as defined here.
C
Calculate the semantic similarity between the two outputs using an embedding-based metric like BERTScore.
Robustness to semantically preserving perturbations can be assessed by measuring how similar the model's outputs are. BERTScore captures semantic equivalence, so a high score indicates the model produced consistent meaning despite input changes. This directly quantifies robustness. NeMo Evaluator can integrate BERTScore as a custom metric to automate this comparison.
D
Compute the BLEU score between outputs from original and perturbed prompts.
Why it fails: BLEU measures n-gram overlap between a candidate and reference, not the consistency between two model outputs. Using BLEU to compare outputs from original and perturbed inputs would not directly quantify robustness; it would only show lexical similarity, which may be high even if the meaning changes. Thus, it is not the appropriate metric.

Question 20hardmultiple choice

A team is evaluating a large language model (LLM) deployed with NVIDIA Triton Inference Server for a real-time question-answering system. They observe that the model's responses are accurate but latency spikes during peak load, causing timeouts. They need to evaluate the model's performance under high concurrency to identify the maximum throughput while maintaining a 95th percentile latency below 200 ms. Which evaluation approach should they use?

Trap 1: Measure GPU utilization with NVIDIA Management Library (NVML)…

GPU utilization from NVML indicates how busy the GPU is but does not provide latency or throughput metrics under concurrent load. A single long-running inference does not simulate multiple simultaneous requests, so it cannot reveal latency spikes or maximum throughput. Thus, this method fails to evaluate the system's behavior under high concurrency.

Trap 2: Run a one-off inference with a batch size of 1 and measure latency.

A single inference with batch size 1 does not simulate concurrent load and will not reveal latency spikes under high concurrency. It only measures baseline latency without contention for GPU resources. Therefore, it cannot identify the maximum throughput that meets the latency SLA. This approach is inadequate for the scenario's requirement to evaluate under peak load.

Trap 3: Compute the model's FLOPS and compare against the GPU's peak…

FLOPS comparison gives a theoretical upper bound on compute but ignores memory bandwidth, kernel launch overhead, and concurrency effects. It does not measure actual latency or throughput under real workloads. Therefore, it cannot determine the maximum throughput that meets a specific latency SLA. This approach is too abstract for the scenario's operational evaluation.

A
Measure GPU utilization with NVIDIA Management Library (NVML) during a single long-running inference.
Why it fails: GPU utilization from NVML indicates how busy the GPU is but does not provide latency or throughput metrics under concurrent load. A single long-running inference does not simulate multiple simultaneous requests, so it cannot reveal latency spikes or maximum throughput. Thus, this method fails to evaluate the system's behavior under high concurrency.
B
Run a one-off inference with a batch size of 1 and measure latency.
Why it fails: A single inference with batch size 1 does not simulate concurrent load and will not reveal latency spikes under high concurrency. It only measures baseline latency without contention for GPU resources. Therefore, it cannot identify the maximum throughput that meets the latency SLA. This approach is inadequate for the scenario's requirement to evaluate under peak load.
C
Compute the model's FLOPS and compare against the GPU's peak theoretical FLOPS.
Why it fails: FLOPS comparison gives a theoretical upper bound on compute but ignores memory bandwidth, kernel launch overhead, and concurrency effects. It does not measure actual latency or throughput under real workloads. Therefore, it cannot determine the maximum throughput that meets a specific latency SLA. This approach is too abstract for the scenario's operational evaluation.
D
Use NVIDIA Triton's Performance Analyzer with concurrency sweep and latency constraints.
NVIDIA Triton's Performance Analyzer can simulate concurrent requests by sweeping concurrency levels and measuring latency percentiles. It supports setting a latency constraint (e.g., 95th percentile < 200 ms) and automatically finds the maximum throughput that satisfies it. This directly addresses the need to evaluate performance under high concurrency and identify the optimal operating point.