CCNA Ncp Evaluation Questions

BERTScore aligns tokens between candidate and reference using contextual embeddings, computing precision, recall and F1 over cosine similarity. This satisfies the stem's requirement for embedding-based semantic assessment that captures meaning despite surface-level phrasing variation, without human annotators.

BLEU (Bilingual Evaluation Understudy) is designed for machine translation evaluation. It computes n-gram precision between generated and reference translations, with a brevity penalty. It is the standard metric for translation quality and is directly applicable when a reference translation exists, making it the correct choice for this scenario.

Robustness to semantically preserving perturbations can be assessed by measuring how similar the model's outputs are. BERTScore captures semantic equivalence, so a high score indicates the model produced consistent meaning despite input changes. This directly quantifies robustness. NeMo Evaluator can integrate BERTScore as a custom metric to automate this comparison.

A baseline is only useful if it is reproducible and comparable to later runs. Fixing the prompt template, decoding parameters, and dataset split means any future change in accuracy can be attributed to training rather than to evaluation drift. Recording the full configuration lets reviewers reproduce the number and detect accidental leakage into the fine-tuning data.

A factual consistency score uses an NLI model to check whether the generated answer is entailed by a trusted knowledge source (e.g., a reference document). This directly measures factual correctness, the second required criterion. It is robust to paraphrasing and can be automated within NeMo Evaluation to flag hallucinations or contradictions.

Average response length directly measures the mean number of tokens or words in generated responses. Since the issue is verbosity, this metric quantifies the problem precisely. It can be computed easily within NeMo Evaluation and compared against a target threshold to guide fine-tuning adjustments.

Using a strong LLM like GPT-4 as a judge with a detailed rubric has been shown to correlate well with human judgments for instruction-following tasks. The rubric can specify criteria such as adherence to format, constraints, and correctness. This method captures nuanced aspects that n-gram metrics miss, making it the most suitable for this scenario.

NLI models determine whether a hypothesis (generated answer) is entailed by, neutral to, or contradicts a premise (retrieved document). By running an NLI model, you can flag contradictions directly. This approach is robust for RAG evaluation because it assesses factual consistency without requiring reference answers, aligning with the goal of detecting contradictions.

Distinct-n measures the number of unique n-grams divided by the total number of n-grams in the generated text. It is specifically designed to quantify lexical diversity and repetition. For evaluating headline diversity, Distinct-n provides a direct measure of how varied the word choices are, making it the appropriate metric.

A same-family judge can favor outputs that resemble its own style, and pairwise judges are known to prefer whichever answer appears first. Swapping presentation order removes position bias, while a human-labeled subset quantifies how often the judge agrees with people. Only after those checks does a 62-win margin carry real evidentiary weight.

ROUGE-1 recall measures the proportion of unigram overlaps between the generated summary and the reference summary, relative to the reference. Low recall indicates missing content. Since the issue is omission of key information, recall is the appropriate metric to quantify missing content, as it directly reflects how much of the reference is captured.

pass@k measures the percentage of problems for which at least one of k generated samples passes all unit tests. It directly evaluates functional correctness by executing the code against test cases. This metric is standard for code generation tasks and aligns with the scenario's goal of measuring how many generated functions pass unit tests. It accounts for the stochastic nature of LLM outputs by considering multiple samples.

Factual consistency metrics like SummaC or Q2 evaluate whether generated summaries are entailed by the source document, detecting hallucinations. They are designed to catch fabricated facts by comparing summary claims against source content. In this healthcare scenario, applying such a metric directly addresses the need to ensure no invented medical facts, making it the most suitable approach.

The symptom points to a capability regression outside the fine-tuning domain, so the evaluation must cover format validity and previously working skills, not just the target task. Running the same structured-output prompts against the base model establishes whether the fine-tune caused the breakage. A format checker converts the vague observation into a measurable pass rate.

NVIDIA NeMo Guardrails is a toolkit for adding programmable guardrails to LLM-based conversational systems. It can be configured to detect and block toxic or biased outputs using predefined or custom policies. It integrates with models to enforce safety guidelines in real time, making it the appropriate tool for automatically evaluating and mitigating toxicity and bias in chatbot responses.

NVIDIA Triton's Performance Analyzer can simulate concurrent requests by sweeping concurrency levels and measuring latency percentiles. It supports setting a latency constraint (e.g., 95th percentile < 200 ms) and automatically finds the maximum throughput that satisfies it. This directly addresses the need to evaluate performance under high concurrency and identify the optimal operating point.

ROUGE-L measures n-gram overlap with a single reference, so it penalizes valid paraphrases and rewards surface copying. A 0.03 deficit is well inside that noise floor, meaning the metric cannot resolve the question. Because the goal is domain behavior rather than lexical matching, pairwise preference judging with human verification is the appropriate instrument here.

NeMo Guardrails allows defining programmable rules (e.g., using Colang) to detect and block harmful or biased content. By applying these guardrails to model outputs and quantifying violation rates, the team can systematically evaluate safety and bias. This approach integrates with NeMo Evaluator to log and analyze flagged instances, providing a quantifiable metric for compliance.

BERTScore leverages contextual embeddings to compute semantic similarity between generated and reference texts, aligning with the requirement to assess meaning rather than surface form. NeMo Evaluator supports custom metrics, allowing integration of BERTScore. This approach handles paraphrases well and is efficient for 500 prompts, meeting the low-latency evaluation goal.

A fact-checking module can verify each claim in the model's output against a curated medical knowledge base, directly detecting hallucinations or unsupported facts. This approach provides a granular, evidence-based assessment of factual accuracy, which is critical in healthcare. Integrating such a module with NeMo Evaluator allows automated flagging of unsupported statements.

A frozen, genuinely unseen set prevents leakage and version-to-version drift in the test data itself. If prompts were recycled during prompt engineering or fine-tuning, scores inflate because the model has effectively seen the answers. Keeping the set immutable also makes run-over-run deltas interpretable, since any change in score can be attributed to the model rather than to a shifting benchmark.

NLI models determine whether a hypothesis (the summary) is entailed by a premise (the source note). If the summary contains fabricated information, the NLI model would predict contradiction or neutral rather than entailment. This approach directly assesses factual consistency and is widely used for hallucination detection in summarization, making it the most appropriate method here.

Semantic similarity measures how closely the model's responses align in meaning when the same question is asked with different wording. High similarity indicates consistent understanding and stable generation. In this scenario, computing similarity across paraphrased loan queries directly quantifies the observed inconsistency. This metric is appropriate for detecting and reducing output variance.