LLM Fundamentals practice questions

Encoder-only is trained for text generation; decoder-only is trained for classification

Why wrong: Typical training objectives are reversed.

Both use the same attention pattern but differ in number of layers

Why wrong: They use fundamentally different attention patterns.

Encoder-only uses causal attention; decoder-only uses bidirectional attention

Why wrong: That’s the opposite of the correct description.

BLEU

Why wrong: BLEU is precision-oriented and does not specifically focus on recall.

Perplexity

Why wrong: Perplexity measures language model fluency, not factual recall.

Cosine similarity

Why wrong: Cosine similarity measures semantic closeness but not recall of specific phrases.

It produces fixed-length token sequences

Why wrong: Token sequences vary in length.

It reduces the number of tokens by merging all letters into single tokens

Why wrong: BPE merges frequent character pairs, but does not merge all letters.

It eliminates the need for a vocabulary altogether

Why wrong: BPE still requires a vocabulary of subword units.

Top-k sampling with k=50

Why wrong: Top-k introduces randomness by sampling from the top 50 tokens.

Temperature sampling with temperature=0.8

Why wrong: Temperature > 0 flattens the distribution, adding randomness.

Top-p (nucleus) sampling with p=0.9

Why wrong: Top-p samples from a cumulative probability mass, introducing randomness.

WordPiece tokenizer

Why wrong: WordPiece may split code into more tokens due to character-level merging.

SentencePiece tokenizer with unigram LM

Why wrong: Unigram tokenizer may not be as efficient for code as BPE.

Character-level tokenization

Why wrong: Character-level yields many tokens per code snippet.

High cost of training

Why wrong: RAG does not reduce training cost.

Slow inference speed

Why wrong: RAG may even add latency due to retrieval.

Lack of multilingual support

Why wrong: RAG does not directly address multilingual capability.

Pre-training uses labeled data; fine-tuning uses unlabeled data

Why wrong: Pre-training typically uses unlabeled data; fine-tuning uses labeled data.

Pre-training requires a GPU; fine-tuning can be done on CPU

Why wrong: Both typically require accelerators.

Pre-training is only for encoder models; fine-tuning is only for decoder models

Why wrong: Both encoder and decoder models can be pre-trained and fine-tuned.

Perplexity

Why wrong: Perplexity measures language model fit, not summary quality.

BLEU

Why wrong: BLEU is primarily for machine translation.

F1 score

Why wrong: F1 is for classification, not summarization.

To assign positional encodings to each word

Why wrong: Positional encoding is separate from tokenization.

To convert tokens into dense vector representations

Why wrong: That is the role of the embedding layer, not tokenization.

To remove stop words and punctuation from the input

Why wrong: Tokenization does not remove words; it splits text. Stop word removal is a separate preprocessing step.

Embedding models cannot be used with RAG; generation models can

Why wrong: Embedding models are key components in RAG for retrieval.

Embedding models are used only for classification; generation models are used only for translation

Why wrong: Embeddings have many uses beyond classification; generation models are used for many tasks beyond translation.

Embedding models require fine-tuning; generation models do not

Why wrong: Both types of models can be used zero-shot or fine-tuned.

SentencePiece

Why wrong: SentencePiece is used by models like T5 and ALBERT, not by BERT or GPT-2.

WordPiece

Why wrong: WordPiece is used by BERT, but not by GPT-2; since the question includes GPT-2, BPE is the correct choice.

Unigram Language Model

Why wrong: Unigram Language Model is used by models like XLNet, not by BERT or GPT-2.

BLEU

Why wrong: BLEU is precision-oriented and more suited for translation tasks.

Perplexity

Why wrong: Perplexity measures model confidence, not quality of summary content.

BERTScore

Why wrong: BERTScore is semantic and uses embeddings, not n-gram overlap.

The context window is too short

Why wrong: Context window affects input length, not knowledge of recent events.

The sampling strategy is too greedy

Why wrong: Greedy decoding does not cause factual errors about recent events.

The fine-tuning dataset was too small

Why wrong: Small dataset may cause overfitting but not systematic missing of recent events.

Feed-forward neural network

Why wrong: Feed-forward networks process each position independently and do not model interactions between words.

Positional encoding

Why wrong: Positional encoding adds information about the position of tokens, but does not weigh importance.

Layer normalization

Why wrong: Layer normalization stabilizes training but does not weigh word importance.

Increase temperature to 1.2 and use top-k=50

Why wrong: Higher temperature flattens the distribution, increasing risk of sampling low-probability tokens; top-k=50 may still include improbable tokens.

Use beam search with width=5

Why wrong: Beam search finds high-probability sequences but can still produce invalid code if the probability distribution does not penalize syntax errors.

Set temperature=1.0 and use greedy decoding

Why wrong: Greedy decoding always picks the token with highest probability, which is deterministic but temperature=1.0 is irrelevant with greedy; still may not fix syntax if high-probability tokens are incorrect.

To compress the input sequence length

Why wrong: Positional encoding does not compress sequences; it adds information.

To increase the dimensionality of the hidden states

Why wrong: While added to embeddings, the primary purpose is not dimensionality increase.

To reduce the effect of vanishing gradients

Why wrong: Vanishing gradients are addressed by residual connections and layer normalization, not positional encoding.

Model optimized for text summarization tasks

Why wrong: Summarization optimization is irrelevant for code generation.

Model with the highest number of parameters regardless of performance

Why wrong: Parameter count alone does not guarantee performance; larger models are also more expensive.

Positional encoding

Why wrong: Positional encoding injects information about token order, not attention weights.

Layer normalization

Why wrong: Layer normalization stabilizes training but does not weigh token importance.

Feed-forward network

Why wrong: Feed-forward networks process each token independently without cross-token interaction.

The embedding model should be a generative model like GPT

Why wrong: Generative models output text, not dense vectors optimized for similarity; embedding models are specialized for representation.

The embedding model should be fine-tuned on a classification task

Why wrong: Fine-tuning on classification may not improve semantic similarity; embedding models are often trained with contrastive learning.

The embedding model should have a large context window to encode long reviews

Why wrong: While context window matters, the primary requirement is that the embeddings capture semantic meaning; a large context is secondary.

Beam search

Why wrong: Beam search maintains multiple candidate sequences, not deterministic single-token selection.

Temperature sampling

Why wrong: Temperature sampling adjusts the probability distribution and samples stochastically.

Top-k sampling

Why wrong: Top-k sampling restricts to the k most probable tokens and then samples from them.