NCP-GENL LLM Architecture — Timed 20 Questions

Question 1

An engineer is analyzing a 70B-parameter decoder-only LLM that uses Grouped Query Attention (GQA) with 8 key/value heads and 64 query heads. During inference with a long prompt, they notice that the KV cache memory is still a bottleneck on a single GPU, and they want to reduce it further without retraining the model from scratch. Which approach best describes a valid architectural modification that reduces KV cache size while preserving the model's learned attention behavior as much as possible?

20 minute time limit — choose an answer to begin.

Accepted Answer

Apply weight-only quantization to the key/value projection weights and store the KV cache in a lower-precision format such as FP8 or INT8.. The KV cache bottleneck can be addressed by reducing the precision of stored keys and values. Quantizing the KV cache to FP8 or INT8, along with weight-only quantization of the key/value projections, lowers memory usage substantially while keeping the GQA head configuration intact. This post-training approach avoids retraining and preserves the model's learned attention behavior, unlike changing the number of heads or pruning.

Answer

Replace GQA with Multi-Query Attention (MQA) by using a single key/value head for all query heads, without any additional training.

Answer

Reduce the model's hidden dimension by pruning entire attention heads across all layers and fine-tuning only the remaining heads.

Answer

Increase the number of key/value heads to match the number of query heads, converting GQA to multi-head attention.

Question 2

An engineer is deploying a 70B-parameter LLM for real-time chat. During load testing, the first token latency is acceptable, but subsequent tokens are generated at only 8 tokens per second, far below the target of 30. Profiling shows that the GPU memory bandwidth is saturated during decoding, and the KV cache is consuming over 60 GB. Which optimization is most likely to increase decoding throughput without retraining the model?

Accepted Answer

Switch from FP16 to INT8 quantization of the KV cache.. During autoregressive decoding, the model reads the entire KV cache for every generated token, making the process memory-bandwidth bound. Quantizing the KV cache to INT8 halves the bytes read per step, directly relieving the bottleneck and increasing tokens per second. The other options either increase memory pressure, require retraining, or add complexity without reducing the fundamental bandwidth cost of cache reads.

Answer

Apply structured pruning to remove 30% of the attention heads.

Answer

Increase the batch size to amortize the weight reads across more sequences.

Answer

Enable speculative decoding with a smaller draft model.

Question 3

An engineer inspects a decoder-only Transformer and notices that each attention sublayer is followed by a residual addition and then a normalization, and the same pattern repeats in the MLP sublayer. They ask why the residual connection is added before the normalization rather than after. What is the primary purpose of the residual connection in this Pre-LN arrangement?

Accepted Answer

It allows gradients to flow directly through the network, mitigating vanishing gradients in deep stacks.. Residual connections provide an identity path around each sublayer, letting gradients bypass nonlinear transformations and flow directly to earlier layers. In deep Transformer stacks this is essential to avoid vanishing gradients and to stabilize training. In a Pre-LN design the normalization is applied before the sublayer, so the residual stream stays unnormalized and serves as a clean information highway.

Answer

It reduces the KV cache by sharing key and value projections across layers.

Answer

It enforces causality by masking future tokens in the self-attention scores.

Answer

It projects the hidden state into the vocabulary space for next-token prediction.

Question 4

A developer is building a retrieval-augmented generation pipeline and needs embeddings whose dot products reflect semantic similarity rather than magnitude. They plan to use a decoder-only LLM's hidden states as embeddings. Which architectural detail should they account for to make the similarity scores meaningful?

Accepted Answer

Apply layer normalization to the final hidden states and then L2-normalize the vectors before computing dot products.. Hidden states from a decoder-only LLM differ in magnitude across tokens and layers, so a raw dot product conflates direction with scale. Normalizing the pooled vector and applying L2 normalization turns the dot product into cosine similarity, which captures semantic direction. This normalization step is what makes LLM-derived embeddings usable for retrieval similarity.

Answer

Average the attention weights across all heads and use that as the embedding.

Answer

Use the logits from the language modeling head as the embedding vector instead of hidden states.

Answer

Extract embeddings from the first layer only, since deeper layers overfit to the pretraining objective.

Question 5

A research team is training a decoder-only LLM and notices that gradients in the early layers are vanishing, causing slow convergence. They review their architecture and see that the residual connections are placed after the layer normalization in each sublayer, following a pre-norm design. They want to improve gradient flow without changing the model size. Which adjustment is most appropriate?

Accepted Answer

Keep the pre-norm placement and ensure the residual path is not normalized.. Pre-norm transformers place layer normalization inside each sublayer before the attention or feed-forward computation, while the residual connection adds the sublayer output to the unnormalized input. This unnormalized residual path acts as an identity highway that lets gradients reach early layers directly. Preserving this design is the correct way to maintain healthy gradient flow without altering model size.

Answer

Replace the residual connections with dense skip connections across all layers.

Answer

Move layer normalization to operate after the residual addition in each sublayer.

Answer

Apply layer normalization only to the final output of the transformer stack.

Question 6

An engineer is designing a Transformer-based model for long-context document summarization. They decide to replace the standard dense self-attention mechanism with a sliding window attention approach. What is the primary architectural implication of this change?

Accepted Answer

The computational complexity of the self-attention mechanism is reduced from quadratic to linear.. Sliding window attention restricts the receptive field of each token to a local neighborhood, drastically reducing the quadratic memory complexity of standard attention to linear. This is critical for scaling LLMs to long contexts, as it prevents the O(n²) memory growth that typically causes GPU out-of-memory errors on large input sequences while maintaining local coherence.

Answer

The model's total parameter count increases due to added positional embedding layers.

Answer

The model loses the ability to perform parallel training across multiple GPU nodes.

Answer

The model is no longer compatible with standard softmax normalization functions.

Question 7

Refer to the exhibit. An engineer is fine-tuning an LLM using the provided configuration. What is the primary purpose of applying 'yarn' scaling in this architecture?

Accepted Answer

It enables the model to process sequences longer than the original pre-trained limit.. YaRN (Yet another RoPE extension) is designed to extend the context window of pre-trained LLMs beyond their original training length. By modifying the Rotary Positional Embeddings (RoPE) base frequency, it allows the model to interpolate position indices without severe degradation in perplexity. This is essential for enterprise deployments requiring document retrieval or analysis tasks where input lengths exceed the base model's pre-trained constraints.

Answer

It increases the number of attention heads to improve feature extraction.

Answer

It compresses the model weights to reduce storage requirements.

Answer

It replaces the attention mechanism with a recurrent neural network.

Question 8

Which TWO of the following statements correctly describe the role of Grouped Query Attention (GQA) in modern LLM architectures?

Accepted Answer

GQA reduces the memory footprint of the KV cache during inference.. GQA balances the performance of Multi-Head Attention (MHA) and the memory efficiency of Multi-Query Attention (MQA). By sharing keys and values across groups of heads, it reduces the size of the KV cache during inference. This is crucial for high-throughput deployment environments, as it optimizes memory bandwidth utilization without sacrificing the granular representational capacity that individual query heads provide.

Answer

GQA eliminates the need for separate query, key, and value projection layers.

Answer

GQA improves model training speed by increasing the number of compute operations.

Answer

GQA prevents the model from using positional embeddings.

Question 9

In the context of the Transformer architecture, what is the primary function of the Feed-Forward Network (FFN) layers applied after the attention mechanism?

Accepted Answer

To process the hidden state representations with non-linear activations.. The FFN layers provide non-linear transformations that allow the model to process information extracted by the attention mechanism. While attention focuses on relationships between tokens, the FFN applies point-wise non-linearities to project these features into higher-dimensional spaces. This is essential for learning complex representations and mappings that ultimately drive the model's predictive accuracy and reasoning capabilities across diverse input patterns.

Answer

To calculate the attention scores between different tokens in the sequence.

Answer

To manage the memory allocation for the KV cache during multi-token generation.

Answer

To reduce the sequence length of the input tokens to a fixed size.

Question 10

Why do many modern LLMs use SwiGLU as their activation function in the feed-forward network instead of the traditional ReLU?

Accepted Answer

It provides better gradient propagation and higher model performance.. SwiGLU is a gated linear unit that incorporates the Swish activation, providing a smoother gradient flow and improved representational capacity over ReLU. ReLU's 'dying gradient' problem can hinder training progress, whereas SwiGLU's multiplicative gating allows the model to learn more flexible feature activation patterns. This is fundamental for stabilizing the training of very deep models and achieving state-of-the-art performance in complex linguistic tasks.

Answer

It reduces the number of parameters by half compared to ReLU.

Answer

It allows the model to perform faster matrix multiplications.

Answer

It makes the model compatible with 4-bit quantization.

Question 11

Which THREE architectural features are essential for enabling efficient inference of massive LLMs on multi-GPU systems?

Accepted Answer

KV Caching to store previous sequence states.. Efficient inference at scale requires techniques that address memory constraints, compute latency, and communication overhead. Model parallelism, KV caching, and weight quantization are fundamental pillars that allow large models to fit within limited memory pools while maintaining high throughput. These techniques collectively ensure that the model remains responsive and cost-effective when serving large-scale requests in a production environment.

Answer

Batch Size reduction to increase memory throughput.

Answer

Increasing the learning rate during inference.

Question 12

Refer to the exhibit. The model is encountering an OOM error during long-context processing. Which architectural adjustment is most appropriate to resolve this while maintaining context length?

Accepted Answer

Switch to FlashAttention-2 kernels to optimize memory usage.. When sequence lengths scale to 128k, the attention matrix size grows quadratically, consuming massive memory. Implementing FlashAttention-2 or similar memory-efficient attention kernels is the industry-standard solution. These kernels optimize the memory layout and tile operations to compute attention without materializing the massive N×N matrix, drastically reducing peak memory usage and enabling the processing of very long sequences within existing GPU capacity.

Answer

Increase the number of hidden layers.

Answer

Reduce the embedding dimension size.

Answer

Disable the KV cache entirely.

Question 13

In Mixture-of-Experts (MoE) architectures, why does the use of a router mechanism significantly impact performance compared to dense models?

Accepted Answer

It allows scaling parameters while keeping compute costs manageable.. The router determines which expert blocks to activate for a given input token, ensuring that only a subset of the model's total parameters are used per forward pass. This decoupling of model size from computation latency allows for the creation of massive, high-capacity models that run with the speed of much smaller dense models. This architectural choice is central to modern scaling laws.

Answer

It forces all parameters to update at every training step.

Answer

It forces the model to use all experts simultaneously for every input.

Answer

It replaces the attention mechanism with a standard linear layer.

Question 14

What is the architectural role of Layer Normalization in a Transformer, and where is it typically placed to ensure stable training?

Accepted Answer

It is placed before each sub-layer (Pre-LN) to improve training stability.. Layer normalization stabilizes the hidden states by ensuring their mean and variance remain within a controlled range throughout the network layers. In modern LLMs, placing it before the attention and FFN blocks (Pre-LN) is preferred over the original Post-LN. This prevents gradient explosion early in training, allowing for higher learning rates and more reliable convergence for deep models during their pre-training phase.

Answer

It normalizes the entire batch to reduce training time.

Answer

It is used to perform dimensionality reduction on input embeddings.

Answer

It replaces the need for residual connections in the model.

Question 15

Which TWO of the following are benefits of using Rotary Positional Embeddings (RoPE) compared to absolute positional embeddings?

Accepted Answer

RoPE allows for better extrapolation to unseen sequence lengths.. RoPE encodes relative position information by rotating vectors in complex space, which allows for better generalization to sequence lengths not seen during training. This relative approach is significantly more effective than absolute embeddings, which assign a fixed position to every index, as it allows the model to interpret the relationships between tokens regardless of their exact position in the sequence.

Answer

RoPE completely removes the need for attention mechanisms.

Answer

RoPE reduces the parameter count of the embedding layer.

Answer

RoPE requires retraining from scratch if sequence length increases.

Question 16

What is the function of the 'Masked' component in a Decoder-only Transformer's self-attention during training?

Accepted Answer

To prevent the model from attending to future tokens in the sequence.. The mask ensures that each token can only attend to itself and the tokens that precede it in the sequence. This autoregressive property is essential for training decoder models, as it prevents the model from 'cheating' by looking at future tokens. This ensures that the model learns to predict the next word based solely on the context that would be available during actual inference.

Answer

To filter out stop words from the input text.

Answer

To increase the randomness of the model's predictions.

Answer

To reduce the computation of the attention mechanism by 50%.

Question 17

Which architectural component is responsible for projecting the model's hidden states back into the vocabulary space to predict the next token?

Accepted Answer

The LM Head linear layer.. The language model head (LM Head) is a final linear projection layer that maps the high-dimensional hidden states from the Transformer blocks into a vector representing the probability distribution over the entire vocabulary. This projection is the final step in the forward pass, converting the learned internal representation into actionable predictions that can be sampled to generate text.

Answer

The Feed-Forward Network.

Answer

The Softmax normalization layer.

Answer

The positional embedding layer.

Question 18

A team is pretraining a decoder-only Transformer LLM on a large corpus of code and natural language. They observe that the model's training loss decreases smoothly, but during generation it sometimes produces degenerate repetition, and attention entropy on long sequences collapses. They suspect the issue is related to the positional encoding scheme. Which architectural change is most likely to mitigate the attention entropy collapse while preserving the model's ability to generalize to sequences longer than those seen during pretraining?

Accepted Answer

Replace learned absolute positional embeddings with Rotary Positional Embeddings (RoPE) applied to queries and keys in every self-attention layer.. The symptoms point to a positional encoding that does not generalize beyond the pretraining sequence length. Rotary Positional Embeddings inject relative position directly into the attention computation by rotating queries and keys, so attention scores depend on relative offsets rather than absolute indices. This preserves attention entropy on longer sequences and reduces degenerate repetition, making it the appropriate architectural change for this scenario.

Answer

Increase the number of attention heads while keeping the total hidden dimension constant, without changing the positional encoding.

Answer

Apply Layer Normalization before the self-attention sublayer instead of after it, leaving the positional encoding unchanged.

Answer

Add a sinusoidal positional encoding to the input embeddings and freeze those embeddings during training.

Question 19

A developer is building a decoder-only generative model and wants to prevent the model from attending to future tokens during training so that each position can only use information from itself and earlier positions. Which architectural mechanism should they implement in the self-attention layer?

Accepted Answer

A causal attention mask that sets attention scores for future positions to negative infinity before the softmax.. Decoder-only generative models must not see future tokens during training, otherwise next-token prediction becomes trivial and the model will not generalize to autoregressive inference. A causal attention mask accomplishes this by making future positions invisible to the softmax, ensuring each position only conditions on itself and earlier tokens.

Answer

A padding mask that ignores tokens added to equalize sequence lengths across a batch.

Answer

A learned positional embedding added to the token embeddings at the input layer.

Answer

A residual connection that adds the attention output back to the input hidden state.

Question 20

A team is training a large language model and wants to reduce the memory used by the optimizer without changing the model architecture. They are using Adam and notice that optimizer state consumes more GPU memory than the model weights. Which technique should they apply to reduce optimizer memory while keeping the model architecture unchanged?

Accepted Answer

Switch to a memory-efficient optimizer such as 8-bit Adam or Adafactor that stores reduced-precision or factored optimizer state.. Adam maintains two full-precision moment estimates for every trainable parameter, so optimizer state can be several times the size of the model weights. Memory-efficient optimizers such as 8-bit Adam or Adafactor reduce this footprint by quantizing or factoring the state, directly lowering the dominant memory cost without changing the model architecture.

Answer

Replace the feed-forward activation with a gated variant to reduce activation memory.

Answer

Reduce the number of attention heads while keeping the hidden size constant.

Answer

Increase the batch size proportionally to the number of GPUs so optimizer states are shared across more tokens.