NCP-GENL LLM Architecture — 10 Questions

Question 1

An engineer is analyzing a 70B-parameter decoder-only LLM that uses Grouped Query Attention (GQA) with 8 key/value heads and 64 query heads. During inference with a long prompt, they notice that the KV cache memory is still a bottleneck on a single GPU, and they want to reduce it further without retraining the model from scratch. Which approach best describes a valid architectural modification that reduces KV cache size while preserving the model's learned attention behavior as much as possible?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Apply weight-only quantization to the key/value projection weights and store the KV cache in a lower-precision format such as FP8 or INT8.. The KV cache bottleneck can be addressed by reducing the precision of stored keys and values. Quantizing the KV cache to FP8 or INT8, along with weight-only quantization of the key/value projections, lowers memory usage substantially while keeping the GQA head configuration intact. This post-training approach avoids retraining and preserves the model's learned attention behavior, unlike changing the number of heads or pruning.

Answer

Replace GQA with Multi-Query Attention (MQA) by using a single key/value head for all query heads, without any additional training.

Answer

Reduce the model's hidden dimension by pruning entire attention heads across all layers and fine-tuning only the remaining heads.

Answer

Increase the number of key/value heads to match the number of query heads, converting GQA to multi-head attention.

Question 2

An engineer is deploying a 70B-parameter LLM for real-time chat. During load testing, the first token latency is acceptable, but subsequent tokens are generated at only 8 tokens per second, far below the target of 30. Profiling shows that the GPU memory bandwidth is saturated during decoding, and the KV cache is consuming over 60 GB. Which optimization is most likely to increase decoding throughput without retraining the model?

Accepted Answer

Switch from FP16 to INT8 quantization of the KV cache.. During autoregressive decoding, the model reads the entire KV cache for every generated token, making the process memory-bandwidth bound. Quantizing the KV cache to INT8 halves the bytes read per step, directly relieving the bottleneck and increasing tokens per second. The other options either increase memory pressure, require retraining, or add complexity without reducing the fundamental bandwidth cost of cache reads.

Answer

Apply structured pruning to remove 30% of the attention heads.

Answer

Increase the batch size to amortize the weight reads across more sequences.

Answer

Enable speculative decoding with a smaller draft model.

Question 3

An engineer inspects a decoder-only Transformer and notices that each attention sublayer is followed by a residual addition and then a normalization, and the same pattern repeats in the MLP sublayer. They ask why the residual connection is added before the normalization rather than after. What is the primary purpose of the residual connection in this Pre-LN arrangement?

Accepted Answer

It allows gradients to flow directly through the network, mitigating vanishing gradients in deep stacks.. Residual connections provide an identity path around each sublayer, letting gradients bypass nonlinear transformations and flow directly to earlier layers. In deep Transformer stacks this is essential to avoid vanishing gradients and to stabilize training. In a Pre-LN design the normalization is applied before the sublayer, so the residual stream stays unnormalized and serves as a clean information highway.

Answer

It reduces the KV cache by sharing key and value projections across layers.

Answer

It enforces causality by masking future tokens in the self-attention scores.

Answer

It projects the hidden state into the vocabulary space for next-token prediction.

Question 4

A developer is building a retrieval-augmented generation pipeline and needs embeddings whose dot products reflect semantic similarity rather than magnitude. They plan to use a decoder-only LLM's hidden states as embeddings. Which architectural detail should they account for to make the similarity scores meaningful?

Accepted Answer

Apply layer normalization to the final hidden states and then L2-normalize the vectors before computing dot products.. Hidden states from a decoder-only LLM differ in magnitude across tokens and layers, so a raw dot product conflates direction with scale. Normalizing the pooled vector and applying L2 normalization turns the dot product into cosine similarity, which captures semantic direction. This normalization step is what makes LLM-derived embeddings usable for retrieval similarity.

Answer

Average the attention weights across all heads and use that as the embedding.

Answer

Use the logits from the language modeling head as the embedding vector instead of hidden states.

Answer

Extract embeddings from the first layer only, since deeper layers overfit to the pretraining objective.

Question 5

A research team is training a decoder-only LLM and notices that gradients in the early layers are vanishing, causing slow convergence. They review their architecture and see that the residual connections are placed after the layer normalization in each sublayer, following a pre-norm design. They want to improve gradient flow without changing the model size. Which adjustment is most appropriate?

Accepted Answer

Keep the pre-norm placement and ensure the residual path is not normalized.. Pre-norm transformers place layer normalization inside each sublayer before the attention or feed-forward computation, while the residual connection adds the sublayer output to the unnormalized input. This unnormalized residual path acts as an identity highway that lets gradients reach early layers directly. Preserving this design is the correct way to maintain healthy gradient flow without altering model size.

Answer

Replace the residual connections with dense skip connections across all layers.

Answer

Move layer normalization to operate after the residual addition in each sublayer.

Answer

Apply layer normalization only to the final output of the transformer stack.

Question 6

An engineer is designing a Transformer-based model for long-context document summarization. They decide to replace the standard dense self-attention mechanism with a sliding window attention approach. What is the primary architectural implication of this change?

Accepted Answer

The computational complexity of the self-attention mechanism is reduced from quadratic to linear.. Sliding window attention restricts the receptive field of each token to a local neighborhood, drastically reducing the quadratic memory complexity of standard attention to linear. This is critical for scaling LLMs to long contexts, as it prevents the O(n²) memory growth that typically causes GPU out-of-memory errors on large input sequences while maintaining local coherence.

Answer

The model's total parameter count increases due to added positional embedding layers.

Answer

The model loses the ability to perform parallel training across multiple GPU nodes.

Answer

The model is no longer compatible with standard softmax normalization functions.

Question 7

Refer to the exhibit. An engineer is fine-tuning an LLM using the provided configuration. What is the primary purpose of applying 'yarn' scaling in this architecture?

Accepted Answer

It enables the model to process sequences longer than the original pre-trained limit.. YaRN (Yet another RoPE extension) is designed to extend the context window of pre-trained LLMs beyond their original training length. By modifying the Rotary Positional Embeddings (RoPE) base frequency, it allows the model to interpolate position indices without severe degradation in perplexity. This is essential for enterprise deployments requiring document retrieval or analysis tasks where input lengths exceed the base model's pre-trained constraints.

Answer

It increases the number of attention heads to improve feature extraction.

Answer

It compresses the model weights to reduce storage requirements.

Answer

It replaces the attention mechanism with a recurrent neural network.

Question 8

Which TWO of the following statements correctly describe the role of Grouped Query Attention (GQA) in modern LLM architectures?

Accepted Answer

GQA reduces the memory footprint of the KV cache during inference.. GQA balances the performance of Multi-Head Attention (MHA) and the memory efficiency of Multi-Query Attention (MQA). By sharing keys and values across groups of heads, it reduces the size of the KV cache during inference. This is crucial for high-throughput deployment environments, as it optimizes memory bandwidth utilization without sacrificing the granular representational capacity that individual query heads provide.

Answer

GQA eliminates the need for separate query, key, and value projection layers.

Answer

GQA improves model training speed by increasing the number of compute operations.

Answer

GQA prevents the model from using positional embeddings.

Question 9

In the context of the Transformer architecture, what is the primary function of the Feed-Forward Network (FFN) layers applied after the attention mechanism?

Accepted Answer

To process the hidden state representations with non-linear activations.. The FFN layers provide non-linear transformations that allow the model to process information extracted by the attention mechanism. While attention focuses on relationships between tokens, the FFN applies point-wise non-linearities to project these features into higher-dimensional spaces. This is essential for learning complex representations and mappings that ultimately drive the model's predictive accuracy and reasoning capabilities across diverse input patterns.

Answer

To calculate the attention scores between different tokens in the sequence.

Answer

To manage the memory allocation for the KV cache during multi-token generation.

Answer

To reduce the sequence length of the input tokens to a fixed size.

Question 10

Why do many modern LLMs use SwiGLU as their activation function in the feed-forward network instead of the traditional ReLU?

Accepted Answer

It provides better gradient propagation and higher model performance.. SwiGLU is a gated linear unit that incorporates the Swish activation, providing a smoother gradient flow and improved representational capacity over ReLU. ReLU's 'dying gradient' problem can hinder training progress, whereas SwiGLU's multiplicative gating allows the model to learn more flexible feature activation patterns. This is fundamental for stabilizing the training of very deep models and achieving state-of-the-art performance in complex linguistic tasks.

Answer

It reduces the number of parameters by half compared to ReLU.

Answer

It allows the model to perform faster matrix multiplications.

Answer

It makes the model compatible with 4-bit quantization.