LLM Architecture practice questions

Replace GQA with Multi-Query Attention (MQA) by using a single key/value head for all query heads, without any additional training.

Why it fails: MQA uses one key/value head, which drastically reduces KV cache size, but switching from GQA to MQA without training changes the attention computation and typically degrades quality. The model was trained with 8 key/value heads, so collapsing them to one without adaptation breaks learned behavior. This violates the requirement to preserve learned attention behavior and avoid retraining from scratch.

Reduce the model's hidden dimension by pruning entire attention heads across all layers and fine-tuning only the remaining heads.

Why it fails: Pruning entire attention heads changes the model's capacity and requires fine-tuning to recover quality, which is a form of retraining. It also reduces the number of query heads and can degrade the model's learned representations. While it may reduce KV cache size, it does not preserve learned attention behavior as well as lower-precision storage and is not a post-training-only modification.

Increase the number of key/value heads to match the number of query heads, converting GQA to multi-head attention.

Why it fails: Increasing key/value heads to match query heads converts the model to full multi-head attention, which increases KV cache size proportionally. This directly worsens the memory bottleneck rather than reducing it. It also changes the attention computation and would require retraining or fine-tuning to preserve quality, contradicting the goal of reducing cache without retraining from scratch.

Apply structured pruning to remove 30% of the attention heads.

Why it fails: Structured pruning removes parameters and typically requires fine-tuning to recover accuracy, which violates the no-retraining constraint. Even if applied post-hoc, pruning attention heads reduces model quality and may not proportionally reduce KV cache bandwidth because the cache size depends on head dimension and sequence length. This is a risky, quality-degrading change that does not cleanly solve the bandwidth bottleneck.

Increase the batch size to amortize the weight reads across more sequences.

Why it fails: Increasing batch size can improve GPU utilization during prefill but during decoding it increases the KV cache size proportionally, worsening the 60 GB memory pressure and potentially causing out-of-memory errors. It also does not reduce the per-token bandwidth cost of reading the KV cache, so the 8 tokens per second bottleneck would not be resolved. This option targets the wrong bottleneck.

Enable speculative decoding with a smaller draft model.

Why it fails: Speculative decoding can increase throughput by generating multiple tokens per verification step, but it does not reduce the KV cache size or the memory bandwidth consumed by the target model's cache reads. The profiling clearly points to bandwidth saturation from the cache, so a draft model adds complexity without addressing the root cause. It may even increase memory pressure if both models reside on the same GPU.

It reduces the KV cache by sharing key and value projections across layers.

Why it fails: Sharing key and value projections across layers is a specific efficiency technique, sometimes called cross-layer KV sharing, and is not what a residual connection does. A residual addition simply sums the sublayer input with its output. It has no effect on how many key/value tensors are cached or on cross-layer parameter sharing.

It enforces causality by masking future tokens in the self-attention scores.

Why it fails: Causality is enforced by adding a mask to the attention logits before the softmax, setting future positions to negative infinity. The residual connection is unrelated to masking; it operates on the sublayer output after attention has already been computed. Conflating the two would misrepresent how decoder-only models prevent tokens from attending to future positions.

It projects the hidden state into the vocabulary space for next-token prediction.

Why it fails: Projecting hidden states to vocabulary logits is the job of the final language modeling head, typically a linear layer tied to the embedding matrix. The residual connection inside each sublayer does not perform any vocabulary projection; it simply adds the sublayer input back to the sublayer output to preserve information and gradient flow through the network.

Average the attention weights across all heads and use that as the embedding.

Why it fails: Attention weights describe how tokens route information to each other and have shape sequence by sequence, not a fixed-dimensional vector per input. They are not a semantic embedding and averaging across heads does not produce a comparable representation. Similarity computed on attention maps would reflect token interaction patterns, not overall meaning.

Use the logits from the language modeling head as the embedding vector instead of hidden states.

Why it fails: Logits from the language modeling head are unnormalized scores over the vocabulary and are shaped by the output projection, so they do not represent a compact semantic embedding. Their dimensionality equals the vocabulary size, and their scale is arbitrary. Using them for similarity would reflect next-token prediction preferences rather than a stable semantic representation of the input.

Extract embeddings from the first layer only, since deeper layers overfit to the pretraining objective.

Why it fails: Deeper layers generally capture more abstract, task-relevant semantics, and there is no rule that they overfit the pretraining objective in a way that ruins embeddings. Using only the first layer would yield shallow, token-local representations. The real issue for similarity is vector scale, which normalization addresses, not the choice of layer alone.

Replace the residual connections with dense skip connections across all layers.

Why it fails: Dense skip connections, as in DenseNet, connect each layer to all subsequent layers, which dramatically increases memory and parameter usage. This changes the model size and architecture significantly, violating the constraint of not changing model size. It also complicates the computational graph and is not a standard remedy for vanishing gradients in transformer LLMs.

Move layer normalization to operate after the residual addition in each sublayer.

Why it fails: Post-norm places layer normalization after the residual addition, which was the original transformer design but is known to cause more unstable training and vanishing gradients in deep networks. Moving from pre-norm to post-norm would likely worsen the vanishing gradient problem rather than improve it. This change is the opposite of what is needed for stable deep training.

Apply layer normalization only to the final output of the transformer stack.

Why it fails: Removing per-sublayer normalization and applying it only at the end would leave intermediate activations unnormalized, causing severe scale drift and unstable training. Each sublayer's input distribution would vary widely, making optimization difficult and likely worsening vanishing gradients. This is not a recognized architectural pattern for deep transformer LLMs.

The model's total parameter count increases due to added positional embedding layers.

Why it fails: Sliding window attention does not require additional parameter layers; rather, it modifies the attention computation mask. Parameters are generally independent of the attention pattern, focusing instead on feature projections in the query, key, and value heads, meaning parameter count remains largely unaffected by the windowing mechanism.

The model loses the ability to perform parallel training across multiple GPU nodes.

Why it fails: Parallel training remains fully functional with sliding window attention. Since the attention mask is locally constrained, it can still be computed efficiently across distributed systems. The transformation of the attention matrix does not introduce serial dependencies that would prevent standard data or tensor parallelism during the training process.

The model is no longer compatible with standard softmax normalization functions.

Why it fails: Softmax remains the standard normalization function for calculating attention weights, regardless of the window size. The windowing approach simply applies a mask to the input of the softmax function to zero out distant tokens. The mathematical properties of softmax and its differentiability are preserved throughout the computation.

It increases the number of attention heads to improve feature extraction.

Why it fails: YaRN scaling is strictly a mechanism for modifying positional embeddings to support longer sequences; it does not alter the number of attention heads. Increasing attention heads is an architectural change that modifies the model's hidden dimensions and projection matrices, whereas RoPE scaling operates solely on the positional embedding logic.

It compresses the model weights to reduce storage requirements.

Why it fails: YaRN does not perform model compression or weight quantization. Weight compression techniques focus on reducing the precision of model parameters, such as moving from FP16 to INT8 or INT4. YaRN modifies how tokens are positioned relative to each other, which is an architectural interpolation strategy, not a storage optimization.

It replaces the attention mechanism with a recurrent neural network.

Why it fails: YaRN is specifically designed for Transformers that use RoPE, which is an attention-based positioning technique. It does not alter the underlying Transformer architecture to a recurrent structure. Replacing the attention mechanism with an RNN would fundamentally change the model's capacity to parallelize, which is unrelated to positional embedding scaling.

GQA eliminates the need for separate query, key, and value projection layers.

Why it fails: GQA still utilizes distinct projections for queries, keys, and values. It simply reduces the number of unique key and value heads relative to the number of query heads. The projection architecture remains intact, ensuring that the model retains its ability to attend to complex patterns across different subspaces.

GQA improves model training speed by increasing the number of compute operations.

Why it fails: GQA is designed for efficiency rather than increased compute; it actually reduces the number of operations required for key and value updates. The primary benefit is speed during inference due to memory bandwidth constraints, rather than an increase in computational volume during the training phase of the model.

GQA prevents the model from using positional embeddings.

Why it fails: GQA is entirely agnostic to the type of positional embedding used. It works perfectly alongside RoPE, ALiBi, or absolute positional embeddings. The grouping mechanism is applied at the attention head level, while positional information is injected into the queries and keys independently of how those heads are grouped.

To calculate the attention scores between different tokens in the sequence.

Why it fails: Attention scores are computed within the multi-head attention blocks, not the feed-forward network. The attention mechanism uses dot products of query and key vectors to determine relevance, whereas the FFN is a separate module focused on feature transformation and non-linear processing of the resulting hidden states.

To manage the memory allocation for the KV cache during multi-token generation.

Why it fails: Memory management is a system-level function performed by the inference engine (e.g., vLLM or TensorRT-LLM), not by the neural network's architecture layers. The FFN is a static part of the model weights and does not interact with the dynamic memory allocation strategies used for the KV cache.

To reduce the sequence length of the input tokens to a fixed size.

Why it fails: The FFN operates on each token independently and does not modify the sequence length. Sequence length reduction would require pooling layers or strided attention, which are not part of the standard FFN block. The FFN maintains the dimensionality and sequence length of its input throughout its transformation.

It reduces the number of parameters by half compared to ReLU.

Why it fails: SwiGLU does not reduce parameter count; in fact, it often introduces slightly more parameters due to the gating mechanism. The focus of SwiGLU is on accuracy and training stability, not on reducing the memory footprint or the parameter count of the model's feed-forward layers.

It allows the model to perform faster matrix multiplications.

Why it fails: The speed of matrix multiplication is determined by the hardware and the dimensions of the tensors, not the activation function. While SwiGLU requires an extra element-wise product, the difference in compute time is negligible compared to the benefits of improved convergence and model performance during the training phase.

It makes the model compatible with 4-bit quantization.

Why it fails: Quantization compatibility is independent of the activation function used in the FFN. Whether a model uses ReLU, GeLU, or SwiGLU, it can be quantized using techniques like GPTQ or AWQ. The choice of activation function is driven by model architecture design, not by the requirements of low-bit precision formats.

Batch Size reduction to increase memory throughput.

Why it fails: Reducing batch size decreases total system throughput, as it fails to leverage the massive parallel compute capabilities of modern GPUs. Efficient inference usually prioritizes increasing batch sizes or using dynamic batching to maximize hardware utilization, rather than reducing batch size, which would hurt efficiency and overall request handling.

Increasing the learning rate during inference.

Why it fails: Learning rate is a hyperparameter for training, not inference. Changing the learning rate during inference is nonsensical as weights are fixed. Inference relies on static weights to generate tokens, and attempting to adjust learning parameters would have no impact on the generative capability or speed of the model.

Increase the number of hidden layers.

Why it fails: Adding more hidden layers increases the model's overall memory requirement, as each layer contains additional weight matrices and cache space. This would worsen the OOM error rather than resolve it. Layer depth is generally fixed after training, and increasing it would not help with memory-bound sequence length constraints.

Reduce the embedding dimension size.

Why it fails: The embedding dimension is a fixed hyperparameter determined during the model's initial training. Modifying it post-training would require re-training the entire model, as the weight matrices would no longer align. It is not a viable strategy for resolving inference-time memory issues on an already trained model.

Disable the KV cache entirely.

Why it fails: Disabling the KV cache would make inference extremely slow, as the model would have to recompute keys and values for every new token. While it saves memory, it is not a practical solution for generative tasks. Furthermore, it does not solve the underlying issue of the massive attention computation.

It forces all parameters to update at every training step.

Why it fails: The router actually prevents all parameters from updating at once; only the active experts and the router weights are updated for a specific token. This sparsity is the primary advantage of MoE, as it allows training massive models efficiently by distributing the computational load across different parameter subsets.

It forces the model to use all experts simultaneously for every input.

Why it fails: If every expert were used for every input, the model would simply be a very large dense network, defeating the purpose of the MoE architecture. The router is designed to ensure sparsity, enabling efficient inference and training by only activating a small, specific set of experts for each individual token.

It replaces the attention mechanism with a standard linear layer.

Why it fails: MoE and attention are complementary; the attention mechanism is still required to handle sequence dependencies. The MoE structure is typically applied to the feed-forward layers within the Transformer blocks, not as a replacement for the attention mechanism, which is responsible for token-to-token relationship modeling.

It normalizes the entire batch to reduce training time.

Why it fails: Batch normalization is rarely used in NLP models because it depends on the statistics of the batch, which can vary significantly across sequence lengths. Layer normalization is preferred because it normalizes features within each individual token, maintaining consistency across sequences and avoiding dependencies on the broader training batch statistics.

It is used to perform dimensionality reduction on input embeddings.

Why it fails: Layer normalization performs statistical standardization, not dimensionality reduction. Reducing the dimension would imply discarding information or projecting it into a smaller space, whereas normalization preserves the original feature count while scaling the magnitude of the values to ensure the network remains stable during backpropagation.

It replaces the need for residual connections in the model.

Why it fails: Residual connections and normalization are complementary; they work together to enable the training of deep models. Residual connections allow gradients to flow through the network without disappearing, while normalization ensures these signals stay within a range where the model's non-linearities can effectively function and learn complex patterns.

RoPE completely removes the need for attention mechanisms.

Why it fails: RoPE is a technique specifically applied to improve the attention mechanism, not to replace it. It provides a way to inject spatial awareness into the dot-product calculations of the attention layer, ensuring the model understands order without sacrificing the fundamental ability of the Transformer to attend to input tokens.

RoPE reduces the parameter count of the embedding layer.

Why it fails: RoPE does not impact the parameter count of the token embedding layer, which is determined by vocabulary size and hidden state dimensionality. It is a mathematical transformation applied to the query and key projections, which is computationally efficient but doesn't change the size of the model's parameter footprint.

RoPE requires retraining from scratch if sequence length increases.

Why it fails: One of the key advantages of RoPE is that it facilitates length extension through interpolation, which does not require a full retraining of the model. By adjusting the RoPE base frequency, the model can adapt to longer contexts without needing a fresh training cycle on the longer data.

To filter out stop words from the input text.

Why it fails: Masking in Transformers is about temporal causality, not linguistic content filtering. Stop words are handled during the preprocessing or tokenization stages. The attention mask is a structural mechanism used to enforce the autoregressive nature of the generation process, which is unrelated to the semantics of the words being processed.

To increase the randomness of the model's predictions.

Why it fails: Randomness in model predictions is controlled by parameters like temperature, top-k, or top-p sampling during inference, not by the attention mask. The attention mask is a deterministic architectural constraint designed to enforce sequence ordering; it does not introduce or manage stochasticity in the model's output generation process.

To reduce the computation of the attention mechanism by 50%.

Why it fails: While masking does effectively compute fewer values, it is not an efficiency-focused optimization; it is a correctness-focused architectural requirement. The computation reduction is a side effect of ignoring future positions, but the primary goal is ensuring that the model does not violate the causal dependencies of language generation.

The Feed-Forward Network.

Why it fails: The FFN is a component within each Transformer layer, not the final projection layer. While it performs feature transformation, it does not map states to the vocabulary size. The FFN maps to the same hidden dimension as the input, whereas the LM head specifically maps to the vocabulary dimension.

The Softmax normalization layer.

Why it fails: Softmax is an activation function used to turn raw logits into probability scores. It does not perform the projection from hidden state to vocabulary space. The projection is done by a linear transformation layer, and the softmax is applied to the output of that layer to normalize the distribution.

The positional embedding layer.

Why it fails: The positional embedding layer is used at the beginning of the model to inject information about token order into the hidden states. It is not involved in the final prediction or vocabulary projection. Its role is solely to provide spatial context to the model at the input level.

Increase the number of attention heads while keeping the total hidden dimension constant, without changing the positional encoding.

Why it fails: Adding more heads with a fixed hidden dimension reduces the per-head dimension, which can change representational capacity but does not alter how position is encoded. The root cause described is positional extrapolation and entropy collapse, not head count. This change is unlikely to fix repetition on long sequences and may even degrade per-head expressiveness.

Apply Layer Normalization before the self-attention sublayer instead of after it, leaving the positional encoding unchanged.

Why it fails: Pre-LN versus post-LN affects gradient flow and training stability, but it does not change the positional encoding mechanism. The scenario attributes the repetition and attention entropy collapse to positional handling during long-sequence generation. Reordering normalization alone will not provide relative-position generalization or prevent entropy collapse when extrapolating beyond the pretraining length.

Add a sinusoidal positional encoding to the input embeddings and freeze those embeddings during training.

Why it fails: Sinusoidal encodings are fixed and added only at the input, so they do not provide the relative-position inductive bias inside each attention layer. Freezing input embeddings further limits the model's ability to adapt position representations to the code and language mixture. This does not address entropy collapse in deep attention layers or improve length generalization in the way the scenario requires.

A padding mask that ignores tokens added to equalize sequence lengths across a batch.

Why it fails: A padding mask only excludes padded positions from attention so they do not contribute to the output. It does not prevent a real token from attending to later real tokens, so it cannot enforce autoregressive causality in a decoder-only model.

A learned positional embedding added to the token embeddings at the input layer.

Why it fails: Positional embeddings inject order information into the token representations, but they do not restrict which positions can attend to which. Without a causal mask, a decoder-only model could still look ahead during training and fail to learn proper next-token prediction.

A residual connection that adds the attention output back to the input hidden state.

Why it fails: Residual connections help gradient flow and preserve information across layers, but they do not control attention visibility. They cannot prevent a token from attending to future positions, so they are unrelated to enforcing causal, left-to-right generation.

Replace the feed-forward activation with a gated variant to reduce activation memory.

Why it fails: Changing the activation can affect activation memory and compute, but it modifies the architecture and does not address the optimizer's stored moments. The scenario explicitly requires keeping the architecture unchanged, so this does not meet the requirement.

Reduce the number of attention heads while keeping the hidden size constant.

Why it fails: Altering the head count changes the architecture and can degrade model quality. It also does not shrink the optimizer state, which scales with the number of trainable parameters, not with how those parameters are organized into heads.

Increase the batch size proportionally to the number of GPUs so optimizer states are shared across more tokens.

Why it fails: Increasing batch size changes the optimization dynamics and memory profile but does not reduce the per-parameter optimizer state. Adam still maintains two moments per parameter, so memory per parameter is unchanged regardless of how many tokens are processed per step.