CCNA LLM Architecture Questions

SwiGLU is a gated linear unit that incorporates the Swish activation, providing a smoother gradient flow and improved representational capacity over ReLU. ReLU's 'dying gradient' problem can hinder training progress, whereas SwiGLU's multiplicative gating allows the model to learn more flexible feature activation patterns. This is fundamental for stabilizing the training of very deep models and achieving state-of-the-art performance in complex linguistic tasks.

Post-layer-normalization Transformers suffer from residual stream variance that grows with depth, producing large gradients early in training. Scaling residual branch outputs at initialization by a depth-dependent factor keeps variance bounded, directly stabilizing optimization without changing normalization placement or the optimizer schedule. Optimizer tweaks and activation swaps do not address the architectural root cause.

Layer normalization stabilizes the hidden states by ensuring their mean and variance remain within a controlled range throughout the network layers. In modern LLMs, placing it before the attention and FFN blocks (Pre-LN) is preferred over the original Post-LN. This prevents gradient explosion early in training, allowing for higher learning rates and more reliable convergence for deep models during their pre-training phase.

Autoregressive language models require a causal mask that sets attention scores to negative infinity for positions after the current token. When that mask is absent, the training objective becomes trivial because the target token is visible in the input context, producing the fast loss drop and incoherent generation. Positional encodings, dropout, and normalization placement do not enforce temporal directionality.

When many requests share a long system prompt, recomputing attention over that prefix for every request wastes prefill compute. Prefix caching persists the key/value tensors for the shared prefix and reuses them, cutting redundant work and improving throughput under concurrency. Speculative decoding targets decode latency, beam search adds work, and KV cache quantization saves memory but not duplicated prefill computation.

Causal masking is the standard mechanism that makes decoder-only attention autoregressive. By adding negative infinity to scores of future positions before softmax, those positions receive zero weight, so each token can only attend to itself and prior tokens. Other listed techniques affect numerical stability or positional encoding but do not enforce the temporal constraint.

The router determines which expert blocks to activate for a given input token, ensuring that only a subset of the model's total parameters are used per forward pass. This decoupling of model size from computation latency allows for the creation of massive, high-capacity models that run with the speed of much smaller dense models. This architectural choice is central to modern scaling laws.

Long-context retrieval failures in decoder-only LLMs typically stem from two sources: pretraining on sequences much shorter than the target context, which prevents the model from learning to attend across the full window, and positional encoding schemes that do not generalize beyond training lengths. Together these cause the model to underuse information at the start of a long document. The causal mask, feed-forward network, and vocabulary size are not primary causes of this behavior.

Dense retrieval requires a model that maps each passage to a single vector such that semantically similar passages are close in the embedding space. Bidirectional encoders trained with contrastive objectives are purpose-built for this, producing high-quality passage embeddings efficiently. Generative LLMs, cross-encoders, and summarization models are designed for other tasks and do not provide the required independent passage vectors.

Adam maintains two full-precision moment estimates for every trainable parameter, so optimizer state can be several times the size of the model weights. Memory-efficient optimizers such as 8-bit Adam or Adafactor reduce this footprint by quantizing or factoring the state, directly lowering the dominant memory cost without changing the model architecture.

RoPE encodes relative position information by rotating vectors in complex space, which allows for better generalization to sequence lengths not seen during training. This relative approach is significantly more effective than absolute embeddings, which assign a fixed position to every index, as it allows the model to interpret the relationships between tokens regardless of their exact position in the sequence.

Underfitting in a large decoder-only LLM indicates the model lacks sufficient capacity for the data distribution. Scaling both depth and width increases the parameter count and the number of nonlinear transformations applied to the residual stream, which improves the model's ability to capture long-range dependencies. The other options either rebalance existing capacity, change positional encoding without adding parameters, or reduce computation without improving fit.

Sparse MoE replaces a single feed-forward block with many expert blocks plus a router that, for each token, selects the top-k experts to run. This yields token-level dynamic routing and keeps per-token FLOPs tied to k and expert size rather than the total expert population. Dense averaging, static assignment, and automatic k growth all contradict the sparse, learned, token-specific routing that defines the architecture.

Parameter-efficient fine-tuning methods such as LoRA freeze the pretrained weights and train only small injected matrices, which reduces optimizer memory and limits drift from the base model. This is well suited to small domain datasets where full fine-tuning would overfit and degrade general language ability.

The symptoms point to a positional encoding that does not generalize beyond the pretraining sequence length. Rotary Positional Embeddings inject relative position directly into the attention computation by rotating queries and keys, so attention scores depend on relative offsets rather than absolute indices. This preserves attention entropy on longer sequences and reduces degenerate repetition, making it the appropriate architectural change for this scenario.

GQA balances the performance of Multi-Head Attention (MHA) and the memory efficiency of Multi-Query Attention (MQA). By sharing keys and values across groups of heads, it reduces the size of the KV cache during inference. This is crucial for high-throughput deployment environments, as it optimizes memory bandwidth utilization without sacrificing the granular representational capacity that individual query heads provide.

YaRN (Yet another RoPE extension) is designed to extend the context window of pre-trained LLMs beyond their original training length. By modifying the Rotary Positional Embeddings (RoPE) base frequency, it allows the model to interpolate position indices without severe degradation in perplexity. This is essential for enterprise deployments requiring document retrieval or analysis tasks where input lengths exceed the base model's pre-trained constraints.

When sequence lengths scale to 128k, the attention matrix size grows quadratically, consuming massive memory. Implementing FlashAttention-2 or similar memory-efficient attention kernels is the industry-standard solution. These kernels optimize the memory layout and tile operations to compute attention without materializing the massive N×N matrix, drastically reducing peak memory usage and enabling the processing of very long sequences within existing GPU capacity.

Position interpolation and YaRN both operate on positional encoding to reconcile desired sequence lengths with the range seen during pretraining. Interpolation compresses indices, while YaRN adjusts rotary frequencies to improve extrapolation. Both are context-extension methods that require only light fine-tuning, unlike changes to head count, MLP activation, or vocabulary size, which do not address positional generalization.

The language model head (LM Head) is a final linear projection layer that maps the high-dimensional hidden states from the Transformer blocks into a vector representing the probability distribution over the entire vocabulary. This projection is the final step in the forward pass, converting the learned internal representation into actionable predictions that can be sampled to generate text.

Grouped Query Attention reduces the number of key/value heads relative to query heads, so the KV cache that must be stored for each token shrinks by the grouping factor. Because the cache dominates HBM during long-context decoding, this architectural change directly relieves memory pressure and permits larger batches, and it can be adopted via uptraining rather than a full retrain.

Sparse MoE activates only a few experts per token, so FLOPs are low, but every expert's weights must remain resident and be routed to, often across devices. Inference becomes bound by memory bandwidth and all-to-all communication rather than compute, so a larger total parameter count does not yield proportional latency gains and can even hurt if routing is imbalanced.

Autoregressive decoding generates one token at a time, and without caching the model would recompute keys and values for the entire prefix at every step, wasting compute. A key-value cache stores those projections so each step only processes the new token, which is the standard way to make long-context generation practical.

The FFN layers provide non-linear transformations that allow the model to process information extracted by the attention mechanism. While attention focuses on relationships between tokens, the FFN applies point-wise non-linearities to project these features into higher-dimensional spaces. This is essential for learning complex representations and mappings that ultimately drive the model's predictive accuracy and reasoning capabilities across diverse input patterns.

Sliding window attention restricts the receptive field of each token to a local neighborhood, drastically reducing the quadratic memory complexity of standard attention to linear. This is critical for scaling LLMs to long contexts, as it prevents the O(n²) memory growth that typically causes GPU out-of-memory errors on large input sequences while maintaining local coherence.

The bottleneck is static weight storage on a single 40GB device. Weight-only INT8 quantization halves the parameter footprint while preserving the architecture and offering strong quality retention, and TensorRT-LLM provides a supported path for this on NVIDIA hardware. The other choices target attention compute, KV cache paging, or multi-GPU partitioning, none of which reduce the resident weight memory on one GPU.

Post-LN places normalization inside the residual branch, so gradients must pass through normalization at every layer, attenuating them in deep stacks. Pre-LN normalizes the sublayer input and leaves the residual stream unnormalized, giving a direct gradient highway from the loss to early parameters. That structural change, plus a final normalization before the output head, is the recognized fix for the described symptom.

Sinusoidal absolute positional embeddings are deterministic functions of position, adding no parameters and allowing evaluation at any index. Learned absolute embeddings allocate a trainable vector per position, consuming parameters and capping usable length at the trained maximum. The other options either reverse the extrapolation behavior or misattribute relative-offset abilities to learned absolute embeddings.

Efficient inference at scale requires techniques that address memory constraints, compute latency, and communication overhead. Model parallelism, KV caching, and weight quantization are fundamental pillars that allow large models to fit within limited memory pools while maintaining high throughput. These techniques collectively ensure that the model remains responsive and cost-effective when serving large-scale requests in a production environment.

The mask ensures that each token can only attend to itself and the tokens that precede it in the sequence. This autoregressive property is essential for training decoder models, as it prevents the model from 'cheating' by looking at future tokens. This ensures that the model learns to predict the next word based solely on the context that would be available during actual inference.

Decoder-only generative models must not see future tokens during training, otherwise next-token prediction becomes trivial and the model will not generalize to autoregressive inference. A causal attention mask accomplishes this by making future positions invisible to the softmax, ensuring each position only conditions on itself and earlier tokens.