NCP-GENL LLM Architecture Practice Question

Answer analysis

Option-by-option breakdown

For each option: why learners choose it and why it is or isn't the right answer here.

• ✗

  The feed-forward network in each layer compresses the hidden state, discarding information from early tokens.

  Why it's wrong here

  The feed-forward network operates on each position independently and does not selectively discard early-token information. While it transforms representations, the residual stream preserves information across layers. The bottleneck for long-context retrieval is attention and positional encoding, not feed-forward compression, which is not a recognized cause of this failure mode.

• ✗

  The vocabulary size is too small to represent the document's domain-specific terms, causing tokenization errors.

  Why it's wrong here

  Vocabulary size affects tokenization granularity but does not explain why information at the beginning of a long context is ignored. Even with a large vocabulary, the model could still fail to attend across 32,000 tokens if it was not trained on such lengths or if positional encoding does not generalize. This factor is unrelated to the described long-context retrieval failure.

• ✓

  The positional encoding scheme may not generalize well beyond the sequence lengths seen during pretraining.

  Why this is correct

  Many positional encoding schemes, especially learned absolute embeddings or RoPE without scaling, degrade when sequences exceed the lengths seen during training. The model cannot reliably distinguish or weight positions far beyond its training range, so attention to early tokens becomes imprecise. This directly contributes to the failure to retrieve information from the beginning of a long document.

• ✗

  Causal attention masks prevent each token from attending to future tokens, which limits bidirectional reasoning over the document.

  Why it's wrong here

  Causal masking is inherent to decoder-only models and is not a defect; it ensures autoregressive generation. The question is about attending to earlier tokens, which causal attention does allow. The failure to use early information stems from training distribution and positional encoding behavior, not from the causal mask itself, which is functioning as intended.

• ✓

  The model was pretrained primarily on sequences much shorter than 32,000 tokens, so it did not learn to attend across the full context.

  Why this is correct

  If pretraining used sequences far shorter than the target context length, the attention patterns never learned to connect distant tokens. Extending context at inference without training on long sequences leads to poor utilization of early tokens. This is a well-documented cause of the lost-in-the-middle effect, where information at the extremes of a long context is underused.