NCP-GENL LLM Architecture Practice Question

Answer analysis

Option-by-option breakdown

For each option: why learners choose it and why it is or isn't the right answer here.

• ✗

  It forces all parameters to update at every training step.

  Why it's wrong here

  The router actually prevents all parameters from updating at once; only the active experts and the router weights are updated for a specific token. This sparsity is the primary advantage of MoE, as it allows training massive models efficiently by distributing the computational load across different parameter subsets.

• ✓

  It allows scaling parameters while keeping compute costs manageable.

  Why this is correct

  MoE allows for a massive total parameter count while maintaining a constant amount of active parameters per token. By routing tokens to specialized experts, the model achieves high capacity and knowledge breadth without the linear compute cost increases that would occur in a fully dense architecture of equivalent size.

• ✗

  It forces the model to use all experts simultaneously for every input.

  Why it's wrong here

  If every expert were used for every input, the model would simply be a very large dense network, defeating the purpose of the MoE architecture. The router is designed to ensure sparsity, enabling efficient inference and training by only activating a small, specific set of experts for each individual token.

• ✗

  It replaces the attention mechanism with a standard linear layer.

  Why it's wrong here

  MoE and attention are complementary; the attention mechanism is still required to handle sequence dependencies. The MoE structure is typically applied to the feed-forward layers within the Transformer blocks, not as a replacement for the attention mechanism, which is responsible for token-to-token relationship modeling.