NCP-GENL LLM Architecture Practice Question

Answer analysis

Option-by-option breakdown

For each option: why learners choose it and why it is or isn't the right answer here.

• ✓

  Routing decisions are computed per token, allowing different tokens in the same sequence to be processed by different expert subsets

  Why this is correct

  The gating network produces a distribution over experts independently for each token, so token routing is dynamic and token-specific. This token-level granularity is what allows an MoE layer to specialize experts across linguistic or semantic patterns while keeping computation sparse for any individual token in the sequence.

• ✗

  Increasing the number of experts proportionally increases the number of activated experts per token

  Why it's wrong here

  Adding experts expands total parameters and capacity but does not change k, the number of experts activated per token. The router still selects only the top-k experts, so per-token compute stays roughly constant while model capacity grows. This decoupling of capacity from compute is a central property of sparse MoE designs.

• ✗

  The router assigns tokens to experts once at initialization and keeps the assignment fixed throughout training

  Why it's wrong here

  Router parameters are learned jointly with the rest of the network, and assignments change continuously as gating weights update. A static assignment would remove the adaptive specialization that makes MoE effective. Fixed routing also ignores input context, defeating the purpose of a learned gating function that responds to token representations.

• ✓

  Only the experts selected by the router for a given token are activated, so FLOPs per token scale with k rather than total expert count

  Why this is correct

  In sparse MoE layers, the router scores each token against all experts, then dispatches the token only to the top-k experts. Because only those experts run their feed-forward computation for that token, per-token compute depends on k and expert size, not on how many experts exist in total, enabling capacity growth without proportional FLOPs.

• ✗

  All experts receive every token, but their outputs are averaged with learned scalar weights

  Why it's wrong here

  That describes a dense ensemble or soft mixture, not sparse top-k routing. In sparse MoE, unselected experts do not receive or process the token at all. Averaging all expert outputs would eliminate the compute savings that motivate MoE and would contradict the sparse activation pattern the architecture is designed to provide.