NCP-GENL LLM Architecture Practice Question

Answer analysis

Option-by-option breakdown

For each option: why learners choose it and why it is or isn't the right answer here.

• ✓

  Initialize all residual output projections with a small scaling factor such as 1/sqrt(2N).

  Why this is correct

  Scaling residual branch outputs at initialization by a depth-dependent factor keeps the variance of the residual stream roughly constant across layers. This directly tames the growth that causes exploding gradients in post-LN Transformers, allowing stable early training without moving normalization. It is a recognized architectural fix used in deep Transformer initialization schemes such as those in Megatron-style training.

• ✗

  Apply learning rate warmup over the first several thousand steps.

  Why it's wrong here

  Warmup mitigates large early updates and is standard practice, but it does not address the root cause of post-LN instability: the residual stream magnitude grows with depth, so gradients through deep blocks remain ill-conditioned. Spikes can still occur after warmup completes, and the question asks for an architectural technique, not an optimizer schedule change.

• ✗

  Insert RMSNorm immediately after the token embedding layer only.

  Why it's wrong here

  Adding RMSNorm only after embeddings normalizes the input scale but does nothing to constrain the growing activations inside each Transformer block, which is where the gradient spikes originate. Early training instability with post-LN placement comes from unnormalized residual stream growth across many layers, so a single normalization at the input cannot prevent the exploding gradients observed during the first few hundred steps.

• ✗

  Replace GELU activations in the feed-forward network with ReLU.

  Why it's wrong here

  Changing the activation function alters nonlinearity shape but does not control residual stream variance, which is the mechanism behind the observed gradient spikes. ReLU can even worsen dead-neuron issues at high learning rates. The instability is a depth-scaling problem, so swapping activations leaves the fundamental cause untouched and may reduce model quality.