Transformer Architecture: Self-Attention and Feed-Forward Networks

Quick Answer

The correct answer is the feed-forward neural network, along with the self-attention mechanism, as these are the two essential components of the Transformer architecture. Self-attention allows each token in an input sequence to directly attend to every other token, computing attention scores from queries, keys, and values to capture long-range dependencies without the sequential bottleneck of recurrent networks. The feed-forward network then processes each position independently, applying non-linear transformations to the attention outputs. On the Oracle Cloud Infrastructure Generative AI Professional 1Z0-1127 exam, this question tests your understanding of the core building blocks that enable parallel processing and contextual modeling in models like GPT and BERT. A common trap is confusing positional encodings or layer normalization as essential components—while important, they are supporting mechanisms, not the two foundational pillars. Memory tip: think of the Transformer as a two-step dance—first, every token looks at every other token (self-attention), then each token thinks alone (feed-forward).