Self-Attention Mechanism in Transformers
What is the primary purpose of the self-attention mechanism in a transformer model?
Quick Answer
The correct answer captures the two things self-attention was specifically designed to solve at the same time, and seeing why they come as a pair is the real lesson here. Every token computes an attention score against every other token in the sequence, so a word near the end can directly weigh a word near the beginning without information having to pass through a long chain of intermediate steps. That direct, all-pairs connection is what lets the model capture long-range dependencies, since nothing about the relationship between two tokens depends on how far apart they sit in the sequence. The second half of the answer is just as important: because those attention scores for all token pairs can be computed at once rather than one step after another, the whole sequence can be processed in parallel. This is the direct contrast with recurrent architectures like RNNs, which must process tokens sequentially and therefore struggle both to preserve information over long distances and to take advantage of parallel hardware. An answer that only mentions parallelism or only mentions long-range context is incomplete, since self-attention's defining advantage is delivering both from the same mechanism. When a question asks what self-attention or a transformer fundamentally improves on compared to older sequence models, expect the correct answer to pair parallel computation with whole-sequence context in a single description.
⚠ Common exam trap
The 1Z0-1127 exam often tests the distinction between the self-attention mechanism's core function (parallel processing and long-range dependencies) and other transformer components like embeddings or causal masking, leading candidates to confuse the purpose of self-attention with the overall autoregressive nature of the decoder.
Answer choices
Why each option matters
Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.
Correct answer & explanation
✓
To process tokens in parallel while modeling long-range dependencies
The self-attention mechanism allows each token in the input sequence to attend to every other token, computing a weighted sum of their representations. This enables the model to capture long-range dependencies directly without the sequential processing constraints of RNNs, and because the attention scores for all tokens can be computed simultaneously, the mechanism supports parallel processing of the entire sequence.
Answer analysis
Option-by-option breakdown
For each option: why learners choose it and why it is or isn't the right answer here.
- ✗
To reduce the number of parameters in the model
Why it's wrong here
Self-attention does not reduce parameters; it adds them.
- ✗
To convert tokens into fixed-length vectors
Why it's wrong here
That is the role of token embeddings, not self-attention.
- ✗
To ensure the model is autoregressive
Why it's wrong here
Autoregressive property is achieved by masking, not by self-attention itself.
- ✓
To process tokens in parallel while modeling long-range dependencies
Why this is correct
Self-attention enables parallelization by computing attention scores between all token pairs simultaneously, and its receptive field covers the entire sequence.
Go deeper
Related to this question
About these practice questions
This 1Z0-1127-25 question is part of Courseiva's 768-question bank — original exam-style content with full explanations and wrong-answer analysis, never real exam questions or exam dumps. Learn why practice questions differ from exam dumps →
Same concept, more angles
3 more ways this is tested on 1Z0-1127-25
These questions test the same concept from different angles. Work through them to make sure you can recognise it however the exam phrases it.
Variation 1. Which of the following best describes the role of the self-attention mechanism in a Transformer model?
easy- A.It encodes the order of tokens in the sequence
- ✓ B.It computes a weighted sum of all input token representations, where weights depend on pairwise compatibility between tokens
- C.It applies a convolutional filter over local windows of tokens
- D.It replaces the need for positional encoding by using recurrence
Why B: The self-attention mechanism computes a weighted sum of all input token representations, where the weights are determined by the pairwise compatibility (attention scores) between tokens. This allows each token to dynamically attend to every other token in the sequence, capturing global dependencies without the limitations of fixed local windows or recurrence.
Variation 2. In the self-attention mechanism, what is the role of the 'scaling factor' (division by sqrt(d_k)) in the softmax computation?
hard- A.To make the attention mechanism translation invariant
- ✓ B.To prevent the softmax from saturating and producing small gradients
- C.To increase the variance of attention scores
- D.To ensure the sum of attention weights equals 1
Why B: Scaling prevents the dot products from growing too large in magnitude, which would push softmax into regions with extremely small gradients.
Variation 3. What is the primary purpose of the self-attention mechanism in a Transformer model?
easy- A.To generate token embeddings in parallel
- B.To reduce the dimensionality of token embeddings
- C.To encode positional information of tokens
- ✓ D.To compute a weighted sum of all token representations based on pairwise relevance
Why D: Self-attention allows each token to attend to every other token in the sequence, capturing contextual relationships regardless of distance.
JA
Written by Johnson Ajibi, MSc IT Security
Senior Network & Security Engineer · founder of Courseiva
This 1Z0-1127-25 practice question is part of Courseiva's free Oracle certification practice question bank. Courseiva provides original exam-style practice questions with explanations, topic-based practice, mock exams, readiness tracking, and study analytics to help learners prepare for the 1Z0-1127-25 exam.