AI-900 Practice Question: Describe features of generative AI workloads on Azure
What is 'speculative decoding' and how does it improve LLM inference speed?
⚠ Common exam trap
Many exam-takers confuse speculative decoding with simple input prediction or CPU/GPU offloading, but Microsoft often tests the specific mechanism of using a draft model for parallel token verification as the defining characteristic of speculative decoding.
Answer choices
Why each option matters
Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.
Correct answer & explanation
✓
Using a small draft model to generate candidate tokens that a large model verifies in parallel — improving throughput
Speculative decoding improves LLM inference speed by using a small, fast draft model to generate multiple candidate tokens in sequence, which are then verified in parallel by the large target model. This parallel verification allows the large model to accept or reject entire blocks of tokens at once, significantly reducing the number of sequential autoregressive steps required. The technique leverages the observation that draft models can produce acceptable continuations most of the time, and the large model only needs to correct mistakes, leading to higher throughput without sacrificing output quality.
Answer analysis
Option-by-option breakdown
For each option: why learners choose it and why it is or isn't the right answer here.
- ✗
Predicting user input before they finish typing to pre-compute responses
Why it's wrong here
This is a user-interface feature that predicts likely keystrokes or words to pre-render responses before a user finishes typing, improving perceived responsiveness. Speculative decoding is unrelated to user behavior; it is an inference-time optimization for autoregressive language models, where a draft model generates multiple candidate tokens and the target model validates them concurrently. The goal is reducing generation latency per pass, not anticipating human input.
- ✓
Using a small draft model to generate candidate tokens that a large model verifies in parallel — improving throughput
Why this is correct
This correctly identifies speculative decoding: a lightweight draft model autoregressively proposes a batch of candidate tokens, and the large target model verifies them in parallel with a single forward pass, accepting consistent tokens and resampling where needed. Because verification matches the target model's exact distribution, output quality is preserved while throughput increases and latency decreases across multiple tokens per pass.
- ✗
Generating speculative forecasts about future events using language model knowledge
Why it's wrong here
This refers to using a language model to generate probabilistic statements about future events, such as market trends or weather conditions, which is a domain-specific application of the model's learned knowledge. Speculative decoding, however, is a generic inference-efficiency technique that accelerates token generation for any model, independent of the semantic content of the outputs. It is a low-level algorithmic optimization, not a forecasting capability.
- ✗
Running model inference on the CPU while the GPU processes the next request in parallel
Why it's wrong here
This describes heterogeneous pipeline parallelism, where separate inference requests are scheduled across CPU and GPU to boost overall hardware utilization. In contrast, speculative decoding is a token-level decoding algorithm that operates within a single inference call, using a small draft model to propose candidate tokens that a larger model verifies in one forward pass. It targets generation efficiency rather than cross-device scheduling.
Go deeper
Related to this question
About these practice questions
This AI-900 question is part of Courseiva's 985-question bank — original exam-style content with full explanations and wrong-answer analysis, never real exam questions or exam dumps. Learn why practice questions differ from exam dumps →
JA
Written by Johnson Ajibi, MSc IT Security
Senior Network & Security Engineer · founder of Courseiva
This AI-900 practice question is part of Courseiva's free Microsoft certification practice question bank. Courseiva provides original exam-style practice questions with explanations, topic-based practice, mock exams, readiness tracking, and study analytics to help learners prepare for the AI-900 exam.