AI-900 Practice Question: Describe features of generative AI workloads on Azure

Answer analysis

Option-by-option breakdown

For each option: why learners choose it and why it is or isn't the right answer here.

• ✗

  Temperature

  Why it's wrong here

  Temperature adjusts the probability distribution over the vocabulary by scaling logits before sampling; a higher value (e.g., 1.5) increases randomness, while a lower value (e.g., 0.2) makes outputs more deterministic. However, it applies the same scaling globally to all candidate tokens at each step and does not track which tokens have already appeared in the generated sequence. Consequently, even with high temperature, the model can still fall into repetitive loops because there is no mechanism to explicitly penalize previously used tokens. Only a dedicated repetition penalty directly reduces the likelihood of generating the same token again.

• ✗

  Top_p (nucleus sampling)

  Why it's wrong here

  Top_p (nucleus sampling) dynamically selects the smallest set of tokens whose cumulative probability mass equals or exceeds the threshold p (e.g., 0.9) and then samples only from that set. This can add variety by discarding extremely low-probability tokens and expanding the candidate pool, which might indirectly reduce some repetitive patterns in short generations. However, top_p does not incorporate any history of token usage; it chooses from the same plausible candidates regardless of whether they were already generated. Therefore, if the model assigns high probability to repeating a token due to repetition patterns in its training data, top_p will not prevent that repetition, because it never adjusts probabilities based on prior occurrences.

• ✓

  Frequency penalty

  Why this is correct

  Frequency penalty is the correct parameter because it explicitly subtracts a value proportional to how many times a token has already appeared in the current generated text. In Azure OpenAI, the penalty is applied to the log-probability of each token as frequency_penalty * count_of_token_occurrences, so a token used twice gets twice the penalty of a token used once. This directly discourages the model from reusing the same words or phrases over and over, making the output more novel and varied. It is specifically designed for repetition control, unlike temperature or top_p which only shape the overall probability distribution without referencing generated content.

• ✗

  Max tokens

  Why it's wrong here

  Max tokens sets the absolute upper bound on the number of tokens the Azure OpenAI service will generate, commonly used to limit response length, cost, and latency. It does not alter the token selection logic at any individual generation step, so it cannot prevent the model from choosing repeated tokens before hitting that limit. While a shorter max_tokens might truncate a repetitive sequence and make the output seem less repetitive, the repeated content already generated remains, and the model's per-step probabilities are unchanged. Thus, max tokens is a length ceiling, not a repetition-control mechanism; the writer should use frequency penalty to directly address repetitive text.

Go deeper

Related to this question