Generative AI Optimization practice questions

Disable the system message.

Why it fails: Disabling the system message removes control over model behavior.

Increase the temperature parameter to 1.0.

Why it fails: High temperature increases randomness, worsening hallucinations.

Switch from GPT-4o to GPT-3.5 Turbo.

Why it fails: Smaller models are generally more prone to hallucinations in complex RAG tasks.

Azure Network Watcher.

Why it fails: Used for monitoring network traffic.

Azure Advisor.

Why it fails: Used for best practice recommendations, not fine-grained cost reporting.

Azure Policy.

Why it fails: Used for compliance and governance.

Changing the model to GPT-4o.

Why it fails: This doesn't change token efficiency.

Removing the API key from the request.

Why it fails: This would break the API call.

Using a higher temperature.

Why it fails: Does not impact token count.

top_k

Why it fails: Azure OpenAI uses top_p, not top_k.

max_tokens

Why it fails: Controls length, not style.

The model's temperature is set too high.

Why it fails: Temperature governs sampling randomness at inference, not gradient updates during fine-tuning, so it cannot cause training loss oscillation. It is tempting because temperature is the familiar randomness control, but it would be the correct adjustment when tuning response variability in deployed completions, not training dynamics.

The training dataset size is too large.

Why it fails: A larger dataset reduces variance in gradient estimates and generally smooths the loss curve; it does not cause fluctuation. The tempting association is that big data complicates training, yet dataset size becomes the relevant factor when overfitting or underfitting is observed, not oscillation.

The model lacks enough epochs.

Why it fails: Too few epochs produces consistently high loss that has not yet converged, not a fluctuating curve. Fluctuating loss typically reflects an excessive learning rate or small batch size. More epochs would be the right remedy when training loss plateaus above the target without oscillation.

max_tokens

Why it fails: max_tokens caps response length only; it does not alter sampling, so outputs stay non-deterministic. It is tempting because it visibly constrains output, and would be the right parameter when you need to limit response size or cost rather than control randomness.

presence_penalty

Why it fails: presence_penalty alters the likelihood of introducing tokens already present in the text, changing topic repetition rather than removing sampling randomness. It is tempting because it shapes output style, but determinism requires temperature near zero and a fixed seed, which constrain token selection directly.

frequency_penalty

Why it fails: frequency_penalty scales token probabilities by how often they have already appeared, reducing verbatim repetition but leaving random sampling intact. It is tempting because it steadies wording, yet repeatable output demands temperature set to zero plus a fixed seed, which control the sampling distribution itself.

Azure Data Factory.

Why it fails: Azure Data Factory orchestrates data movement and transformation pipelines; it exposes no per-user token accounting for model inference. Tracking token consumption per user requires Microsoft Entra ID-authenticated usage metrics surfaced through Azure Monitor or Application Insights, which Data Factory cannot capture.

Azure Cognitive Search.

Why it fails: Azure Cognitive Search indexes and queries documents; it exposes no per-user token consumption metrics for a chatbot. It would be the right integration when the chatbot needs retrieval over a document corpus, not for metering model usage.

Azure SQL Database.

Why it fails: Azure SQL Database stores relational data but does not capture model token counts or attribute them per user. It would be correct for persisting chatbot conversation records or application state, not for monitoring token consumption.

Using a smaller batch size.

Why it fails: Batch size governs how many requests are processed per call, not how many tokens each request carries, so it leaves the context window cost untouched. It is tempting because batching genuinely reduces per-request overhead and improves throughput in high-volume inference workloads — but the stem asks specifically about long-context token consumption.

Disabling streaming responses.

Why it fails: Disabling streaming changes only how tokens are delivered, not how many are billed; the full prompt still occupies the context window, so cost stays identical. It is tempting because streaming affects perceived latency and throughput tuning, and disabling it would be the right choice when a client cannot handle server-sent events or needs complete responses for buffering.

Increasing the frequency penalty.

Why it fails: Increasing the frequency penalty only discourages repeated tokens during generation; it does nothing to shrink the input tokens billed against the context window. It is tempting because it genuinely curbs verbatim repetition in long outputs, making it the right lever when generated text loops or reuses phrases excessively — but that is an output-quality control, not a context-cost reduction.

Increase the top_p parameter.

Why it fails: top_p controls nucleus sampling breadth, altering output diversity rather than prompt length. Raising it can lengthen or destabilise completions, adding latency. It is tempting because sampling parameters feel like performance levers, but they govern randomness; the stem's high input tokens require prompt reduction instead.

Enable dynamic scaling in Azure App Service.

Why it fails: App Service dynamic scaling adds web-tier instances, but the latency originates in the Azure OpenAI deployment's token processing, not the hosting layer. It is tempting because scaling feels like a latency fix, yet it cannot shorten prompts. Reducing input tokens is what lowers inference time.

Decrease the number of parallel requests.

Why it fails: Reducing parallel requests lowers throughput and queues callers, so latency per caller rises rather than falls. Concurrency limits address quota or throttling errors, not token-driven inference time. Trimming input tokens, through summarisation or prompt compression, directly shortens the prompt the model processes.

Use a higher model version (e.g., o1-preview).

Why it fails: Reasoning-oriented models such as o1-preview generate lengthy internal reasoning chains, increasing latency substantially. Higher model versions are chosen for complex reasoning accuracy, not throughput. The bottleneck here is input token volume, which prompt compression or summarisation addresses directly.

Regional failover.

Why it fails: Regional failover redirects traffic to another region during an outage; it does not reserve model capacity, so throughput still fluctuates with demand. It tempts because failover improves availability and resilience, which would be the right choice when the requirement is continuity during a regional service disruption rather than guaranteed throughput.

Auto-scaling groups.

Why it fails: Auto-scaling groups adjust compute instance counts for workloads such as VMs; they do not govern Azure OpenAI token throughput or reserve model capacity. It tempts because auto-scaling maintains performance under load, which would be correct for scaling a self-hosted application tier, not for provisioning dedicated model throughput.

Token throttling.

Why it fails: Token throttling caps request rates to prevent quota exhaustion; it constrains consumption rather than reserving guaranteed throughput, so performance still degrades under high demand. It tempts because throttling protects service stability, which would be appropriate when the goal is limiting abusive or excessive callers rather than ensuring consistent capacity.

Increase the fine-tuning learning rate.

Why it fails: A higher learning rate takes larger steps away from the pretrained weights, worsening catastrophic forgetting. It is tempting because raising it speeds convergence on the small dataset, and would be right when the model underfits and training loss plateaus too early.

Change the model architecture to GPT-4.

Why it fails: Swapping to GPT-4 discards the fine-tuned weights entirely and does not address forgetting during fine-tuning. It is tempting because a stronger base model performs well generally, and would be right when the task needs broader capability than the current architecture provides.

Reduce the batch size to 1.

Why it fails: Batch size 1 raises gradient variance and update frequency, accelerating drift from pretrained weights rather than preserving them. It is tempting because small batches are often recommended for limited data, and would be right when memory is constrained or when seeking noisier updates for generalisation.

Increases the model's accuracy.

Why it fails: Streaming affects delivery timing of already-generated tokens, not the model's predictive quality; accuracy depends on model choice, prompt design and parameters. It is tempting because faster perceived responses can feel like improved output, and streaming would be correct when you want to display partial results to users before completion.

Removes the need for a system prompt.

Why it fails: Streaming only changes how response tokens are delivered to the client, returning them incrementally as they are generated; it has no bearing on system prompt requirements, which remain part of the request regardless. It is tempting because streaming reduces perceived latency, and it would be the right choice when you need progressive display of long responses.

Decreases the total number of tokens consumed.

Why it fails: Streaming changes only the transport of tokens to the client; the same prompt and completion are generated, so billed token counts are unchanged. It is tempting because incremental delivery can feel lighter, and streaming would be the correct choice when you need lower time-to-first-token rather than reduced consumption.

It bypasses the safety alignment layer.

Why it fails: Chain-of-Thought operates purely at the prompt and decoding level, generating reasoning tokens; it does not touch or bypass safety alignment, which remains applied. It is tempting because jailbreak-style prompts can evade filters, but that is a separate adversarial technique unrelated to reasoning quality.

It forces the model to use more input tokens, reducing costs.

Why it fails: Chain-of-Thought improves accuracy by generating intermediate reasoning steps, not by inflating token counts; extra tokens raise cost, not reduce it. It is tempting because longer prompts do consume more input tokens, but that is a billing consequence, not a performance mechanism.

It reduces the context window size.

Why it fails: Chain-of-Thought expands the generated output with intermediate reasoning, consuming context rather than shrinking it. It is tempting because concise prompts use fewer tokens, but reducing context window size is a configuration or truncation concern, not a reasoning technique, and would harm edge-case handling.

Changing the embedding model to a smaller one.

Why it fails: A smaller embedding model changes vector dimensionality and semantic quality but does not filter irrelevant chunks from the result set. It is tempting because smaller models reduce cost and latency, and it would be correct when resource constraints or throughput, rather than retrieval precision, are the limiting factor.

Increasing the number of chunks retrieved.

Why it fails: Retrieving more chunks increases the volume of context, amplifying the noisy information already degrading the response. It is tempting because raising top-k helps when relevant passages are being missed, but this scenario requires filtering or reranking to reduce irrelevant content, not widening the retrieval net.

Reducing the temperature to 0.

Why it fails: Temperature governs sampling randomness in generation, not which documents retrieval returns, so noisy context still reaches the prompt. It is tempting because lowering temperature improves determinism, and it would be correct when the model hallucinates or produces inconsistent wording despite clean retrieved context.

Monitoring the total request count.

Why it fails: Request count measures traffic volume, not output quality, so it stays identical regardless of which prompt performs better. It is tempting because it is easy to instrument, but quantitative prompt evaluation requires scoring responses against ground-truth data using defined metrics.

Asking developers for subjective feedback.

Why it fails: Subjective developer opinions are qualitative and not reproducible, so they cannot quantify a difference between prompt versions. It is tempting because human judgement captures nuance, but reliable comparison requires automated scoring against a labelled evaluation dataset with consistent metrics.

Checking if the model is running on GPT-4.

Why it fails: The underlying model version is fixed across both prompt variants, so it contributes no comparative data about prompt quality. It is tempting because model capability affects output, but quantitative prompt comparison requires scoring outputs against a dataset with defined metrics.

Delete the deployment and recreate it.

Why it fails: Deleting and recreating the deployment does not alter the tokens-per-minute quota allocated to it, so rate limiting recurs immediately. It is tempting because redeployment resets transient faults, and it would be correct when a deployment is stuck in a failed provisioning state rather than throttled.

Disable the rate limit in the portal.

Why it fails: Rate limits are enforced by the service's quota system and cannot be disabled from the portal; the setting does not exist. It is tempting because portal configuration appears authoritative, and it would be correct to adjust quota there by requesting an increase for the subscription.

Increase the temperature of the model.

Why it fails: Temperature controls sampling randomness in generated completions and has no effect on the request quota enforced per deployment. It is tempting because it is a tunable model parameter, and it would be correct when adjusting output creativity or determinism for a use case.

Training the model from scratch.

Why it fails: Training from scratch requires a large labelled corpus, substantial GPU compute and full MLOps lifecycle, so it maximises cost rather than minimising it, and risks worse results than an adapted pretrained model. It is tempting because it offers total control, and it would be correct when the domain is unlike any existing pretrained data and funding is ample.

Prompt engineering alone.

Why it fails: Prompt engineering changes only the input context; it cannot alter model weights, so domain-specific vocabulary and patterns remain unlearned, and per-request token costs recur. It is tempting because it is cheap and immediate, and it would be correct when the base model already handles the domain and only output format or tone needs steering.

Full parameter fine-tuning.

Why it fails: Full parameter fine-tuning updates every weight, demanding high GPU memory, long training runs and a full copy of the model per domain, so cost is not minimised. It is tempting because it yields the strongest adaptation, and it would be correct when maximum accuracy matters and compute budget is unrestricted.

max_tokens

Why it fails: max_tokens caps the total length of generated output; it does not penalise repeated tokens, only truncates generation once the limit is reached. It is tempting because shorter outputs may contain less visible repetition, but the mechanism is length limiting. frequency_penalty directly reduces the probability of tokens by occurrence count.

presence_penalty

Why it fails: presence_penalty penalises tokens that have already appeared at all, pushing the model toward new topics rather than suppressing repeated phrasing. frequency_penalty, which scales with each token's occurrence count, is what targets repetition. Presence penalty suits encouraging topical diversity, not fixing redundant output.

top_p

Why it fails: top_p is nucleus sampling: it restricts the candidate pool to the smallest set of tokens whose cumulative probability reaches the threshold. It controls randomness, not repetition. Lowering top_p makes output more deterministic, which is why it seems relevant, but it applies no penalty based on tokens already generated.

Process all documents in the prompt.

Why it fails: Stuffing every document into the prompt inflates token count and inference time rather than speeding retrieval, and exceeds context limits on a large corpus. It is tempting because prompt-based grounding needs no vector index, and it would be correct for a handful of short documents.

Increase the model's max_tokens.

Why it fails: Raising max_tokens changes how many tokens the model may generate, not how documents are indexed or searched, so retrieval latency is untouched. It is tempting because max_tokens governs output length, and it would be the right lever when responses are being truncated mid-answer.

Perform a brute-force search on all documents.

Why it fails: Brute-force comparison against every document scales linearly with corpus size, so latency grows as the document set grows. It is tempting because exact brute-force search guarantees perfect recall, and it would be correct for a small corpus where approximate nearest-neighbour recall loss is unacceptable.

Increase the temperature.

Why it fails: Raising temperature increases sampling randomness, making output less deterministic and more likely to break JSON syntax. It is tempting when answers look repetitive, since higher temperature diversifies wording, but that is the opposite of what format compliance needs. Structured output with temperature near zero is the correct approach for deterministic schemas.

Add 'output must be JSON' to the prompt.

Why it fails: Prompt instructions are probabilistic guidance, so the model can still emit prose, trailing commas or markdown fences. It is tempting because it needs no code change and works for quick prototypes, but it cannot guarantee syntactic validity. Structured output or JSON mode constrains decoding itself, which is what this scenario requires.

Post-process the output with a regex script.

Why it fails: Regex cannot parse nested JSON reliably, so malformed or deeply structured output passes through or is silently corrupted. It is tempting as a lightweight fix when the model is already close to valid JSON, but it repairs symptoms rather than constraining generation. Schema-enforced structured output prevents invalid tokens being produced at all.