Hard Difficulty Questions

The instance count is too high.

Why it fails: This increases cost, not startup time.

The ACR is private.

Why it fails: Workspace managed identity handles ACR access.

The compute is too small.

Why it fails: Compute size affects execution, not image pulling.

Delete the datastore

Why it fails: Deleting the datastore removes the registered data location that the failing job reads from, so it destroys the very input the container needs and cannot expose container-level errors. It is tempting because datastores are managed through the workspace and are used to fix broken data paths, but that is a data-management action, not a debugging one.

Use the Azure ML SDK to download the container image

Why it fails: Downloading the image via the Azure ML SDK retrieves the container's layers for local inspection, but the failure occurs at runtime inside the job, so static image contents reveal nothing about the actual error. This suits auditing or reproducing an environment, not diagnosing a live failing training run.

Submit a new training job without parameters

Why it fails: Omitting parameters discards the container configuration, image and entrypoint needed to reproduce the failure, so nothing is investigated. It tempts as a quick rerun, but debugging requires the same environment; the correct approach inspects logs or opens a shell inside the failing container.

Manual model testing

Why it fails: Manual testing cannot run on every commit, so regressions reach production undetected and results are not reproducible across environments. It is tempting for small teams validating a model before release; it would be right for exploratory one-off evaluation, not a continuous MLOps pipeline.

Using only local workstations

Why it fails: Local workstations cannot reproduce training environments or scale, so runs are inconsistent and untracked, breaking the repeatability MLOps requires. It is tempting for individual experimentation on small datasets; it would be correct for prototyping, not for shared, governed production pipelines.

In the 'Endpoint' YAML file.

Why it fails: The endpoint YAML defines the endpoint's name, auth mode and traffic routing, not per-deployment compute. Instance type belongs in the deployment YAML, since one endpoint can host several deployments on different SKUs. Specifying it here is tempting because the endpoint is the top-level object you create first.

In the 'Environment' definition.

Why it fails: The environment definition specifies the Docker image, conda dependencies and environment variables for the scoring container, not compute SKUs. Instance type is declared in the deployment YAML. Confusing the two is tempting because both are YAML artefacts attached to a deployment, but they control entirely separate concerns.

In the 'Workspace' settings.

Why it fails: Workspace settings hold shared configuration such as storage, keys and container registry, and apply across all endpoints, so they cannot set per-deployment compute. The deployment YAML carries the instance type. Workspace-level configuration is tempting because it feels like the natural home for defaults, but it lacks deployment granularity.

The workspace is too large

Why it fails: Workspace size does not affect file resolution; a pipeline step resolves datastore paths through registered datastore references, not workspace capacity. It is tempting because large workspaces can slow operations, and would be relevant if the failure were a timeout or storage-quota error rather than a file-not-found error.

The compute cluster is off

Why it fails: A stopped compute cluster prevents the step from running at all, producing a compute-unavailable error, not a file-not-found error. It is tempting because compute must be running for steps to execute, and would be correct if the pipeline failed with a cluster provisioning or target-compute error instead of a missing datastore path.

The subscription is expired

Why it fails: An expired subscription would block the whole workspace, not selectively hide one datastore file from a pipeline step. It is tempting because subscription issues do cause broad pipeline failures, and would be correct if every step failed with authorisation or quota errors rather than a single missing-file path error.

CPU usage of the inference cluster

Why it fails: CPU usage reflects infrastructure load, not model accuracy, so a busy cluster can still serve correct predictions. Host resource metrics suit capacity planning and autoscaling, where you add nodes as utilisation climbs, rather than triggering retraining when the model itself drifts.

Network latency

Why it fails: Network latency measures transport delay between client and service, not prediction quality, so it can rise while the model remains accurate. Latency monitoring suits SLO compliance and network troubleshooting, where you alert on response-time breaches, not retraining triggers.

Number of active users

Why it fails: Active-user counts measure demand, not model accuracy, so they cannot signal performance degradation; a spike may simply reflect a marketing campaign. Monitoring user numbers suits capacity planning or licence forecasting, where headcount drives infrastructure sizing, not retraining decisions.

max_tokens

Why it fails: max_tokens caps the total length of generated output; it does not penalise repeated tokens, only truncates generation once the limit is reached. It is tempting because shorter outputs may contain less visible repetition, but the mechanism is length limiting. frequency_penalty directly reduces the probability of tokens by occurrence count.

presence_penalty

Why it fails: presence_penalty penalises tokens that have already appeared at all, pushing the model toward new topics rather than suppressing repeated phrasing. frequency_penalty, which scales with each token's occurrence count, is what targets repetition. Presence penalty suits encouraging topical diversity, not fixing redundant output.

top_p

Why it fails: top_p is nucleus sampling: it restricts the candidate pool to the smallest set of tokens whose cumulative probability reaches the threshold. It controls randomness, not repetition. Lowering top_p makes output more deterministic, which is why it seems relevant, but it applies no penalty based on tokens already generated.

Hardcode the key in the flow configuration.

Why it fails: Hardcoding exposes the credential in source control, logs and flow exports, so any reader gains the vector store key. It is tempting for quick local testing, but production secrets belong in Azure Key Vault, retrieved at runtime through a managed identity or connector connection.

Pass it as a parameter in the code.

Why it fails: Parameters surface in run histories, debug output and code repositories, exposing the credential to anyone with pipeline read access. Parameterisation suits non-secret configuration such as index names or endpoints; secrets require Azure Key Vault retrieval at runtime instead.

Store it in an environment variable.

Why it fails: Environment variables are readable by any process on the host and leak into logs, crash dumps and container images, so they provide no access control or rotation. They suit non-secret settings like endpoint URLs; credentials belong in Azure Key Vault with managed identity access.

Relevance

Why it fails: Relevance scores how well retrieved context matches the query, not whether the answer stays grounded in that context, so fabricated content can still rank as relevant. Relevance is the correct metric when tuning retrieval quality or ranking passages.

Performance

Why it fails: Performance measures latency and throughput, not whether generated text is grounded in retrieved documents. It is tempting because performance tuning matters when a RAG pipeline is slow or costly at scale, making it the right focus for optimisation work. Here, however, hallucination requires a groundedness or faithfulness metric that scores answer claims against the retrieved context.

Fluency

Why it fails: Fluency measures grammatical and linguistic naturalness, so a fluent hallucination scores well and the metric cannot detect unsupported claims. Fluency evaluation is the right choice when assessing generated text readability or comparing language quality of model outputs.

The Azure ML workspace name

Why it fails: The workspace name has no bearing on inference latency; it is only an organisational label. Checking it is tempting because the workspace is the top-level container holding the endpoint, and would be relevant when locating or moving resources, not when diagnosing response times.

The subscription limit

Why it fails: Subscription limits cap resource counts and quota, producing deployment failures rather than slow responses on an already-running endpoint. It is tempting because quota exhaustion is a common Azure issue, and would be correct when a deployment or scale-out operation is rejected, not when latency degrades.

The storage account size

Why it fails: Storage account size does not affect Managed Online Endpoint inference latency; the endpoint serves from provisioned compute, not blob storage. It is tempting because storage underpins model artefacts and logging, and would matter when diagnosing model registration or logging failures, not request latency.

It bypasses the safety alignment layer.

Why it fails: Chain-of-Thought operates purely at the prompt and decoding level, generating reasoning tokens; it does not touch or bypass safety alignment, which remains applied. It is tempting because jailbreak-style prompts can evade filters, but that is a separate adversarial technique unrelated to reasoning quality.

It forces the model to use more input tokens, reducing costs.

Why it fails: Chain-of-Thought improves accuracy by generating intermediate reasoning steps, not by inflating token counts; extra tokens raise cost, not reduce it. It is tempting because longer prompts do consume more input tokens, but that is a billing consequence, not a performance mechanism.

It reduces the context window size.

Why it fails: Chain-of-Thought expands the generated output with intermediate reasoning, consuming context rather than shrinking it. It is tempting because concise prompts use fewer tokens, but reducing context window size is a configuration or truncation concern, not a reasoning technique, and would harm edge-case handling.

Unit testing code snippets

Why it fails: Red teaming probes a deployed AI system adversarially to expose harmful, unsafe or jailbreak outputs; unit testing verifies individual code functions against expected results during development. Unit testing is the right choice for validating deterministic logic, but it exercises no model behaviour and cannot surface prompt-injection or content-safety failures.

Automated performance testing for latency

Why it fails: Red teaming is adversarial simulation that probes defences by mimicking attacker techniques, not measuring latency. Performance testing is tempting because it also exercises the application under load, but it belongs to non-functional or load-testing scenarios, where throughput and response times, not exploitation of weaknesses, are the objective.

A/B testing two different model versions

Why it fails: Red teaming adversarially attacks an application to uncover exploitable weaknesses; A/B testing compares model versions against business or quality metrics. A/B testing is tempting because it evaluates deployed models, but it belongs to experimentation and MLOps scenarios, where statistical comparison of variants, not adversarial exploitation, is the goal.

Token usage tracking

Why it fails: Token usage tracking measures prompt and completion length, which correlates with cost and verbosity, not adversarial intent; a jailbreak can be short or long. It is tempting because unusual token counts sometimes flag anomalous prompts, but detecting instruction-bypass attempts requires analysing semantic intent, such as classifier or prompt-injection scoring.

Latency threshold monitoring

Why it fails: Latency threshold monitoring detects slow responses, which reflects model or infrastructure performance rather than whether an input attempts to bypass system instructions. It is tempting because adversarial prompts can occasionally trigger longer processing, but jailbreak detection needs content-level analysis of the input, not timing metrics.

Semantic similarity scoring

Why it fails: Semantic similarity scoring compares meaning between texts, so it measures topical closeness rather than whether an input is crafted to override system instructions. It is tempting because it is used for RAG relevance and groundedness checks, but jailbreak detection requires identifying adversarial intent, typically via a dedicated classifier or prompt-injection detector.

Disable the 'Load Balancer' to save costs.

Why it fails: Disabling the load balancer removes traffic distribution and health-based failover, leaving one instance serving all requests. It is tempting as a cost-saving measure, but the load balancer is precisely what routes around failed replicas to maintain availability.

Use a 'Standard' tier App Service plan.

Why it fails: App Service plans are less flexible than AKS for custom model containers.

Deploy the container on a single large VM.

Why it fails: A single large VM creates a single point of failure, so a host outage takes the model offline entirely. It is tempting because vertical scaling gives substantial compute capacity for inference, but high availability requires multiple replicas distributed across nodes or availability zones.

Change the VM SKU to a burstable instance type.

Why it fails: Burstable instances are not generally applicable to the managed compute runtimes used by Prompt Flow.

Implement a script to delete the runtime when the flow is finished.

Why it fails: This is an inefficient operational pattern compared to using native runtime shutdown policies.

Set the concurrency limit to zero.

Why it fails: Setting concurrency to zero stops the flow from processing, but does not release the underlying compute infrastructure.

In the 'Python' score script.

Why it fails: The application code is not responsible for SSL.

Inside the Endpoint configuration YAML.

Why it fails: Certificates are not managed in the YAML.

In the 'workspace.json' file.

Why it fails: Workspace files are not for infrastructure security.

Use a smaller Docker container

Why it fails: Container size does not impact the underlying VM disk allocation.

Increase the OS disk size of the VM

Why it fails: This is generally not configurable by the end-user in the ML training job context.

Change the experiment name

Why it fails: Experiment names do not affect disk capacity.

Target storage account URL.

Why it fails: Not part of inference config.

Compute instance size.

Why it fails: Part of the deployment config.

Network Security Group

Why it fails: NSGs control network traffic, not identity-based permissions.

Azure Storage Access Policies

Why it fails: Storage policies do not control compute resource creation.

Azure ML Datastore configuration

Why it fails: Datastores manage data connectivity, not compute administration permissions.

A manual 'Endpoint' health check.

Why it fails: Manual testing is not automated.

The training script.

Why it fails: The training script is for training, not testing deployment.

Azure Machine Learning Studio 'Tests' tab.

Why it fails: Does not exist.