Automated Concept Drift Detection with SageMaker Model Monitor

Exhibit

Refer to the exhibit.
```
2024-01-15 10:23:45,123 [INFO] Starting inference at endpoint ...
2024-01-15 10:23:45,456 [ERROR] Model output contains NaN values.
2024-01-15 10:23:45,457 [WARN] Input feature x has value -9999.0 which is unusual.
```

Refer to the exhibit. A data scientist reviews the CloudWatch Logs from an Amazon SageMaker real-time endpoint. What is the MOST likely root cause of the NaN output?

⚠ Common exam trap

AWS often tests the misconception that NaN outputs are caused by infrastructure issues like overload or file corruption, when the actual root cause is almost always data-related numerical instability in the model's inference logic.

Answer choices

Why each option matters

Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.

Correct answer & explanation

✓

The input data contains out-of-range values not seen during training, causing the model to output NaN.

The NaN (Not a Number) output from a SageMaker real-time endpoint is most commonly caused by input data containing values outside the range seen during training. This can lead to numerical instability in the model's forward pass, such as division by zero, log of zero, or exponent overflow, which propagates through layers and results in NaN predictions.

Answer analysis

Option-by-option breakdown

For each option: why learners choose it and why it is or isn't the right answer here.

✗
The model weights became corrupted due to a disk write error.
Why it's wrong here
Disk write errors corrupting weights would surface as load or checksum failures at endpoint startup, not as NaN predictions from an otherwise running endpoint. Corruption is the right cause when logs show artifact integrity errors, but the exhibit indicates malformed feature values passed at inference time.
✓
The input data contains out-of-range values not seen during training, causing the model to output NaN.
Why this is correct
Out-of-range inputs violate the feature distribution the model learned, so activations or gradients overflow and produce NaN. This directly satisfies the stem's real-time endpoint scenario: CloudWatch Logs capture inference-time anomalies, and unseen extreme values are the most likely trigger rather than training or infrastructure faults.
✗
The endpoint is overloaded and returning a default NaN response.
Why it's wrong here
Overload produces throttling, timeouts or 5xx responses, not NaN values; SageMaker does not substitute NaN as a default. Overload is the correct diagnosis when logs show latency spikes and capacity errors, whereas NaN output originates from the inference input or model computation itself.
✗
The model artifact failed to load correctly, resulting in NaN weights.
Why it's wrong here
A failed artifact load typically makes the endpoint invocation fail outright or return an error, not silently emit NaN predictions. Artifact loading is the right suspect when logs show container startup or deserialisation errors, but the exhibit points to input data containing nulls reaching the model.

Go deeper

Automated Concept Drift Detection with SageMaker Model Monitor

Exhibit

Why each option matters

Option-by-option breakdown

Related to this question