Scenario practice questions

A machine learning engineer at a retail company is monitoring a production model that predicts inventory demand. The model's prediction accuracy has dropped significantly over the past week. The engineer checks the model's input data and notices a new product category was introduced with a different distribution. Which concept is most likely causing the performance degradation?

Trap 1: Concept drift

Concept drift refers to changes in the underlying relationship between features and target, not input distribution.

Trap 2: Data leakage

Data leakage involves the model seeing information it shouldn't, not input distribution change.

Trap 3: Model decay

Model decay is a general term for performance degradation, not the specific cause here.

A
Concept drift
Why wrong: Concept drift refers to changes in the underlying relationship between features and target, not input distribution.
B
Covariate shift
Covariate shift occurs when the distribution of input features changes over time.
C
Data leakage
Why wrong: Data leakage involves the model seeing information it shouldn't, not input distribution change.
D
Model decay
Why wrong: Model decay is a general term for performance degradation, not the specific cause here.

Question 2mediummulti select

A data scientist is preparing a dataset with a categorical feature that has 20 levels. The target variable is continuous. Which THREE encoding methods are appropriate for this scenario? (Select THREE.)

Trap 1: Binary encoding

Binary encoding is a variant of one-hot for high cardinality, but not among the three most common; the question expects one-hot, ordinal, target.

Trap 2: Label encoding

Label encoding assigns arbitrary integers, which may imply ordinal relationship; not generally recommended for nominal categories.

A
One-hot encoding
One-hot encoding creates binary columns for each category; works for any categorical feature.
B
Ordinal encoding
Ordinal encoding is suitable if categories have a natural order; otherwise, use one-hot.
C
Target encoding
Target encoding replaces categories with the mean of the target; works well for high-cardinality and continuous targets.
D
Binary encoding
Why wrong: Binary encoding is a variant of one-hot for high cardinality, but not among the three most common; the question expects one-hot, ordinal, target.
E
Label encoding
Why wrong: Label encoding assigns arbitrary integers, which may imply ordinal relationship; not generally recommended for nominal categories.

Question 3easymultiple choice

A machine learning engineer needs to deploy a model that requires less than 100 ms inference latency for real-time predictions. The model is a small PyTorch model that fits in a single GPU. Which SageMaker inference option is MOST cost-effective for this scenario?

Trap 1: Asynchronous inference endpoint

Asynchronous inference is designed for large payloads and higher latency, not for sub-100 ms predictions.

Trap 2: Real-time endpoint on ml.g4dn.xlarge

Real-time endpoints are always running and incur costs even when idle, not the most cost-effective for low traffic.

Trap 3: Batch transform job

Batch transform is for offline, bulk predictions, not for real-time requests requiring low latency.

A
Asynchronous inference endpoint
Why wrong: Asynchronous inference is designed for large payloads and higher latency, not for sub-100 ms predictions.
B
Real-time endpoint on ml.g4dn.xlarge
Why wrong: Real-time endpoints are always running and incur costs even when idle, not the most cost-effective for low traffic.
C
Batch transform job
Why wrong: Batch transform is for offline, bulk predictions, not for real-time requests requiring low latency.
D
Serverless inference with max concurrency set to 10
Serverless inference scales to zero when idle and charges only for the compute time used, making it cost-effective for low and variable traffic.

Question 4easymultiple choice

A company wants to deploy a trained XGBoost model for batch inference on a large dataset stored in S3. The inference job should be cost-effective and does not require real-time responses. Which SageMaker inference option should they use?

Trap 1: SageMaker real-time endpoint

Real-time endpoints are for low-latency inference, not cost-effective for batch processing.

Trap 2: SageMaker Asynchronous Inference

Asynchronous is for near-real-time with S3 input, but less cost-effective for large batch jobs than Batch Transform.

Trap 3: SageMaker Serverless Inference

Serverless is for on-demand inference, not batch processing of large datasets.

A
SageMaker Batch Transform
Batch Transform is designed for batch inference on S3 data, cost-effective and no real-time requirement.
B
SageMaker real-time endpoint
Why wrong: Real-time endpoints are for low-latency inference, not cost-effective for batch processing.
C
SageMaker Asynchronous Inference
Why wrong: Asynchronous is for near-real-time with S3 input, but less cost-effective for large batch jobs than Batch Transform.
D
SageMaker Serverless Inference
Why wrong: Serverless is for on-demand inference, not batch processing of large datasets.

Question 5mediummultiple choice

A company notices that the prediction distribution of their deployed model has shifted significantly from the training data distribution, but the input data distribution remains unchanged. Which type of drift is occurring, and what is the MOST likely cause?

Trap 1: Data drift; the training data is no longer representative of the…

Data drift refers to input distribution change, which is not happening here.

Trap 2: Bias drift; the model is making unfair predictions against a…

Bias drift relates to fairness metrics, not overall prediction distribution.

Trap 3: Model drift; the model has degraded due to software issues

Model drift is a general term, not specific to prediction distribution shift with unchanged inputs.

A
Data drift; the training data is no longer representative of the current environment
Why wrong: Data drift refers to input distribution change, which is not happening here.
B
Bias drift; the model is making unfair predictions against a protected group
Why wrong: Bias drift relates to fairness metrics, not overall prediction distribution.
C
Concept drift; the underlying relationship between features and target has changed
Concept drift changes the prediction function while input distribution remains stable.
D
Model drift; the model has degraded due to software issues
Why wrong: Model drift is a general term, not specific to prediction distribution shift with unchanged inputs.

Question 6hardmultiple choice

A company uses SageMaker Model Monitor for data quality. They notice that monitoring jobs are failing intermittently with constraint violations. Upon review, they see that some features have different data types in production compared to the baseline (e.g., string instead of integer). Which type of drift is this?

Trap 1: Concept drift

Concept drift is a change in the relationship between features and the target variable, not in the input data structure.

Trap 2: Statistical drift

Statistical drift involves changes in the distribution (e.g., mean, variance) of feature values, not their data types.

Trap 3: Bias drift

Bias drift refers to changes in fairness metrics, not data types.

A
Schema drift
Schema drift involves changes in the schema, such as data type mismatches, new or missing columns.
B
Concept drift
Why wrong: Concept drift is a change in the relationship between features and the target variable, not in the input data structure.
C
Statistical drift
Why wrong: Statistical drift involves changes in the distribution (e.g., mean, variance) of feature values, not their data types.
D
Bias drift
Why wrong: Bias drift refers to changes in fairness metrics, not data types.

Question 7hardmultiple choice

A machine learning engineer is building a time-series forecasting model to predict daily sales for the next 30 days. The dataset spans two years of daily sales data. To evaluate model performance, the engineer needs to simulate a realistic forecasting scenario where the model is trained on past data and tested on future data without leakage. Which data splitting strategy should they use?

Trap 1: Hold-out validation using a random 80/20 split

Random splitting can introduce future data into the training set, leading to optimistic performance estimates.

Trap 2: Stratified sampling based on sales volume

Stratified sampling maintains class proportions but does not preserve time order.

Trap 3: k-fold cross-validation with random shuffling

Random shuffling destroys temporal order, causing data leakage from the future into the training set.

A
Hold-out validation using a random 80/20 split
Why wrong: Random splitting can introduce future data into the training set, leading to optimistic performance estimates.
B
Walk-forward validation with an expanding window
Walk-forward validation trains on all past data and evaluates on the next unobserved time step, mimicking real-world forecasting.
C
Stratified sampling based on sales volume
Why wrong: Stratified sampling maintains class proportions but does not preserve time order.
D
k-fold cross-validation with random shuffling
Why wrong: Random shuffling destroys temporal order, causing data leakage from the future into the training set.

Question 8mediummultiple choice

A machine learning engineer needs to prepare a dataset with a target variable that has severe class imbalance (1:1000). The dataset has 100,000 rows and 200 features. Which approach should the engineer use to address the class imbalance before training a classification model?

Trap 1: Apply random undersampling to the majority class to match the…

Undersampling would discard 99.9% of data, losing valuable information.

Trap 2: Set class weights inversely proportional to class frequencies in…

Class weights are a model training technique, not a data preparation step.

Trap 3: Use StandardScaler on the features to normalize them.

StandardScaler normalizes features but does not address class imbalance.

A
Use SMOTE to generate synthetic samples for the minority class.
SMOTE creates synthetic examples, balancing the classes without losing majority data.
B
Apply random undersampling to the majority class to match the minority class count.
Why wrong: Undersampling would discard 99.9% of data, losing valuable information.
C
Set class weights inversely proportional to class frequencies in the model.
Why wrong: Class weights are a model training technique, not a data preparation step.
D
Use StandardScaler on the features to normalize them.
Why wrong: StandardScaler normalizes features but does not address class imbalance.

Question 9hardmultiple choice

A financial services company deploys multiple models on a single Amazon SageMaker endpoint using a multi-model endpoint (MME). The models are stored in Amazon S3. Each model is approximately 500 MB and is loaded on demand. Users report high latency for cold-start scenarios. What should the company do to reduce cold-start latency?

Trap 1: Reduce the instance size to increase the number of instances per…

Smaller instances may have less memory, increasing disk swapping and latency.

Trap 2: Increase the number of instances in the endpoint's auto-scaling…

More instances spread the load but each still may have cold starts.

Trap 3: Deploy each model on a separate endpoint to avoid concurrent…

This increases management overhead and cost, and doesn't directly address cold start.

A
Reduce the instance size to increase the number of instances per unit cost.
Why wrong: Smaller instances may have less memory, increasing disk swapping and latency.
B
Increase the number of instances in the endpoint's auto-scaling group.
Why wrong: More instances spread the load but each still may have cold starts.
C
Deploy each model on a separate endpoint to avoid concurrent loading.
Why wrong: This increases management overhead and cost, and doesn't directly address cold start.
D
Configure the endpoint to use a larger 'ModelCacheSize' parameter.
Increasing the model cache size allows more models to be cached in memory, reducing load time.

Question 10hardmultiple choice

A data scientist is preparing a large dataset (50 GB) for training a TensorFlow model on SageMaker. The dataset consists of many small CSV files. Training is slow due to I/O bottlenecks. Which data preparation strategy most effectively accelerates training?

Trap 1: Convert the dataset to Parquet format and use Apache Arrow for…

Parquet is good for analytics but not as efficient for TensorFlow training as TFRecord.

Trap 2: Compress the CSV files and decompress during data loading

Decompression adds overhead and does not solve file fragmentation.

Trap 3: Use a larger instance type with more vCPUs

This does not address I/O bottlenecks from many small files.

A
Convert the dataset to TFRecord format and use tf.data pipeline with prefetching
TFRecord combines many records into a few large files, and prefetching improves data pipeline efficiency.
B
Convert the dataset to Parquet format and use Apache Arrow for loading
Why wrong: Parquet is good for analytics but not as efficient for TensorFlow training as TFRecord.
C
Compress the CSV files and decompress during data loading
Why wrong: Decompression adds overhead and does not solve file fragmentation.
D
Use a larger instance type with more vCPUs
Why wrong: This does not address I/O bottlenecks from many small files.

Question 11mediummultiple choice

A machine learning engineer is troubleshooting a model that is producing unexpectedly low accuracy in production. The engineer examines the model's training data and finds that the distribution of the target variable in production is significantly different from the training set. What type of drift is the model experiencing?

Trap 1: Prior probability shift

Prior probability shift is a specific case of concept drift where class proportions change.

Trap 2: Data drift

Data drift is a broad term; concept drift is more specific.

Trap 3: Covariate shift

Covariate shift refers to changes in input features, not the target.

A
Prior probability shift
Why wrong: Prior probability shift is a specific case of concept drift where class proportions change.
B
Concept drift
Concept drift is a change in the statistical properties of the target variable.
C
Data drift
Why wrong: Data drift is a broad term; concept drift is more specific.
D
Covariate shift
Why wrong: Covariate shift refers to changes in input features, not the target.

Question 12hardmultiple choice

A company needs to deploy a large language model (LLM) on SageMaker with the Triton Inference Server to maximize GPU utilization and reduce latency. They have an NVIDIA A100 GPU. Which SageMaker inference option supports Triton?

Question 13easymultiple choice

A company wants to use SageMaker to deploy a model that requires GPU acceleration for inference but also needs to keep costs low when traffic is low. Which SageMaker feature should they use?

Trap 1: SageMaker Debugger

Debugger is for debugging training jobs.

Trap 2: SageMaker Managed Spot Training

Spot Training is for training jobs, not inference.

Trap 3: SageMaker Model Monitor

Model Monitor is for monitoring inference quality.

A
SageMaker Debugger
Why wrong: Debugger is for debugging training jobs.
B
SageMaker Managed Spot Training
Why wrong: Spot Training is for training jobs, not inference.
C
SageMaker Elastic Inference
Elastic Inference attaches GPU acceleration to any SageMaker instance, reducing cost.
D
SageMaker Model Monitor
Why wrong: Model Monitor is for monitoring inference quality.