Scenario practice questions

A data scientist is training a binary classification model on an imbalanced dataset (95% negative class, 5% positive class). The model achieves 95% accuracy but only predicts the negative class for all examples. Which metric should the scientist use to evaluate model performance more appropriately?

Trap 1: F1 score

F1 score is useful but may still be high if the model predicts only the majority class; AUC-ROC is more robust.

Trap 2: Mean squared error

MSE is for regression tasks, not classification.

Trap 3: Accuracy

Accuracy can be high even if the model always predicts the majority class, which is misleading.

A
F1 score
Why wrong: F1 score is useful but may still be high if the model predicts only the majority class; AUC-ROC is more robust.
B
Mean squared error
Why wrong: MSE is for regression tasks, not classification.
C
Accuracy
Why wrong: Accuracy can be high even if the model always predicts the majority class, which is misleading.
D
AUC-ROC
AUC-ROC evaluates the model's ability to distinguish between classes regardless of threshold and is robust to imbalance.

Question 2mediummultiple choice

A company is building a recommendation system for an e-commerce platform. The data includes user-item interactions and features such as user demographics and item categories. Which algorithm would be most appropriate for generating personalized recommendations?

Trap 1: XGBoost

XGBoost is a tree-based algorithm that does not naturally model feature interactions; it requires manual feature engineering to capture interactions.

Trap 2: k-means clustering

k-means groups similar users or items but does not directly produce personalized recommendations for each user.

Trap 3: Principal Component Analysis (PCA)

PCA is used for dimensionality reduction, not for modeling user-item interactions.

A
XGBoost
Why wrong: XGBoost is a tree-based algorithm that does not naturally model feature interactions; it requires manual feature engineering to capture interactions.
B
Factorization Machines
Factorization Machines model pairwise feature interactions and work well with sparse data, making them suitable for recommendation systems.
C
k-means clustering
Why wrong: k-means groups similar users or items but does not directly produce personalized recommendations for each user.
D
Principal Component Analysis (PCA)
Why wrong: PCA is used for dimensionality reduction, not for modeling user-item interactions.

Question 3easymultiple choice

A machine learning engineer is training a regression model to predict house prices using Amazon SageMaker. The dataset contains 10,000 samples and 50 numerical features. After training a linear regression model, the engineer notices that the training loss is low, but the validation loss is high. The engineer suspects overfitting. The dataset is already normalized. Which action should the engineer take to reduce overfitting?

Trap 1: Increase the learning rate to speed up convergence.

Incorrect: Higher learning rate may overshoot minima and worsen performance.

Trap 2: Reduce the number of features using PCA.

PCA reduces dimensionality by projecting features onto orthogonal components that maximise variance, but this does not directly address overfitting caused by the model memorising noise in the training data. The engineer already has normalised numerical features, and PCA may discard information relevant to house price prediction while failing to penalise large coefficients. It is tempting because PCA is commonly used to simplify high-dimensional datasets and can reduce variance in some models, but here regularisation (e.g., L2) would directly constrain coefficient magnitudes to combat overfitting.

Trap 3: Decrease the mini-batch size during training.

Incorrect: Smaller batch size adds noise but is not a standard regularization technique.

A
Increase the learning rate to speed up convergence.
Why wrong: Incorrect: Higher learning rate may overshoot minima and worsen performance.
B
Reduce the number of features using PCA.
Why wrong: PCA reduces dimensionality by projecting features onto orthogonal components that maximise variance, but this does not directly address overfitting caused by the model memorising noise in the training data. The engineer already has normalised numerical features, and PCA may discard information relevant to house price prediction while failing to penalise large coefficients. It is tempting because PCA is commonly used to simplify high-dimensional datasets and can reduce variance in some models, but here regularisation (e.g., L2) would directly constrain coefficient magnitudes to combat overfitting.
C
Add L2 regularization (weight decay) to the loss function.
Correct: L2 regularization penalizes large weights and reduces overfitting.
D
Decrease the mini-batch size during training.
Why wrong: Incorrect: Smaller batch size adds noise but is not a standard regularization technique.

Question 4easymultiple choice

A company is building a recommendation system using collaborative filtering. The dataset contains implicit feedback (clicks) from users on items. Which algorithm is best suited for this scenario?

Trap 1: Linear Regression

Linear Regression is a supervised regression algorithm, not designed for collaborative filtering with implicit feedback.

Trap 2: K-means clustering

K-means clustering is for grouping similar items, not for generating recommendations from implicit feedback.

Trap 3: Singular Value Decomposition (SVD)

Singular Value Decomposition (SVD) typically works with explicit ratings, not implicit feedback like clicks.

A
Linear Regression
Why wrong: Linear Regression is a supervised regression algorithm, not designed for collaborative filtering with implicit feedback.
B
Alternating Least Squares (ALS)
Alternating Least Squares (ALS) is specifically designed for implicit feedback datasets in collaborative filtering, making it the best choice.
C
K-means clustering
Why wrong: K-means clustering is for grouping similar items, not for generating recommendations from implicit feedback.
D
Singular Value Decomposition (SVD)
Why wrong: Singular Value Decomposition (SVD) typically works with explicit ratings, not implicit feedback like clicks.

Question 5easymultiple choice

A company is building a model to classify customer reviews as positive or negative. The dataset has 10,000 positive and 100 negative reviews. Which metric is most appropriate for evaluating model performance?

Trap 1: Accuracy.

Accuracy is not suitable for imbalanced classes.

Trap 2: Mean squared error.

MSE is for regression, not classification.

Trap 3: AUC-ROC.

AUC-ROC may be too optimistic for severe imbalance.

A
F1 score.
F1 score considers both precision and recall, good for imbalance.
B
Accuracy.
Why wrong: Accuracy is not suitable for imbalanced classes.
C
Mean squared error.
Why wrong: MSE is for regression, not classification.
D
AUC-ROC.
Why wrong: AUC-ROC may be too optimistic for severe imbalance.

Question 6mediummultiple choice

A data engineer is performing EDA on a dataset containing user activity logs from a mobile app. The dataset has 10 million rows and includes columns: 'user_id', 'event_type', 'timestamp', 'device_type', and 'session_duration'. The engineer uses Amazon Athena to query the data stored in S3 as CSV files. The engineer runs a query to find the average session_duration per device_type, but the query takes over 5 minutes and scans 100 GB of data. The engineer wants to reduce query cost and improve performance for future EDA. The dataset is not partitioned, and the engineer anticipates frequent queries filtering on 'timestamp' and 'device_type'. Which action will most effectively reduce data scanned?

Trap 1: Use random sampling to query a subset of data.

Sampling reduces accuracy and may not be acceptable for EDA.

Trap 2: Convert the data to Parquet format and use columnar storage.

Parquet reduces bytes scanned but without partitioning, still full scan of columns.

Trap 3: Partition the table by device_type.

Partitioning by device_type helps only if filtering by device_type, not timestamp.

A
Partition the table by date derived from timestamp and convert to Parquet.
Combining partitioning and columnar storage maximizes reduction in scanned data.
B
Use random sampling to query a subset of data.
Why wrong: Sampling reduces accuracy and may not be acceptable for EDA.
C
Convert the data to Parquet format and use columnar storage.
Why wrong: Parquet reduces bytes scanned but without partitioning, still full scan of columns.
D
Partition the table by device_type.
Why wrong: Partitioning by device_type helps only if filtering by device_type, not timestamp.

Question 7hardmultiple choice

A company uses Amazon EMR to run Spark jobs on a large dataset stored in Amazon S3. The jobs are failing with 'OutOfMemoryError' in the executors. The data is not skewed. Which configuration change will most likely resolve the issue?

Trap 1: Enable Kryo serialization

Kryo reduces memory usage but may not be sufficient to resolve OOM if memory overhead is too low.

Trap 2: Decrease the number of shuffle partitions

Fewer partitions reduce memory usage but may cause data skew and does not directly address OOM.

Trap 3: Increase the number of executor cores

More cores can increase parallelism but not memory per executor.

A
Enable Kryo serialization
Why wrong: Kryo reduces memory usage but may not be sufficient to resolve OOM if memory overhead is too low.
B
Decrease the number of shuffle partitions
Why wrong: Fewer partitions reduce memory usage but may cause data skew and does not directly address OOM.
C
Increase the spark.executor.memoryOverhead setting
Memory overhead handles JVM overhead and off-heap memory, preventing OOM errors.
D
Increase the number of executor cores
Why wrong: More cores can increase parallelism but not memory per executor.

Question 8mediummultiple choice

A machine learning engineer trains a binary classifier on an imbalanced dataset where the positive class represents 1% of the data. After training, the model achieves 99% accuracy but only 10% recall on the positive class. Which metric should the engineer focus on to evaluate the model's performance on the minority class?

Trap 1: Accuracy

Accuracy is high due to class imbalance and does not reflect minority class performance.

Trap 2: AUC-ROC

AUC-ROC can be high even when recall is low if the model ranks positives well.

Trap 3: Precision

Precision alone may be high if few false positives, but recall could be low.

A
F1 score
F1 score considers both precision and recall, giving a better measure for imbalanced data.
B
Accuracy
Why wrong: Accuracy is high due to class imbalance and does not reflect minority class performance.
C
AUC-ROC
Why wrong: AUC-ROC can be high even when recall is low if the model ranks positives well.
D
Precision
Why wrong: Precision alone may be high if few false positives, but recall could be low.