Scenario practice questions

A data scientist at a retail company is tasked with building a model to predict customer churn. The dataset contains 100,000 records with features such as age, purchase history, customer support interactions, and a binary label indicating whether the customer churned in the past. The team needs a model that can be deployed for real-time inference with low latency. They have limited time and want to use a built-in algorithm from Amazon SageMaker that is optimized for classification tasks. Which approach should they take?

Trap 1: Use Amazon SageMaker PCA algorithm

PCA is for dimensionality reduction, not for building a classification model.

Trap 2: Use Amazon SageMaker K-Means algorithm

K-Means is an unsupervised clustering algorithm, not for supervised classification.

Trap 3: Use Amazon SageMaker BlazingText algorithm

BlazingText is designed for text data, not tabular customer churn data.

A
Use Amazon SageMaker PCA algorithm
Why wrong: PCA is for dimensionality reduction, not for building a classification model.
B
Use Amazon SageMaker XGBoost algorithm
XGBoost is a built-in algorithm for classification and works well with tabular data.
C
Use Amazon SageMaker K-Means algorithm
Why wrong: K-Means is an unsupervised clustering algorithm, not for supervised classification.
D
Use Amazon SageMaker BlazingText algorithm
Why wrong: BlazingText is designed for text data, not tabular customer churn data.

Question 2mediummultiple choice

A data scientist is training a binary classification model to predict customer churn. The dataset has 10,000 records with 9,500 non-churners and 500 churners. After training a logistic regression model, the model achieves 95% accuracy on the test set. However, the business team reports that the model is not useful because it predicts almost all customers as non-churners. Which metric should the data scientist use to evaluate the model's performance in this scenario?

Trap 1: Accuracy

Accuracy is not suitable for imbalanced datasets as it can be high even if the model predicts the majority class only.

Trap 2: R-squared

R-squared is a metric for regression models, not classification.

Trap 3: Precision

Precision measures how many of the predicted churners are actual churners, but it does not reflect how many actual churners were missed.

A
Accuracy
Why wrong: Accuracy is not suitable for imbalanced datasets as it can be high even if the model predicts the majority class only.
B
R-squared
Why wrong: R-squared is a metric for regression models, not classification.
C
Precision
Why wrong: Precision measures how many of the predicted churners are actual churners, but it does not reflect how many actual churners were missed.
D
Recall
Recall measures the proportion of actual churners correctly identified, which is the key metric for this imbalanced problem.

Question 3mediummultiple choice

A data scientist is training a model using Amazon SageMaker and notices the training loss is decreasing but validation loss starts increasing after a few epochs. Which technique should they apply to address this?

Trap 1: Increase batch size

Increasing batch size can speed up training but does not directly address overfitting.

Trap 2: Increase the learning rate

Increasing learning rate might worsen the problem and cause divergence.

Trap 3: Add more training data

More data can help generalize but is not the direct solution for overfitting; regularization is more effective.

A
Increase batch size
Why wrong: Increasing batch size can speed up training but does not directly address overfitting.
B
Increase the learning rate
Why wrong: Increasing learning rate might worsen the problem and cause divergence.
C
Add more training data
Why wrong: More data can help generalize but is not the direct solution for overfitting; regularization is more effective.
D
Add regularization (e.g., L1 or L2)
Regularization penalizes large weights and reduces overfitting, which is indicated by increasing validation loss.

Question 4hardmultiple choice

A company is developing an AI system to screen job applications. They want to ensure the system does not discriminate against candidates based on gender. The dataset used contains historical hiring decisions that may reflect past biases. Which type of bias is MOST likely present in this scenario?

Trap 1: Measurement bias

Measurement bias relates to how features are measured; not the primary issue.

Trap 2: Representation bias

Representation bias occurs when certain groups are underrepresented in the data; here the issue is historical decisions.

Trap 3: Aggregation bias

Aggregation bias occurs when groups are merged inappropriately; not directly applicable.

A
Historical bias
Historical bias arises from past inequalities reflected in the training data.
B
Measurement bias
Why wrong: Measurement bias relates to how features are measured; not the primary issue.
C
Representation bias
Why wrong: Representation bias occurs when certain groups are underrepresented in the data; here the issue is historical decisions.
D
Aggregation bias
Why wrong: Aggregation bias occurs when groups are merged inappropriately; not directly applicable.

Question 5easymulti select

A data scientist needs to use Amazon SageMaker to train a model and must ensure that the training data and the model artifacts are encrypted using customer-managed KMS keys. Which TWO resources can be encrypted with KMS keys in this scenario? (Select TWO.)

Trap 1: SageMaker notebook instances

Notebook instances can use KMS, but the question asks for training data and model artifacts.

Trap 2: Amazon SageMaker Model Registry

Model Registry metadata is stored in DynamoDB and can be encrypted, but not typically for training data/artifacts.

Trap 3: Amazon CloudWatch Logs

CloudWatch Logs can be encrypted with KMS, but not directly related to training data or model artifacts.

A
SageMaker ML storage volume (attached to training instances)
SageMaker encrypts the local storage volume using a KMS key.
B
SageMaker notebook instances
Why wrong: Notebook instances can use KMS, but the question asks for training data and model artifacts.
C
Amazon SageMaker Model Registry
Why wrong: Model Registry metadata is stored in DynamoDB and can be encrypted, but not typically for training data/artifacts.
D
Amazon CloudWatch Logs
Why wrong: CloudWatch Logs can be encrypted with KMS, but not directly related to training data or model artifacts.
E
S3 buckets storing training data
S3 can use SSE-KMS for encrypting objects.

Question 6hardmultiple choice

An e-commerce company stores user interaction logs in Amazon S3. They want to use machine learning to segment users based on purchasing behavior. Which unsupervised learning algorithm is most appropriate?

Trap 1: Linear regression

Supervised learning algorithm for predicting continuous values.

Trap 2: Random forest

Supervised ensemble learning method.

Trap 3: Neural network

Can be used but typically supervised; overkill for simple segmentation.

A
Linear regression
Why wrong: Supervised learning algorithm for predicting continuous values.
B
Random forest
Why wrong: Supervised ensemble learning method.
C
K-means clustering
Unsupervised algorithm that groups data into clusters based on similarity.
D
Neural network
Why wrong: Can be used but typically supervised; overkill for simple segmentation.

Question 7easymultiple choice

After deploying a model, a company notices that the distribution of the input features has shifted compared to the training data. Which feature of Amazon SageMaker Model Monitor can alert them to this change?

Trap 1: Model quality monitoring

Model quality monitors prediction accuracy against ground truth, not feature drift.

Trap 2: Bias drift monitoring

Bias drift monitors fairness metrics, not input feature distributions.

Trap 3: Feature importance drift

SageMaker Model Monitor does not have a built-in feature importance drift monitor.

A
Model quality monitoring
Why wrong: Model quality monitors prediction accuracy against ground truth, not feature drift.
B
Bias drift monitoring
Why wrong: Bias drift monitors fairness metrics, not input feature distributions.
C
Feature importance drift
Why wrong: SageMaker Model Monitor does not have a built-in feature importance drift monitor.
D
Data quality monitoring
Data quality monitors for drift in input features (baseline vs. live).

Question 8hardmultiple choice

A machine learning engineer notices that the training loss decreases steadily, but the validation loss starts increasing after a few epochs. Which of the following is the MOST likely cause?

Trap 1: Learning rate too low

A low learning rate would cause slow convergence but not a divergence between training and validation loss.

Trap 2: Underfitting

Underfitting would show high loss on both training and validation sets.

Trap 3: Data leakage from validation set into training set

Data leakage would typically cause both losses to be artificially low, not a divergence.

A
Learning rate too low
Why wrong: A low learning rate would cause slow convergence but not a divergence between training and validation loss.
B
Overfitting
Classic overfitting: model memorizes training data and fails to generalize to validation data.
C
Underfitting
Why wrong: Underfitting would show high loss on both training and validation sets.
D
Data leakage from validation set into training set
Why wrong: Data leakage would typically cause both losses to be artificially low, not a divergence.

Question 9hardmultiple choice

A financial services company is deploying a foundation model on Amazon Bedrock to generate compliance reports from internal audit logs. The model must not output any personally identifiable information (PII). They have configured a Bedrock Guardrail with sensitive information filters set to the 'HIGH' sensitivity level. During testing in a staging environment, testers still observed PII being occasionally generated in the report outputs. The guardrail did not block these instances because the PII was embedded in a context that the guardrail's pattern matching did not catch (e.g., structured JSON data with embedded names). The company requires a solution that minimizes latency and cost, as they process thousands of reports daily. They cannot afford to increase inference time significantly due to strict SLAs. They also want to avoid re-engineering the entire solution. Which additional step should they take to effectively eliminate PII leakage while maintaining performance?

Trap 1: Add a prompt instruction to the model to never output PII, with…

Wrong: Prompt instructions are unreliable and can be ignored by the model; the issue already occurs despite guardrails, so prompting alone won't suffice.

Trap 2: Fine-tune the foundation model on a dataset that excludes PII.

Wrong: Fine-tuning is costly and time-consuming, and may not completely eliminate PII leakage as the model could still generate PII from patterns.

Trap 3: Increase the guardrail sensitivity to 'MAXIMUM'.

Wrong: Maximum sensitivity may increase false positives but still may not catch context-based PII; also may block legitimate content.

A
Add a prompt instruction to the model to never output PII, with few-shot examples of non-PII outputs.
Why wrong: Wrong: Prompt instructions are unreliable and can be ignored by the model; the issue already occurs despite guardrails, so prompting alone won't suffice.
B
Fine-tune the foundation model on a dataset that excludes PII.
Why wrong: Wrong: Fine-tuning is costly and time-consuming, and may not completely eliminate PII leakage as the model could still generate PII from patterns.
C
Increase the guardrail sensitivity to 'MAXIMUM'.
Why wrong: Wrong: Maximum sensitivity may increase false positives but still may not catch context-based PII; also may block legitimate content.
D
Implement a post-processing Lambda function that uses Amazon Comprehend's PII detection to scan and redact any PII from the model output before returning it.
Correct: Amazon Comprehend provides robust PII detection that can catch context-based PII. The Lambda function can be optimized for low latency and added cost is minimal.

Question 10hardmultiple choice

An organization uses Amazon Bedrock to generate content. They have implemented guardrails to block toxic content. However, some users are able to bypass the guardrails by encoding their prompts. What step should be taken to improve security?

Trap 1: Encode the prompts before sending to the model.

Encoding prompts does not prevent bypass; the model would decode and potentially still produce toxic content.

Trap 2: Use a different foundation model that is less susceptible.

Model susceptibility is not the root cause; guardrails can be bypassed regardless.

Trap 3: Restrict access to the model using IAM policies.

Access control does not prevent authorized users from submitting encoded prompts.

A
Encode the prompts before sending to the model.
Why wrong: Encoding prompts does not prevent bypass; the model would decode and potentially still produce toxic content.
B
Enable prompt injection detection in the guardrail configuration.
Prompt injection detection can identify and block encoded or malicious prompts.
C
Use a different foundation model that is less susceptible.
Why wrong: Model susceptibility is not the root cause; guardrails can be bypassed regardless.
D
Restrict access to the model using IAM policies.
Why wrong: Access control does not prevent authorized users from submitting encoded prompts.