ML Model Development practice questions

SageMaker Distributed Data Parallel (SMDDP)

Why it fails: SMDDP replicates the full model on every GPU and shards only the training data, so each device must still hold all parameters, gradients and optimiser states; it cannot relieve memory pressure when the model itself exceeds a single device. It is the right choice when per-device memory suffices and throughput is the bottleneck.

Data parallelism

Why it fails: Data parallelism keeps a complete model replica on each GPU and splits only the batch, so per-device memory consumption is unchanged and the model still cannot fit. It is tempting because it is the default multi-GPU approach and scales throughput well when each device can already hold the full model.

SageMaker Distributed Model Parallel (SMDMP)

Why it fails: SMDMP is correct but the option name is incomplete; however, model parallelism is the general strategy.

collection_configs with save_interval=500 and end_step=100

Why it fails: This would save every 500 steps and stop at step 100, which is the opposite.

collection_configs with start_step=100 and end_step=500

Why it fails: This would save every step between 100 and 500, not every 100 steps.

Use SageMaker Debugger rules to filter steps

Why it fails: Rules can detect issues but do not control tensor saving frequency.

ml.p3.2xlarge

Why it fails: The ml.p3.2xlarge provides a single V100 GPU with 16 GB of GPU memory, which constrains the batch sizes and sequence lengths LoRA fine-tuning of a large foundation model needs. It is tempting because P3 instances are genuinely built for GPU-accelerated deep learning training, so they would suit full fine-tuning workloads that fit within that memory.

ml.c5.2xlarge

Why it fails: The ml.c5.2xlarge is a compute-optimised instance with no GPU, so it cannot accelerate the matrix operations LoRA fine-tuning of a foundation model depends on. It is tempting because C5 instances are genuinely cost-effective for CPU-bound workloads such as data preprocessing, inference on small models, or traditional machine learning.

ml.trn1.2xlarge

Why it fails: The ml.trn1.2xlarge carries a single Trainium chip with 32 GB of accelerator memory, aimed at distributed training of very large models rather than memory-lean LoRA fine-tuning. It is tempting because Trainium instances are genuinely cost-optimised for training large models at scale, so they would suit high-throughput distributed training jobs.

Artifact

Why it fails: Artifacts are the individual inputs and outputs a trial component consumes or produces, such as datasets or model files; they carry no grouping role. The tempting pull is that artifacts do link lineage across a run, but organising related trials into a searchable collection is the experiment's job, not an artifact's.

Trial component

Why it fails: A trial component is a single step within a trial — one training job, processing job or evaluation — holding its own parameters and metrics. It is tempting because metrics and hyperparameters are recorded at this level, yet grouping related trials into a parent collection is performed by the experiment, not by any individual component.

Trial

Why it fails: A trial represents one run of a model-training attempt, holding its own metrics and parameters. It is tempting because trials are what you compare, but a trial is the unit being organised, not the organiser; the experiment groups related trials, while trial components subdivide a single trial.

Clarify with SageMaker Model Monitor

Why it fails: Model Monitor detects data drift and quality issues in deployed endpoints; it does not measure bias or explain feature reliance. It is tempting because both are Clarify-adjacent monitoring tools, but checking dependence on a sensitive attribute requires Clarify bias metrics such as SHAP values.

SHAP analysis

Why it fails: SHAP values explain each feature's contribution to individual predictions, quantifying how much gender influenced outputs. The stem asks to prevent reliance on a sensitive attribute, which requires measuring its correlation with the label before training, not post-hoc attribution. SHAP would suit model interpretability reporting after deployment.

Clarify processing job with pre-training bias metrics

Why it fails: Pre-training bias metrics measure imbalance in the dataset and label, not the trained model's dependence on gender. The stem requires post-training explanation of feature influence, which Clarify provides through SHAP-based analysis of the model. Pre-training metrics suit auditing data collection before any model is fitted.

Bias report generation

Why it fails: A bias report aggregates computed metrics into documentation; it does not itself measure whether the trained model relies on gender. The stem needs the underlying analysis that quantifies feature attribution, which the report merely summarises. Reports suit governance evidence after the appropriate metric has been selected and run.

strategy

Why it fails: strategy selects the tuning approach itself, not early-stopping aggressiveness. It is the parameter you set to Hyperband when you want multi-fidelity early stopping at all, so it would be the answer to a question asking which parameter enables the Hyperband algorithm.

max_jobs

Why it fails: max_jobs caps the total number of training jobs the tuning job runs. It is the parameter you raise to give Hyperband more trials to evaluate, so it answers a question about limiting overall tuning cost rather than controlling how eagerly poor trials are pruned.

max_parallel_jobs

Why it fails: max_parallel_jobs sets how many training jobs run concurrently. It is the parameter you tune to control throughput and resource contention, so it answers a question about parallel capacity, not about how aggressively Hyperband terminates underperforming trials.

Use on-demand instances for reliability

Why it fails: On-demand instances bill at full price and are not reclaimable, so they remove the discount that interruption-tolerant tuning jobs should capture. They are tempting when reliability is prioritised, and would be correct for jobs that cannot tolerate interruption or must meet strict completion deadlines.

Use ml.c5 instances instead of ml.p3

Why it fails: Swapping to compute-optimised instances changes the instance family, not the pricing model, so no interruption discount is gained. It is tempting because ml.c5 is cheaper per hour than ml.p3, and would be correct where GPU acceleration is unnecessary for the training workload.

Increase the number of parallel training jobs

Why it fails: More parallel jobs raise concurrency, not cost efficiency; each still bills at on-demand rates. Spot training with checkpointing is what cuts spend for interruptible jobs. Parallelism suits shortening wall-clock time when budget is not the constraint.

Value network

Why it fails: Incorrect. While some RL algorithms use a value network, the standard RLHF pipeline for LLMs requires a reference model instead. A value network is not essential.

Feature store

Why it fails: Incorrect. A feature store is a repository for storing and managing features for machine learning, but it is not a core component of the RLHF pipeline.

Hyperparameter tuner

Why it fails: Incorrect. A hyperparameter tuner helps optimize training parameters but is not essential to the RLHF process itself.

Tensor parallelism

Why it fails: Tensor parallelism shards individual layers and their matrix operations across GPUs, requiring heavy per-layer communication; it does not partition the model into sequential stages, so it fails the stated requirement. It is tempting because it does split model parameters across devices, but it suits intra-layer scaling within a node rather than stage-based splitting.

Data parallelism

Why it fails: Data parallelism replicates the full model on every GPU and splits only the batch, so a model too large for one GPU still cannot fit. It is tempting because the team already uses it and it scales throughput well, but it addresses batch size, not model size.

Pipeline parallelism

Why it fails: Pipeline parallelism is a form of model parallelism but not the only one; the question asks for a general strategy.

Overfit

Why it fails: The Overfit rule compares training and validation loss to flag overfitting, so it never inspects ReLU activations and cannot detect dead units. It is tempting because it is a genuine built-in Debugger rule for a common training pathology, and would be the right choice when validation loss diverges from training loss.

ExplodingGradients

Why it fails: ExplodingGradients monitors gradient magnitudes for spikes, so it detects unstable gradients rather than ReLU units that output zero. It is tempting because it is a real built-in Debugger rule addressing a well-known neural network training failure, and would be correct when gradients grow uncontrollably during training.

LossNotDecreasing

Why it fails: LossNotDecreasing tracks whether the loss fails to improve across steps, which indicates stalled optimisation rather than dead ReLU units. It is tempting because it is a genuine built-in Debugger rule for diagnosing training that is not converging, and would be the right choice when loss plateaus early.

Using different instance types across the cluster

Why it fails: Different instance types may cause stragglers but it's not a typical cause for non-linear scaling in a homogeneous cluster.

Model size too large for the GPUs

Why it fails: Model size exceeding GPU memory causes out-of-memory failures or forces smaller micro-batches, not sublinear scaling under data parallelism. It is tempting because model size genuinely dictates memory strategy, and sharding a model across devices via SageMaker model parallelism would be the right response when a single GPU cannot hold the model.

Feature importance (SHAP values) on the dataset

Why it fails: Incorrect. Feature importance (SHAP values) requires a trained model to compute and is thus a post-training analysis, not a pre-training analysis.

Model monitoring for data drift

Why it fails: Incorrect. Model monitoring for data drift is a separate SageMaker feature that monitors deployed models over time, not a pre-training analysis.

Explainability report for the model

Why it fails: Incorrect. Explainability reports (e.g., SHAP) require a trained model and are post-training.

Post-training bias metrics (e.g., difference in accuracy across groups)

Why it fails: Incorrect. Post-training bias metrics are computed after model training, not before.

Trial

Why it fails: A trial groups a single run's trial components, not the collection of runs sharing a configuration; it cannot itself span multiple hyperparameter variations. Trials are tempting because they hold the metrics and parameters you compare, but they are the unit being compared, not the container organising them.

Experiment

Why it fails: An experiment is the top-level container that groups trials, so it organises runs rather than being the component compared across hyperparameter configurations. It is tempting because it sounds like the overarching structure, but the question asks which component groups related runs for comparison within that container.

Trial Component

Why it fails: A trial component is a single step within a trial, such as preprocessing or training, so it cannot group multiple runs for comparison. It is tempting because metrics and hyperparameters attach to it, but grouping related runs across configurations requires the trial, which owns those components.

F1

Why it fails: F1 is the harmonic mean of precision and recall, so tuning it rewards recall gains as much as precision, pulling the objective away from the stated goal. It is tempting because F1 handles class imbalance well, but it would be correct only when both precision and recall matter equally.

AUC

Why it fails: AUC measures ranking quality across all classification thresholds, aggregating both positive and negative performance rather than optimising the precision of the deployed threshold. It is tempting for imbalanced data because it is threshold-independent, but it would be correct when overall discriminative ranking ability is the target.

Recall

Why it fails: Recall maximises the proportion of actual frauds caught, which directly trades away precision by accepting more false positives. It is tempting in fraud detection because missing fraud is costly, but recall is the correct objective when minimising false negatives is the priority.

Data parallelism

Why it fails: Data parallelism replicates the full model on every device, multiplying rather than reducing per-device memory, so it does not cut the footprint. It is tempting because it is the standard way to scale training throughput across GPUs, and would be correct when the goal is faster iteration on a model that already fits.

Gradient checkpointing

Why it fails: Gradient checkpointing recomputes activations during the backward pass instead of storing them, but it trades compute for memory rather than shrinking the optimiser or parameter state LoRA targets. It is tempting because it is a genuine memory-saving technique, and would be correct when activation storage, not model weights, dominates usage.

Mixed precision

Why it fails: Mixed precision shrinks activation and gradient storage via fp16/bf16, but LoRA's memory bottleneck is the frozen base weights and optimiser states, which mixed precision does not quantise. It is genuinely useful for speeding up full-precision training throughput, yet combining it with LoRA leaves the dominant footprint untouched.

Use SageMaker Autopilot

Why it fails: Autopilot builds and tunes models from tabular data; it does not import custom PyTorch artefacts. It is tempting because it is a SageMaker training feature, and would be correct when the goal is automated model selection on structured datasets rather than deploying user-supplied framework code.

Use the built-in Image Classification algorithm

Why it fails: The built-in Image Classification algorithm trains its own model from images; it cannot host a custom PyTorch script. It is tempting because it is a SageMaker training option for vision tasks, and would be correct if the team wanted a managed algorithm instead of bringing their own framework container.

Use a larger foundation model with a longer context window and paste all documents into each prompt

Why it fails: Pasting every policy document into each prompt consumes the context window and scales poorly as the corpus grows, and it cannot guarantee retrieval of the relevant clause. Long context suits one-off analysis of a bounded document set, not a monthly-updated knowledge base needing targeted lookup.

Fine-tune a base LLM on the policy documents monthly

Why it fails: Fine-tuning monthly bakes policy text into weights, so each update triggers another costly training run and the model cannot cite or verify sources. Fine-tuning suits teaching a model a persistent style or task format, not injecting facts that change every month.

Train a custom model from scratch on the policy documents each month

Why it fails: Training from scratch each month demands labelled data, GPU compute and ML expertise, and the resulting model still cannot cite source clauses. It would suit building a domain-specific model where no pretrained foundation exists, not a chatbot over frequently revised documents.

DeepAR

Why it fails: DeepAR is a supervised forecasting algorithm for time-series data, so it cannot perform binary classification on tabular rows. It is tempting because it is supervised and handles labelled sequences, but it would be correct when predicting future values in a time series.

K-Means

Why it fails: K-Means is unsupervised clustering, so it cannot use the binary labels required for classification. Its name and SageMaker availability make it tempting for grouping data, but it would be correct for segmenting unlabelled records into clusters, not for predicting a binary target.

Linear Learner

Why it fails: Linear Learner is a supervised algorithm, but it is designed for regression and linear classification on very large, high-dimensional sparse data; with 10,000 rows and 50 features, XGBoost handles this tabular binary classification task. Linear Learner would be correct for massive-scale linear problems or regression.

Pipeline parallelism

Why it fails: Pipeline parallelism is a form of model parallelism but not the primary term used in SageMaker; SageMaker’s model parallelism handles layer partitioning.

Sharded data parallelism

Why it fails: Sharded data parallelism (like ZeRO) reduces memory but still requires the model to fit across GPUs; for a 1B parameter model, pure data parallelism may be insufficient.

Data parallelism

Why it fails: Data parallelism replicates the entire model on each GPU; a 1B parameter model likely exceeds GPU memory.