Core Machine Learning and AI Knowledge practice questions

Stochastic Gradient Descent always converges faster than Adam in all neural network architectures.

Why it fails: Adam is generally faster to converge because it utilizes adaptive learning rates for each parameter, effectively handling sparse gradients and non-stationary objectives. SGD often requires extensive manual tuning of the learning rate schedule and may struggle with complex loss surfaces compared to adaptive methods like Adam.

Momentum-based optimization has no effect on the speed of convergence in deep learning.

Why it fails: Momentum is designed to accelerate gradients in directions of consistent descent while dampening oscillations in high-curvature directions. It acts like a ball rolling down a hill, accumulating velocity, which is crucial for overcoming plateaus and noise during the optimization process in deep networks.

Mini-batch SGD is inherently less efficient than using the full dataset for every update.

Why it fails: Mini-batching provides the best trade-off between the stability of Batch Gradient Descent and the speed of pure SGD. It exploits parallel computation on GPUs, allowing for faster iterations and better memory utilization, which is the standard practice for training modern machine learning models.

Sigmoid

Why it fails: Sigmoid maps outputs to a range between 0 and 1. As inputs become large, the derivative approaches zero, causing the vanishing gradient problem. This makes it unsuitable for deep architectures, as the signal from the loss function fails to propagate back through many layers during backpropagation.

ReLU

Why it fails: ReLU is computationally efficient and helps mitigate vanishing gradients for positive inputs. However, it suffers from the 'dying ReLU' problem where neurons can become inactive and output zero for all inputs, preventing them from updating further. While better than sigmoid, it is often superseded by smoother functions.

Softmax

Why it fails: Softmax is typically used as the final layer in classification tasks to convert raw scores into probabilities. It is not used as an activation function for hidden layers because it is designed to normalize a vector into a probability distribution, which is not appropriate for internal feature representations.

Improved model accuracy by eliminating all quantization noise.

Why it fails: Mixed precision actually introduces quantization noise because FP16 has a lower dynamic range than FP32. While the impact on final model accuracy is usually negligible with proper loss scaling, it does not improve accuracy by 'eliminating' noise; rather, it manages the trade-off between speed and numerical precision.

Automatic conversion of all operations to 8-bit integers.

Why it fails: Mixed precision training utilizes FP16/BF16 and FP32, not 8-bit integers. 8-bit quantization is a separate optimization technique usually applied during inference (post-training quantization) rather than during the training process, where higher precision is required to maintain the stability of gradient calculations.

It replaces the feed-forward network entirely in the transformer block.

Why it fails: Multi-head attention is only one component of the transformer block. The feed-forward network, which follows the attention layer, is essential for applying non-linear transformations to the features. Both components work together to process the information, and one cannot function effectively without the other in modern architectures.

It ensures that the model can only attend to the immediate previous token.

Why it fails: The attention mechanism is global, meaning each token can potentially attend to any other token in the sequence, regardless of distance. Causal masking may limit this during training for generation, but the fundamental MHA mechanism is designed to handle long-range dependencies across the entire sequence.

It is computationally slower than an RNN for sequential tasks.

Why it fails: Transformers, through MHA, are highly parallelizable, whereas RNNs must be processed sequentially. This parallelization is a major advantage of the transformer architecture, allowing it to take full advantage of GPU hardware throughput and process significantly more data in a shorter period compared to traditional RNNs.

Increase the learning rate linearly with the number of GPUs.

Why it fails: Increasing the learning rate alone does not address the underlying communication overhead or synchronization delays. While scaling rules suggest larger batches might require higher learning rates, this change does not reduce the frequency of gradient exchange operations during backpropagation, failing to resolve the specific hang issues described.

Transition to synchronous Stochastic Gradient Descent with all-reduce.

Why it fails: Synchronous SGD using standard all-reduce operations is exactly what causes the synchronization bottleneck in high-latency environments. Moving to a synchronous method does not mitigate the wait time incurred by slow nodes or network congestion, but rather exacerbates the performance impact when multiple GPUs must wait for global synchronization.

Switch from FP16 to FP32 precision to improve numerical stability.

Why it fails: Switching to FP32 increases the memory footprint and the volume of data transmitted during synchronization, which would worsen rather than solve the gradient synchronization bottleneck. Precision settings influence model convergence and hardware utilization, but they do not resolve the latency issues caused by frequent communication cycles between distributed GPUs.

The model weights are corrupted in the checkpoint file.

Why it fails: Checkpoint corruption usually triggers deserialization errors or value errors during model loading, not NCCL system errors. The logs explicitly point to an infrastructure topology mismatch during initialization, indicating the issue lies in the cluster environment setup rather than the state of the saved model weights on disk.

Insufficient VRAM on Rank 0 causing OOM killer.

Why it fails: Out-of-memory errors on NVIDIA hardware are typically reported through CUDA errors or explicit memory allocation exceptions, not as an unhandled system error in the NCCL layer. The logs indicate a communication structure failure, confirming that the cluster environment configuration is the primary culprit rather than per-GPU memory capacity.

Incompatible CUDA toolkit version across the cluster.

Why it fails: While version mismatches can cause runtime issues, they usually manifest as library loading errors or feature incompatibility warnings. The provided log specifically highlights a topology mismatch regarding the number of expected versus found nodes and GPUs, which is a structural deployment error rather than a software version incompatibility.

Enable dynamic shape support for all input layers.

Why it fails: Enabling dynamic shapes across all layers forces the runtime to perform additional checks and prevents specific static graph optimizations. While necessary for some models, it generally adds overhead and reduces the effectiveness of kernel fusion, ultimately decreasing the overall inference throughput compared to fixed-shape execution models.

Always prioritize FP32 precision to maintain maximum accuracy.

Why it fails: Prioritizing FP32 prevents the usage of high-speed Tensor Cores, which are optimized for lower precision formats. Using FP32 ignores the hardware acceleration capabilities of the A100 architecture, leading to suboptimal inference speeds without significant accuracy gains for many modern generative models that are robust to reduced precision.

Bypass the build phase and use direct PyTorch execution.

Why it fails: Bypassing the TensorRT build phase means missing out on engine-specific optimizations like kernel autotuning, workspace management, and graph rewriting. PyTorch eager mode lacks the compiler-level graph analysis provided by TensorRT, resulting in inferior latency and throughput when compared to a fully optimized, serialized TensorRT engine plan.

Upsampling all datasets to have an equal number of samples.

Why it fails: Forcing equal sample counts across heterogeneous datasets can lead to significant overfitting on smaller, potentially low-quality datasets. Balanced training should be achieved through strategic sampling weights rather than artificial upsampling, which can introduce noise and degrade the overall model's performance on its primary intended domain.

Manually labeling all 10 trillion tokens in the corpus.

Why it fails: Manually labeling a 10 trillion token corpus is physically impossible and economically infeasible. Foundation models rely on self-supervised learning on massive, unlabeled data. Manual intervention is only applicable for specific fine-tuning stages, such as RLHF, rather than the initial, large-scale data pre-processing phase of the training pipeline.

Automatic translation of model code into C++.

Why it fails: NeMo operates primarily within the Python ecosystem using PyTorch. It does not automatically translate model logic into C++. While it utilizes highly optimized C++ and CUDA-based backends for its kernels, the user-facing interface remains Python-centric, which is standard for the vast majority of machine learning research and engineering workflows.

Direct access to proprietary model weights.

Why it fails: NeMo is a framework for building, training, and deploying models; it does not inherently grant users access to proprietary weights from third-party model providers. Users must bring their own models or use open-weight models; NeMo provides the infrastructure to handle these models, not the private intellectual property of other companies.

Batch gradient descent is highly efficient for datasets with billions of parameters.

Why it fails: Batch gradient descent processes the entire dataset before every weight update, making it computationally prohibitive for large-scale models. The memory overhead and time required for a single update step make it impractical compared to stochastic or mini-batch variants, which provide much faster iterations and better convergence characteristics.

Gradient descent is only applicable to convex loss functions.

Why it fails: Deep neural networks involve non-convex loss functions with many local minima and saddle points. Despite this, gradient-based methods are extremely effective in practice. While there is no guarantee of finding the global optimum, these methods reliably find high-quality solutions that perform exceptionally well on complex, real-world tasks.

Learning rate is not a hyperparameter for gradient descent.

Why it fails: The learning rate is a critical hyperparameter that controls the step size taken during each update. If the rate is too high, the model may diverge; if it is too low, training becomes painfully slow. Proper tuning of this parameter is essential for achieving optimal convergence in any network.

Active Learning

Why it fails: Active learning is a technique where the model identifies which data points it needs to be labeled to improve its performance the most. It is about data selection during training, not about using a pre-trained model as a foundation for a new, specific target task as described in the scenario.

Feature Engineering

Why it fails: Feature engineering refers to the manual process of transforming raw data into features that better represent the underlying problem for machine learning models. It is a data preprocessing technique and does not involve the use of pre-trained models or the transfer of learned knowledge between domains as described.

Reinforcement Learning

Why it fails: Reinforcement learning is based on agents maximizing a reward through environmental interaction. It is not a method for adapting pre-trained weights for a different dataset task. While reinforcement learning from human feedback (RLHF) is used for fine-tuning LLMs, the concept of using a base model is specifically called transfer learning.

Increasing the batch size infinitely

Why it fails: Increasing batch size beyond hardware memory limits results in out-of-memory errors. While large batches can stabilize gradients, they do not inherently fix convergence issues caused by poor learning rate scheduling. Furthermore, excessively large batches may lead to 'sharp minima' generalization issues, negatively impacting the model's performance on unseen validation data.

Decreasing the model architecture size

Why it fails: Reducing model size reduces capacity, which likely exacerbates plateauing by limiting the model's ability to learn complex representations. The issue is usually optimization-related rather than capacity-related. Shrinking the model is a strategy for deployment efficiency, not a remedy for stalling training convergence in large transformer architectures.

Switching from FP16 to FP64 precision

Why it fails: FP64 precision requires double the memory bandwidth and compute compared to FP32, making training significantly slower without providing benefits for standard neural network convergence. Most LLMs converge perfectly fine with mixed precision (FP16/BF16). Increasing precision does not address optimization dynamics or learning rate schedules that cause training loss to plateau prematurely.

The model will always exhibit higher variance

Why it fails: Variance is not inherently linked to class imbalance; it is linked to model complexity and data size. While a model might struggle to learn rare classes, this is a bias issue, not a variance issue. High variance would suggest the model is overfitting, which can happen in any dataset regardless of class distribution.

The training will always lead to gradient explosion

Why it fails: Gradient explosion is a consequence of network architecture, initialization, and optimizer settings, not the data distribution. While imbalanced data causes poor convergence on specific classes, it does not directly force the gradients to grow to non-finite values. These are two distinct classes of problems in model training.

The model will fail to utilize GPU acceleration

Why it fails: GPU acceleration is independent of data distribution. Neural networks use the same linear algebra operations and hardware utilization regardless of whether the classes are balanced or imbalanced. Training with imbalanced data consumes the same amount of GPU resources as balanced data, provided the architecture size remains constant.

Evaluate the model on a separate validation set that was used for hyperparameter tuning.

Why it fails: A validation set used for tuning hyperparameters is not an unbiased estimate of generalization because the model selection process has already adapted to it. Reporting performance on that same set can overstate true performance. A held-out test set that was never used for any tuning is needed for an unbiased estimate.

Train the model on all 10,000 examples and report the training accuracy.

Why it fails: Training and evaluating on the same data measures memorization, not generalization. The model can achieve near-perfect training accuracy by overfitting to noise and specific patterns in the training set. This gives an overly optimistic estimate and does not reflect how the model will perform on new, unseen examples, which is the goal of the classifier.

Use k-fold cross-validation on the entire dataset and report the average validation accuracy.

Why it fails: Cross-validation gives a more robust estimate by averaging over multiple splits, but it still uses all data for both training and validation across folds. Without a final untouched test set, the estimate can be slightly optimistic because the model selection may implicitly overfit to the validation folds. For a final performance estimate, a separate test set is still recommended.

Span Corruption

Why it fails: Span corruption, used in T5, replaces spans of text with sentinel tokens and trains the model to reconstruct them. Although it uses context from both sides, it is an encoder-decoder objective and does not specifically target bidirectional representations within a single encoder stack in the same way as MLM or PLM.

Causal Language Modeling (CLM)

Why it fails: CLM predicts the next token given only previous tokens, enforcing a unidirectional left-to-right context. This is used in GPT-style models and does not provide bidirectional context. Although powerful for generation, it does not meet the requirement of learning from both directions.

Next Sentence Prediction (NSP)

Why it fails: NSP trains the model to predict whether two sentences are consecutive. While it helps with sentence-level tasks, it does not directly enforce bidirectional context within a sentence. It is often used alongside MLM but is not itself a bidirectional objective.

Batch normalization

Why it fails: Batch normalization normalizes activations across a batch to stabilize training. It is not a pre-training objective and is less common in transformer architectures due to sequence length variability. While it can improve optimization, it does not directly teach contextual relationships. In this scenario, it is not a primary technique for pre-training large language models.

Weight initialization with small random values

Why it fails: Weight initialization with small random values helps avoid exploding gradients at the start of training, but it is a one-time setup, not an ongoing pre-training technique. It does not contribute to learning contextual relationships or preventing overfitting during pre-training. Other methods like regularization and objectives are more impactful for generalization.

Dropout on input embeddings

Why it fails: Dropout on input embeddings is a regularization technique that randomly zeroes some embedding dimensions during training. While it can help prevent overfitting, it is not a pre-training objective and does not directly enhance the model's ability to capture contextual relationships. It is typically used as a supplementary regularization method, not a core pre-training technique.

Use a higher dropout rate on the attention weights.

Why it fails: Dropout randomly zeroes some attention weights during training, which can regularize the model but does not systematically prevent attending to future tokens. It is not a deterministic mask and does not enforce causality. Future tokens could still be attended to when dropout is not applied, so this does not solve the problem.

Apply layer normalization before the self-attention computation.

Why it fails: Layer normalization stabilizes training but does not prevent attention to future tokens. It normalizes activations across features and is typically applied before or after sublayers. It has no masking effect on attention scores, so it does not enforce causality. The model could still attend to future tokens if no mask is used.

Increase the number of attention heads.

Why it fails: More attention heads allow the model to capture diverse relationships but do not impose any restriction on attending to future positions. Without a causal mask, each head can still look ahead. Increasing heads increases capacity and computation but does not address the need for autoregressive masking.

Increasing the batch size beyond the GPU's memory capacity.

Why it fails: Increasing batch size can improve throughput up to a point, but exceeding GPU memory causes out-of-memory errors and crashes. The scenario specifies real-time inference, where very large batches also increase latency. Therefore, this approach is counterproductive and not a viable technique for the stated goals.

Enabling dynamic batching to group incoming requests.

Why it fails: Dynamic batching can improve throughput by grouping requests, but it often increases latency because requests wait to be batched. In real-time inference, latency is critical, so dynamic batching may not be the most effective technique for reducing latency. While it can help throughput, the scenario prioritizes both latency and throughput, and other techniques like quantization and TensorRT offer a better balance. Thus, it is not one of the two most effective techniques here.

Implementing a larger number of transformer layers.

Why it fails: Adding more transformer layers increases model depth and computational cost, which raises latency and reduces throughput. It does not help achieve real-time inference goals; instead, it makes the model slower and more resource-intensive. The team needs optimizations, not a larger model.

Increase the batch size for each inference request.

Why it fails: Increasing batch size can improve throughput but typically increases latency for individual requests because the GPU processes more data per forward pass. For real-time chatbot inference, low latency per request is critical, so larger batches are counterproductive unless batching multiple concurrent users, which is not the stated goal.

Switch from FP16 to FP32 for all computations.

Why it fails: FP32 computations are slower and consume more memory than FP16, which would increase latency and reduce throughput. The goal is to reduce latency, so moving to higher precision is the opposite of what is needed. FP16 or even INT8 is preferred for inference to maximize speed while maintaining acceptable accuracy.

Increase the number of attention heads in each transformer layer.

Why it fails: Adding attention heads increases the model's capacity and computational cost, which would raise latency rather than reduce it. The architecture is already trained; modifying it would require retraining and would not preserve output quality. This is not an inference optimization technique and would likely degrade real-time performance.

The model is underfitting; increase the number of hidden layers.

Why it fails: Underfitting typically manifests as high bias, where both training and validation losses are high and stagnant. Increasing depth would exacerbate overfitting if the model is already memorizing the training data. The primary issue here is poor generalization, not insufficient model capacity to capture the underlying data patterns.

The learning rate is too low; increase it to accelerate convergence.

Why it fails: A very low learning rate would lead to slow convergence, making the loss decrease gradually rather than plateauing abruptly while validation loss rises. While adjusting the learning rate is standard, it does not directly address the divergence between training and validation error caused by model overfitting.

The dataset is too small; reduce the batch size.

Why it fails: Reducing batch size might introduce more stochasticity into the gradient estimates, but it does not inherently solve overfitting. While more data is ideal, the immediate fix for overfitting on an existing dataset is regularization. Changing batch size primarily affects memory utilization and the stability of the training process.