Hardware practice questions

NPU

Why wrong: NPU stands for Neural Processing Unit, a broad category; not specific to Google.

FPGA

Why wrong: FPGAs are reconfigurable, not designed specifically by Google for ML.

GPU

Why wrong: GPUs are general-purpose accelerators, not Google-specific.

Use a high-core-count CPU with large RAM

Why wrong: CPUs are inefficient for parallel matrix operations common in deep learning; training would be very slow.

Use a single GPU with model parallelism

Why wrong: Model parallelism on a single GPU fails to reduce training time because it partitions the model across the device’s own memory, introducing sequential computation and communication overhead that does not scale throughput. It is tempting because it enables fitting a transformer that exceeds a single GPU’s VRAM, but the correct approach for speed is data parallelism across multiple GPUs, which processes distinct micro-batches concurrently.

Use a single TPU with model parallelism

Why wrong: TPU is specialized but less flexible; single TPU may not provide enough compute for large models, and model parallelism adds complexity.

Increase the batch size to the maximum the GPU memory allows

Why wrong: Very large batch sizes can negatively impact convergence and may require more epochs, potentially increasing total energy.

Use a larger model architecture to achieve higher accuracy faster

Why wrong: Larger models require more computation and energy; they do not reduce energy consumption.

Train the model on CPUs instead of GPUs

Why wrong: CPUs are less energy-efficient for deep learning training than GPUs, typically consuming more energy per training step.

Central Processing Unit (CPU)

Why wrong: CPUs are optimized for sequential tasks and general-purpose computing, not for the massive parallelism required in deep learning training.

Neural Processing Unit (NPU)

Why wrong: NPUs are specialized for inference on edge devices, not typically for training large models.

Tensor Processing Unit (TPU)

Why wrong: TPUs are custom ASICs designed by Google for TensorFlow, but GPUs are more widely used and versatile for training various deep learning models.

Train on larger datasets to improve accuracy

Why wrong: Larger datasets require more compute and energy, increasing the carbon footprint.

Use more powerful GPUs to speed up training

Why wrong: More powerful GPUs may speed up training but often consume more energy, increasing the carbon footprint.

Use older, less efficient hardware to save on manufacturing emissions

Why wrong: Older hardware is typically less energy-efficient, leading to higher operational emissions.

TPU designed for TensorFlow

Why wrong: TPUs are also highly efficient for training, but GPUs are more universally adopted and the question does not specify TensorFlow usage.

CPU with high clock speed

Why wrong: CPUs are optimized for sequential tasks and cannot match the parallel throughput needed for large-scale neural network training.

FPGA for reconfigurable logic

Why wrong: FPGAs are used for low-latency inference, not typically for training large models.

Add more Redis nodes to the cache cluster

Why wrong: Redis caching does not affect model inference latency.

Increase the number of Kafka partitions and consumer threads

Why wrong: This improves parallelism for stream processing but does not reduce model inference time.

Decrease the inference batch size from 32 to 1

Why wrong: Smaller batch sizes can increase overhead and latency due to less efficient computation.

Increase batch size

Why wrong: Batch size affects throughput but not the latency of a single inference request.

Add more layers

Why wrong: Adding layers deepens the network and increases inference time.

Use a larger model

Why wrong: A larger model would increase computational requirements and latency.

Full fine-tuning with gradient checkpointing

Why wrong: Full fine-tuning of a 70B model on a single 24GB GPU is impossible even with gradient checkpointing.

Instruction tuning with a smaller 7B model

Why wrong: While smaller model is feasible, it does not meet the requirement to fine-tune a 70B model.

LoRA (Low-Rank Adaptation) alone

Why wrong: LoRA reduces trainable parameters but the base model still requires full precision memory, likely exceeding 24GB for a 70B model.

Train on the largest possible dataset

Why wrong: Larger datasets increase training time and energy use.

Use redundant backup servers

Why wrong: Redundant servers increase energy consumption without contributing to efficiency.

Increase batch size to maximum

Why wrong: Larger batch sizes may improve throughput but can increase peak power draw.

Apply model distillation to create a smaller student model

Why wrong: Model distillation reduces model size but still requires a neural network inference pipeline, which on severely resource-constrained IoT devices may exceed available compute cycles and battery budget; the scenario demands a model that runs locally with minimal latency, and distillation alone does not guarantee the student model meets the strict memory or energy limits of the hardware. This technique is tempting because it is a standard approach for deploying deep learning on edge devices when the target platform has moderate resources, such as a smartphone or a Raspberry Pi, where a distilled model can achieve acceptable speed and accuracy trade-offs.

Deploy the model on a cloud server and stream data

Why wrong: Cloud deployment introduces network latency and requires constant connectivity, unsuitable for real-time patient monitoring on IoT devices.

Use ONNX Runtime with a GPU backend

Why wrong: ONNX Runtime with GPU backend is designed for server-class hardware, not for low-power IoT devices.

It automatically quantizes models to INT8

Why wrong: Quantization is an optional step, not inherent to ONNX.

It compresses model size by 90%

Why wrong: Compression depends on quantization or pruning; ONNX itself does not guarantee size reduction.

It reduces training time

Why wrong: ONNX is for inference deployment, not training.

Use the full PyTorch model with JIT scripting

Why wrong: PyTorch JIT is not optimized for mobile; it is more suited for server inference.

Deploy the model on a cloud endpoint and call via REST API

Why wrong: Cloud endpoint adds network latency, which is undesirable for mobile app real-time inference.

Export to ONNX and use ONNX Runtime with GPU

Why wrong: ONNX Runtime with GPU is for server-side, not mobile; mobile GPUs are not standard.

YOLOv4

Why wrong: YOLOv4 is efficient but still may be too heavy for very limited edge devices; MobileNet is more optimized.

ResNet-152

Why wrong: ResNet-152 is a deep network with many parameters, too large for edge devices.

BERT

Why wrong: BERT is a large transformer model for NLP, not suitable for resource-constrained edge devices.

Roll back to the previous model version and restrict use of the AI tool to only European patients.

Why wrong: Rolling back does not solve the problem for the new patient population and scanner; restricting use reduces clinical value.

Retrain the model using the original training data but with increased regularization to avoid overfitting.

Why wrong: Retraining on old data alone will not capture the new distribution; regularization does not adapt to domain shift.

Adjust the model's decision threshold to reduce false positives and then monitor for two weeks.

Why wrong: Threshold adjustment can reduce false positives but may increase false negatives and does not address the underlying data shift.