Software Development practice questions

Increase the chunk size to include the entire knowledge base in every prompt.

Why it fails: Increasing chunk size excessively leads to context window overflow and diluted attention mechanisms. Large chunks introduce irrelevant noise, which degrades generation quality and increases the tokens processed per request, ultimately spiking latency rather than optimizing it for high-throughput production environments.

Disable the reranking stage to reduce the total number of sequential operations.

Why it fails: Disabling reranking reduces sequential steps but severely hurts relevance. Rerankers are essential for surfacing the most pertinent documents retrieved from large vector stores. Without them, the model receives suboptimal data, leading to hallucinations and poor factual accuracy despite the slight reduction in latency.

Switch from asynchronous to synchronous API calls for all knowledge retrieval steps.

Why it fails: Synchronous API calls block the execution thread, causing severe bottlenecks in high-concurrency environments. Moving to synchronous patterns prevents the system from handling multiple retrieval requests in parallel, negating the throughput benefits provided by NVIDIA's asynchronous NIM microservices architecture and standard performance optimization practices.

The temperature setting is too low for effective text generation.

Why it fails: A temperature of 0.7 is standard for balanced creativity and coherence. It does not negatively impact raw performance metrics like latency or throughput. While it changes the probability distribution of tokens, it is unrelated to the underlying hardware resource utilization or inference speed constraints.

The n_gpu_layers is set to 40, which is too high for an 8B parameter model.

Why it fails: Setting layers to 40 for an 8B model is typical and usually fits within modern GPU memory. This is not a performance bottleneck; rather, it is a standard deployment setting. The issue lies within the lack of engine-level optimizations rather than the allocation of layers to the device.

The max_tokens limit is exceeding the KV cache capacity.

Why it fails: A limit of 512 tokens is relatively small for current GPU architectures. It is unlikely to cause memory exhaustion or cache overflow. The bottleneck is likely in the absence of optimized execution flags rather than the token constraint, which is well within standard operational limits for production models.

Increasing the model temperature to 1.5.

Why it fails: Increasing temperature to 1.5 creates high randomness, which directly increases the probability of hallucinations. A high temperature encourages the model to take risks with vocabulary and logic, making it significantly less reliable for factual retrieval tasks where precision and adherence to source data are required.

Reducing the context window to only the last 10 tokens.

Why it fails: Reducing the context window to only 10 tokens removes the necessary background information required for the model to answer correctly. This makes the model unable to use the retrieved context, forcing it to rely on its internal training data, which increases, rather than decreases, potential hallucination.

The TensorRT-LLM inference engine.

Why it fails: TensorRT-LLM handles the execution and computation of the model, not the initial tokenization of input text. While it sits in the pipeline, it expects pre-processed token IDs, making it responsible for the transformation and inference steps rather than the initial linguistic-to-numerical conversion process.

The Vector Database.

Why it fails: The vector database stores and retrieves existing embeddings. It does not perform the transformation from raw text to vectors; it expects the embeddings to be provided by an embedding model or encoder. Therefore, it is a storage layer rather than a conversion component.

The KV Cache.

Why it fails: The KV cache is a dynamic memory structure used to store intermediate attention states during token generation. It has no role in the initial conversion of input text to tokens or embeddings; its primary function is to optimize speed during the decoding phase of LLM inference.

Total number of GPUs installed in the rack.

Why it fails: This is a hardware inventory metric, not a performance or health indicator. It does not provide information about whether the service is running correctly, if requests are failing, or how efficiently the available resources are being utilized to handle incoming user traffic during operations.

The number of users logged into the platform.

Why it fails: User count is a business metric, not a system health or service-level objective (SLO) metric. While it helps estimate load, it doesn't indicate the technical performance of the inference stack. A service might be performing poorly even with few users, which this metric would fail to capture.

The total volume of log files written to disk.

Why it fails: Disk logging volume is a management issue for storage, not a metric of inference service performance. High log volume can indicate a system error, but it is not a direct measure of inference latency or throughput, which are the real-time indicators of application health and performance.

It forces the model to use CPU-offloading for every request.

Why it fails: CPU offloading is typically a fallback when memory is insufficient; it is detrimental to performance. This configuration is explicitly designed to keep execution on the GPU to maintain high speeds. CPU offloading would introduce significant latency and negate the benefits of continuous batching and PagedAttention.

It restricts the model to a single-user request at a time.

Why it fails: Continuous batching is explicitly designed to handle multiple concurrent requests, not limit them. This configuration allows the model to process different requests in different stages of decoding simultaneously, which is the opposite of a single-user restriction and is essential for high-throughput production environments.

It automatically scales the model across multiple nodes.

Why it fails: This configuration manages local GPU resources efficiently but does not implement network-level distributed scaling. Node-level scaling requires cluster orchestration, such as Kubernetes or specialized networking backends. These settings reside at the model execution layer, not the orchestration or distributed infrastructure management layer.

Set the builder flag to use INT8 calibration exclusively.

Why it fails: INT8 calibration is a post-training quantization technique that reduces precision to 8-bit integers. While it improves throughput, it is distinct from FP16 optimization and often requires a calibration dataset to maintain model accuracy, making it an inappropriate choice for simple FP16 precision enablement.

Convert the entire model weights to FP16 in PyTorch before exporting.

Why it fails: Manually converting weights to half-precision in PyTorch before exporting might lead to issues with layer compatibility or overflow during the translation process. TensorRT is designed to perform these optimizations internally during the engine build phase, which is more robust and produces a highly optimized plan file.

Disable the engine layer fusion to keep FP16 kernels separate.

Why it fails: Disabling layer fusion is counter-productive because TensorRT relies on layer fusion to combine multiple operations into a single kernel execution. This reduction in memory access and kernel launches is critical for performance; preventing it would negate the benefits provided by the underlying FP16 optimized kernels.

The count is too high, causing context switching overhead.

Why it fails: A count of two is relatively low for most modern NVIDIA GPUs. Context switching overhead typically becomes a concern when spawning dozens of instances, not when limited to just two. The low utilization is far more likely a result of insufficient concurrency to keep the GPU busy.

The model is bound to the CPU instead of the GPU.

Why it fails: The configuration explicitly specifies KIND_GPU, meaning the model is targeted for the GPU. The issue is not the lack of GPU targeting but rather the inability to generate enough workload to keep the GPU cores saturated due to insufficient concurrent execution instances defined in the config.

The configuration file syntax is invalid for Triton.

Why it fails: The provided configuration format is correct for a Triton 'config.pbtxt' file. The structure of 'instance_group' followed by 'count' and 'kind' is standard, meaning the issue is logical—performance-related—rather than a syntax error that would prevent the server from loading the model correctly.

Rewrite the model architecture using custom C++ code.

Why it fails: TensorRT-LLM provides pre-built implementations for most popular LLM architectures. Developers do not need to manually rewrite the models in C++; they simply use the provided scripts to map the Hugging Face model weights into the pre-optimized TensorRT-LLM structures, saving significant development time and reducing implementation errors.

Deploy the model as a raw PyTorch checkpoint directly.

Why it fails: TensorRT-LLM does not run raw PyTorch checkpoints. The library requires an intermediate conversion and compilation step to produce the optimized engine. Attempting to load raw PyTorch weights will fail because the runtime expects the specific binary layout and metadata defined by the TensorRT-LLM serialization format.

The use of a single-threaded CPU control process.

Why it fails: In-flight batching actually relies on high-performance multi-threaded scheduling on the CPU to manage the state of many requests. A single-threaded approach would create a bottleneck, making it impossible to handle the high throughput requirements of modern LLM inference. Thus, this is not a necessary condition for success.

Disabling GPU-based memory pre-allocation.

Why it fails: Memory pre-allocation is often necessary to ensure the server has enough resources to handle the peak concurrent load. Disabling it would lead to runtime allocation failures under high pressure. Modern GPU-based inference relies on pre-allocated pools to ensure stability and deterministic performance during intense serving scenarios.

Remove layers from the model randomly.

Why it fails: Randomly removing layers results in a broken model that will no longer generate coherent or accurate text. Neural networks rely on all layers for feature extraction and reasoning; pruning must be structured and mathematically grounded to preserve performance, making random removal an incorrect and destructive approach.

Compress the model into a standard ZIP archive.

Why it fails: Zipping model files only reduces storage size on disk, not the footprint in GPU memory during runtime. Once the model is loaded into memory, the uncompressed data will occupy the same amount of space. This does not address the primary memory bottleneck of inference, which is the VRAM usage.

Replace the tokenizer with a smaller version.

Why it fails: The tokenizer is a dictionary-based mapping and consumes very little memory compared to the model parameters. Replacing it would not significantly impact the total memory footprint. Furthermore, changing the tokenizer breaks the model's ability to interpret input data correctly, as it expects the original vocabulary.

To verify the aesthetic quality of the model's generated text.

Why it fails: Benchmarks are quantitative performance metrics, not qualitative aesthetic assessments. Aesthetics are subjective and evaluated via human review or Elo rating systems, not automated benchmarks. Benchmarks focus on objective data points like tokens per second, memory usage, and task accuracy scores, which are essential for engineering production systems.

To automatically retrain the model if it performs poorly.

Why it fails: Benchmarks are observational, not remedial. They report on current state; they do not possess the intelligence to diagnose why a model performs poorly, nor do they trigger retraining pipelines. Retraining is a separate, complex process that requires data preparation, compute allocation, and manual supervision by human engineers.

To replace the need for unit testing in the pipeline.

Why it fails: Unit testing verifies code logic, while benchmarks verify performance. They are complementary. Relying on benchmarks to replace unit tests is a dangerous practice that leaves the system vulnerable to logic errors, bugs, and edge-case failures that performance metrics would completely miss, potentially causing severe production outages.

Double the max_batch_size to 256.

Why it fails: Increasing the batch size forces the GPU to store more activation tensors in memory simultaneously, which will certainly worsen an existing Out-of-Memory condition. This is the opposite of the required action when a model is failing due to memory constraints on the current hardware.

Remove the quantization setting to use FP32.

Why it fails: Using FP32 precision requires significantly more memory than INT8 quantization. Removing quantization will drastically increase the VRAM requirements for model weights and activations, making the OOM error even more frequent and preventing the model from fitting into the current memory architecture.

Disable tensor_parallel entirely.

Why it fails: Disabling tensor parallelism forces the model onto a single GPU or a subset of GPUs, which likely leads to a massive memory deficit. Unless the model is small enough to fit on one card, this will immediately trigger OOM errors as the model cannot be partitioned.

Weight-only quantization to FP64.

Why it fails: Quantizing to FP64 increases precision but also drastically increases memory usage and slows down computational throughput. The goal of optimization is usually to move toward lower precision (like FP16 or INT8) to speed up execution, making this choice detrimental to the latency goals of the system.

Increasing the number of CPU worker threads.

Why it fails: While CPU threads handle request orchestration, increasing them does not solve the underlying latency bottlenecks occurring on the GPU. Excessive threading can lead to context switching overhead, which may negatively impact performance rather than improving the speed of the actual model inference execution on the hardware.

Serial processing of inference requests.

Why it fails: Serial processing is the least efficient way to use a GPU, as it leaves the hardware idle during most of the computation cycle. This approach would result in extremely low throughput and wasted financial resources on high-end hardware that is not being utilized to its full capacity.

Disabling the GPU and using CPU fallback.

Why it fails: Disabling the GPU removes the acceleration benefit entirely, causing performance to drop drastically. CPU fallback is only used as a last resort in case of hardware failure, not as an optimization strategy, as it cannot match the throughput and low latency provided by GPU-accelerated tensor operations.

Increasing the batch size to infinity.

Why it fails: There is a strict physical limit to the batch size based on the available VRAM. Setting an infinite batch size is impossible and would cause an immediate OOM error. Optimization requires finding the maximum batch size that fits in memory, not attempting to exceed hardware limitations.

Increase the batch size during the prefill phase.

Why it fails: Increasing batch size during prefill exacerbates memory congestion and compute contention on the GPU. While it improves throughput, it increases latency per request, worsening the bottleneck rather than resolving the performance issue for individual users needing faster response times.

Implement continuous batching with TensorRT-LLM.

Why it fails: Continuous batching is primarily designed to improve throughput during the decoding phase by handling requests of varying lengths. It does not inherently optimize the compute-intensive prefill phase or reduce the initial latency experienced when the model processes the prompt.

Switch the model architecture to a standard transformer.

Why it fails: Switching to a standard transformer architecture does not solve latency issues related to prefill performance. The core bottleneck is usually the compute-to-memory ratio during the processing of long input sequences, which remains a challenge regardless of the underlying transformer variant chosen.

Hard-code the LoRA weights into the base model file.

Why it fails: Merging weights into the base model is a static approach that destroys the flexibility of the LoRA architecture. This forces a full reload of the entire model whenever the adapter needs to be swapped, causing significant latency and unnecessary resource consumption.

Deploy separate Triton instances for every LoRA adapter.

Why it fails: Running separate instances for each adapter creates massive overhead and inefficient GPU memory utilization. Each instance requires its own copy of the base model, leading to hardware exhaustion and increased maintenance costs that are unnecessary with modern Triton adapter support.

Disable the model repository dynamic loading feature.

Why it fails: Disabling dynamic loading forces the server to initialize all assets at once, which prevents the system from scaling to support multiple adapters. This configuration makes it impossible to manage adapter lifecycle independently of the base model, limiting the agility of the AI application.

Set the global batch size to one for all nodes.

Why it fails: Setting the global batch size to one is highly inefficient and would result in extremely slow convergence. Distributed training environments are designed to handle large batch sizes to saturate GPU compute and maximize throughput across all available hardware nodes.

Disable checkpointing to save disk space.

Why it fails: Disabling checkpointing is dangerous in distributed training, where hardware failures are statistically common. Checkpointing is mandatory for fault tolerance, allowing the training job to resume from the last saved state rather than restarting from the beginning after a node failure.

Manually copy model weights to every GPU.

Why it fails: NeMo and PyTorch Lightning automate weight distribution via the framework's internal orchestrator. Manual weight copying is error-prone, violates parallel consistency, and is not a supported method for initializing distributed training in modern NVIDIA framework environments.

The batch size configured for the LLM inference generation step.

Why it fails: The generation batch size affects the LLM's throughput and latency during token generation, not the retrieval phase. Retrieval happens prior to the generation step when the system queries the vector store, so changing the generation parameters will have zero impact on the time spent searching for relevant context.

The choice of the tokenizer used during the final generation stage.

Why it fails: Tokenization is a preprocessing step for the LLM, not the retrieval system. While important for generation quality and context window management, the tokenizer logic does not interact with the vector database search process, meaning it provides no measurable change to the latency experienced during the retrieval phase.

The number of concurrent users accessing the LLM inference server.

Why it fails: While user concurrency affects total system load and resource contention on the GPU, it is not a direct factor in the retrieval phase latency itself. Retrieval latency is dictated by database indexing efficiency and network throughput, independent of the current load on the separate model inference engine.

Increase the model's temperature to 0.9 to improve reasoning density.

Why it fails: Changing the temperature controls the randomness of the model's token selection process, not the token count. Increasing it will have no effect on the input context size, meaning the error will persist because the model cannot process an input that exceeds its hard architectural limit of 128,000 tokens.

Switch to a different model architecture that supports 1 million tokens.

Why it fails: While changing the model is technically an option, it is rarely the most efficient solution for a token overflow error. It introduces significant infrastructure changes, potential hardware requirements, and model behavior variations that may impact the consistency of the application, whereas reducing context size is a configuration-level fix.

Enable FP8 quantization on the NIM server to increase capacity.

Why it fails: FP8 quantization improves the memory footprint and throughput of the model during inference but does not expand the maximum token limit of the model's attention mechanism. The error is a hard limit on the input sequence length, which quantization cannot bypass or extend by reducing memory overhead.