NCP-AIO Troubleshooting and Optimization Practice Question
An AI operations engineer is tuning a real-time inference service on NVIDIA A100 GPUs. Profiling with Nsight Systems shows that the GPU is idle for long periods while waiting for input data, and that host-to-device memory copies are frequent and small. The service uses a fixed batch size of 1 and a custom data loader. Which two changes are most likely to improve GPU utilization and reduce inference latency? (Choose two.)
⚠ Common exam trap
The trap here is focusing on CPU-side or stream-level tweaks while overlooking that the core inefficiency is the tiny batch size and unoptimized model execution.
Answer choices
Why each option matters
Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.
Correct answer & explanation
✓
Increase the inference batch size and implement dynamic batching to group requests.
The GPU idles because it waits on small, frequent data transfers and processes one sample at a time. Increasing batch size with dynamic batching and optimizing the model with TensorRT in FP16 reduce per-inference overhead and better utilize Tensor Cores. Together they increase work per transfer and speed up computation, directly improving utilization and latency.
Answer analysis
Option-by-option breakdown
For each option: why learners choose it and why it is or isn't the right answer here.
- ✗
Increase the number of CUDA streams used for memory copies and inference kernels.
Why it's wrong here
While multiple streams can overlap data transfer and compute, the scenario describes frequent small copies that are already causing overhead. Without batching, more streams may increase contention and complicate synchronization. The primary issue is the small batch size and inefficient model execution, so streams alone are unlikely to yield substantial improvement.
- ✓
Increase the inference batch size and implement dynamic batching to group requests.
Why this is correct
Larger batches amortize kernel launch and memory copy overhead across more work, keeping the GPU busier. Dynamic batching groups multiple incoming requests into a single inference pass, improving throughput and reducing per-request latency when the service is under load. This directly addresses the idle GPU periods caused by small, frequent transfers and fixed batch size of one.
- ✗
Use `cudaMemcpy` instead of `cudaMemcpyAsync` for all host-to-device transfers.
Why it's wrong here
Using synchronous `cudaMemcpy` would block the host thread and prevent overlap of transfer and compute, increasing latency. Asynchronous copies are preferred to hide transfer time behind computation. This change would degrade performance and does not address the root cause of small, frequent transfers or the fixed batch size of one.
- ✗
Pin the inference process to a single CPU core to reduce context switching.
Why it's wrong here
Pinching to a single CPU core may reduce context switching but severely limits the host-side data preparation and preprocessing throughput. The data loader and request handling would become a bottleneck, worsening GPU idle time. This is not a recommended optimization for a high-throughput inference service and does not address the small, frequent memory copies.
- ✓
Enable NVIDIA TensorRT with FP16 precision and optimize the model for the target GPU.
Why this is correct
TensorRT applies layer fusion, kernel auto-tuning, and precision calibration to reduce compute and memory overhead. Using FP16 on A100 Tensor Cores can significantly increase throughput and reduce latency for compatible models. This optimization targets the compute pipeline and reduces the time the GPU spends per inference, complementing batching to improve overall utilization.
About these practice questions
One of 309 original NCP-AIO practice questions on Courseiva, each with a full explanation and wrong-answer analysis — not exam dumps or protected exam content. Learn why practice questions differ from exam dumps →
JA
Written and reviewed by Johnson Ajibi, MSc IT Security
Senior Network & Security Engineer · founder of Courseiva
Last reviewed September 2026 · checked against the official NVIDIA exam blueprint
This NCP-AIO practice question is part of Courseiva's free NVIDIA certification practice question bank. Courseiva provides original exam-style practice questions with explanations, topic-based practice, mock exams, readiness tracking, and study analytics to help learners prepare for the NCP-AIO exam.