You are deploying a generative AI solution on OCI for a healthcare client that requires strict data residency (data must remain in the EU) and low-latency inference. The solution uses a fine-tuned LLM model (7B parameters) stored in Object Storage in the Frankfurt region. You have set up an OCI Data Science model deployment endpoint with GPU shape VM.GPU.A10.1, using a single replica. During load testing with 50 concurrent users, you observe high latency (average 8 seconds per request) and occasional 504 gateway timeouts. The model deployment logs show no errors, and the model loads successfully. You have confirmed that the Object Storage bucket is in the same region and that the network latency between the client and the endpoint is minimal (under 5 ms). Which action should you take to reduce latency and eliminate timeouts?
Trap 1: Increase the model deployment endpoint timeout setting from 60…
Increasing the timeout does not address the root cause of high latency and 504 errors. The model deployment is overwhelmed by concurrent requests, so longer timeouts only delay the timeout error. Horizontal scaling (more replicas) is needed.
Trap 2: Upgrade the model deployment shape to VM.GPU.A100.4 and keep a…
Upgrading to a more powerful GPU (A100) with a single replica improves per-request throughput but still processes requests sequentially. With 50 concurrent users, the queue wait time remains high. Multiple replicas are required to handle concurrency.
Trap 3: Move the model deployment to the US East (Ashburn) region to…
Moving to the US East region would violate the strict data residency requirement (data must remain in the EU). Additionally, this does not address the concurrency bottleneck; the same single-replica issue would persist in the new region, and latency may increase.
- A
Increase the model deployment endpoint timeout setting from 60 seconds to 300 seconds in the OCI console.
Why wrong: Increasing the timeout does not address the root cause of high latency and 504 errors. The model deployment is overwhelmed by concurrent requests, so longer timeouts only delay the timeout error. Horizontal scaling (more replicas) is needed.
- B
Upgrade the model deployment shape to VM.GPU.A100.4 and keep a single replica.
Why wrong: Upgrading to a more powerful GPU (A100) with a single replica improves per-request throughput but still processes requests sequentially. With 50 concurrent users, the queue wait time remains high. Multiple replicas are required to handle concurrency.
- C
Increase the number of replicas to 3 and enable autoscaling based on CPU utilization.
Correct. Adding replicas (e.g., 3) distributes concurrent requests across multiple endpoints, reducing queue wait time and latency. Enabling autoscaling based on CPU utilization ensures the deployment dynamically adjusts to traffic spikes, preventing timeouts while maintaining data residency.
- D
Move the model deployment to the US East (Ashburn) region to leverage lower-cost GPU capacity and reduce latency.
Why wrong: Moving to the US East region would violate the strict data residency requirement (data must remain in the EU). Additionally, this does not address the concurrency bottleneck; the same single-replica issue would persist in the new region, and latency may increase.