Hard Difficulty Questions

Use a larger generation model

Why wrong: Larger generation models increase generation latency.

Increase the number of shards

Why wrong: More shards may increase parallelism but also overhead; not a guaranteed latency reduction.

Ambiguous instructions

Why wrong: Ambiguous instructions lead to irrelevant or misunderstood outputs, not verbatim repetition.

Conflicting requirements

Why wrong: Conflicting requirements cause the model to choose one or produce inconsistent results.

Insufficient context

Why wrong: Insufficient context results in generic or incomplete answers, not prompt leakage.

The learning rate is too high.

Why wrong: A high learning rate typically causes divergent behavior and both losses would increase or oscillate, not the pattern described.

The validation set is not representative.

Why wrong: An unrepresentative validation set could show poor performance even with good generalization, but the training loss decrease is consistent with overfitting, not just a mismatch.

The training data is too small.

Why wrong: Small data can lead to overfitting, but the immediate symptom of training loss decreasing while validation does not improve is directly indicative of overfitting, not necessarily the root cause.

Overconfidence in predictions

Why wrong: Incorrect: Overconfidence is an outcome, not a characteristic.

Large vocabulary size

Why wrong: Incorrect: Large vocabulary is generally beneficial.

The feed-forward network applies a different set of weights for each token position.

Why wrong: The feed-forward network shares weights across all positions.

The encoder-decoder structure is used in GPT models.

Why wrong: GPT uses a decoder-only architecture.

Use exact search instead of ANN

Why wrong: Exact search does not inherently filter irrelevant results.

Reduce the number of retrieved chunks

Why wrong: Reducing the number may cut relevant chunks as well.

Increase chunk size

Why wrong: Larger chunks may include more irrelevant information.

Reduce the max token limit for all requests.

Why wrong: Reducing tokens may affect output quality and does not address cost structure.

Use a larger model to reduce retries.

Why wrong: Larger models cost more per token, not less.

Increase the number of parallel requests to improve efficiency.

Why wrong: More parallel requests may increase cost due to higher throughput, not reduce it.

Greedy decoding

Why wrong: Greedy decoding is fast but often produces less diverse and lower quality outputs compared to beam search.

Random sampling

Why wrong: Pure random sampling often reduces quality due to high unpredictability.

The load balancer is scaling based on CPU utilization; increase the CPU threshold

Why wrong: Load balancer scaling does not cause such pattern.

The VCN has a network latency issue; use a different availability domain

Why wrong: Latency would affect all requests, not just first.

The inference code has a lazy initialization; pre-load the model in the deployment script

Why wrong: Lazy init would cause cold start but the idle timeout is more likely the cause as it's a deployed service.

Increase the number of nodes to 8

Why wrong: Increasing nodes does not increase per-node memory.

Upgrade to higher memory node shapes

Why wrong: This is a possible solution but may not be feasible due to cost or availability.

Reduce the batch size in inference requests

Why wrong: Batch size reduction may help with memory per request, but the model itself still requires more memory than available.

Change the model to cohere.command-light.

Why wrong: Model choice does not bypass rate limits; each model and endpoint has its own rate limits.

Increase the max-tokens value.

Why wrong: Increasing max-tokens would likely increase resource usage and could exacerbate rate limiting.

Decrease the temperature to 0.0.

Why wrong: Temperature does not affect rate limits; it controls token selection randomness.

OCI Vault for secrets

Why wrong: While good practice, not strictly essential if secrets are managed elsewhere.

OCI Data Flow for data processing

Why wrong: Data Flow is for big data processing, not required for inference.

The inference request batch size is too small

Why wrong: Incorrect: Small batch sizes typically reduce latency.

The model is too large for the cluster memory

Why wrong: Incorrect: This would cause out-of-memory errors, not latency.

The client-side network is slow

Why wrong: Incorrect: While possible, the cluster is the shared environment; latency is more likely from cluster configuration.

Increase max tokens

Why wrong: Max tokens controls output length, not repetition.

Decrease temperature

Why wrong: Lower temperature makes outputs more deterministic, which can exacerbate repetition.

Add a stop sequence

Why wrong: Stop sequences terminate generation but do not reduce repetition within the generated text.

Required response time

Why wrong: Both approaches have similar response times; not a differentiator.

Internet connectivity

Why wrong: Both approaches require internet connectivity to access the service.

The compartment AIResources does not exist.

Why wrong: If the compartment didn't exist, the user would get a different error (e.g., 404).

The fine-tuning API is not yet available in that region.

Why wrong: API availability would cause a 404 or 501, not a 403.

The data scientist is not in the DataScientists group.

Why wrong: If not in the group, the error would be a more generic 403; the policy is defined for the group.

GPT-4

Why wrong: GPT-4 is from OpenAI and not available in OCI GenAI.

Stable Diffusion

Why wrong: Stable Diffusion is an image generation model; OCI GenAI focuses on text.

Use a different base model.

Why wrong: All base models may exhibit similar issues without specific training or prompting.

Fine-tune the model on the small secure code dataset.

Why wrong: Fine-tuning on a small dataset may not generalize well and can lead to overfitting.

Deploy a custom model hosted elsewhere.

Why wrong: This is not leveraging OCI Generative AI service and adds complexity.

Enable OCI Logging for all API calls

Why wrong: Incorrect: Logging captures data but does not enforce data residency.

Use OCI Object Storage with cross-region replication for redundancy

Why wrong: Incorrect: Cross-region replication would copy data out of the required region.

Store encryption keys in an OCI Vault in a different region

Why wrong: Incorrect: Keys can be in same region; storing keys elsewhere doesn't affect data residency.

The model uses a single tool call to answer the question

Why wrong: ReAct may involve multiple tool calls in a loop.

The model generates a final answer without any intermediate steps

Why wrong: ReAct explicitly requires intermediate reasoning and actions.

The model is fine-tuned specifically for tool use

Why wrong: ReAct is a prompting pattern, not a fine-tuning technique.