A machine learning engineer is fine-tuning a Cohere Command R model using OCI Generative AI. They want to evaluate the fine-tuned model's performance before deploying. Which TWO methods can they use?
Trap 1: Check the fine-tuning job's status in the OCI Console for…
The job status shows training metrics, but for evaluating the fine-tuned model's performance on new data, you need to run inference and analyze outputs.
Trap 2: Provision a dedicated AI cluster and monitor the cluster's latency…
Cluster metrics measure infrastructure performance, not model output quality.
Trap 3: Use the OCI CLI to call the model inference endpoint with test data
Using the OCI CLI to call the model inference endpoint with test data provides raw model outputs, which is crucial for *testing basic functionality* or *obtaining predictions from a deployed model*. However, this method alone does not facilitate structured performance *evaluation* before deployment. Evaluation typically involves comparing model outputs against a ground truth dataset and calculating specific metrics like ROUGE or BLEU, which the CLI inference command does not inherently perform. Dedicated evaluation features within OCI Generative AI are designed for this metric-driven assessment.
- A
Check the fine-tuning job's status in the OCI Console for validation metrics
Why wrong: The job status shows training metrics, but for evaluating the fine-tuned model's performance on new data, you need to run inference and analyze outputs.
- B
Use the OCI Generative AI Playground to send test prompts to the fine-tuned model endpoint
If the model is hosted on a dedicated cluster, the Playground can be configured to point to that endpoint for interactive testing.
- C
Provision a dedicated AI cluster and monitor the cluster's latency metrics
Why wrong: Cluster metrics measure infrastructure performance, not model output quality.
- D
Use the OCI CLI to call the model inference endpoint with test data
Why wrong: Using the OCI CLI to call the model inference endpoint with test data provides raw model outputs, which is crucial for *testing basic functionality* or *obtaining predictions from a deployed model*. However, this method alone does not facilitate structured performance *evaluation* before deployment. Evaluation typically involves comparing model outputs against a ground truth dataset and calculating specific metrics like ROUGE or BLEU, which the CLI inference command does not inherently perform. Dedicated evaluation features within OCI Generative AI are designed for this metric-driven assessment.
- E
Use the Python SDK's InferenceClient to programmatically send test prompts and analyze responses
The InferenceClient allows you to send requests to the model endpoint and capture responses for evaluation, suitable for systematic testing.