Performance optimisation and cost management is the practice of making your generative AI workloads run faster and cheaper without sacrificing quality. It matters for the 1Z0-1127 exam because Oracle wants to ensure you understand how to balance speed, resource usage, and budget in real cloud deployments. Without these strategies, you could spend a fortune and still deliver slow, frustrating user experiences.
Jump to a section
A simple way to picture Performance Optimization and Cost Management
A busy restaurant kitchen during Friday night dinner rush. The chef has a stack of orders coming in, each one a request for a specific dish. The kitchen has limited stoves, ovens, and prep stations. If the chef fires up every burner for every single order all at once, the kitchen overheats, ingredients burn, and customers wait forever. That is bad performance. To optimise, the chef batches similar orders, pre-preps common ingredients, and keeps the most popular sauces ready in a small fridge right next to the stove. That fridge is a cache: it stores frequently used items close at hand so the chef does not have to run to the big walk-in cooler every time. Cost management means the chef does not buy a hundred kilos of truffle oil just because one table ordered it; she buys just enough, tracks what gets wasted, and chooses cheaper substitutes for dishes where the flavour difference is tiny. She also decides whether to hire an extra dishwasher for peak hours or to let plates pile up during quiet times. In cloud AI, you make the same choices: you provision just enough compute power, cache repeated model answers, batch your requests to avoid waste, and scale your resources up and down to match actual demand. You do not leave a supercomputer running idle all night just because you needed it for ten minutes at 3 PM.
Performance optimisation means tuning your system to respond quickly to user requests while using as few resources as possible. Think of it like tuning a car engine: you want maximum speed and fuel efficiency at the same time. In generative AI, the 'engine' is the underlying compute hardware, like GPUs (Graphics Processing Units) which are specialised chips that handle the heavy maths for AI models. When a user asks your AI chatbot a question, that request travels through a network to a server, where a model like a large language model (LLM) processes it. The time from request to response is called latency. High latency makes your app feel sluggish. Optimisation reduces latency.
One key technique is caching. A cache is a small, fast storage area that holds copies of frequently accessed data. Instead of re-running the expensive AI model every time someone asks 'What is the weather today?', you store the answer in the cache. The next time a user asks the same question, your system checks the cache first. If the answer is there (a cache hit), it returns it instantly. If not (a cache miss), it runs the model and stores the new answer. This dramatically cuts compute costs because model inference (the act of generating a response) costs money and time. Oracle offers services like OCI Cache with Redis to help with this.
Another crucial concept is concurrency and batching. Concurrency means handling multiple requests at the same time. If ten users ask questions simultaneously, your system should not process them one by one. Instead, you batch them together. Batching groups similar requests so the GPU can process them in one go, which is far more efficient. Oracle's Generative AI service supports batching to maximise throughput (the number of requests processed per second).
Now, cost management. Cloud resources are not free. You pay for compute time, storage, and data transfer. The biggest cost driver in generative AI is the GPU. GPUs are expensive to rent by the hour. If you leave a powerful GPU instance running 24/7 but only use it for two hours a day, you waste money. The solution is elastic scaling. Elastic scaling automatically adds more compute resources when demand spikes and removes them when demand drops. Oracle Cloud Infrastructure (OCI) lets you set up auto-scaling policies that launch new GPU instances during busy periods and shut them down when traffic slows.
You also need to consider model size and precision. Larger models with billions of parameters (the settings that the model learned during training) are more accurate but slower and more costly to run. Smaller models are faster and cheaper but may be less accurate. A strategy called model distillation takes a large, powerful model and trains a smaller 'student' model to mimic its behaviour. The smaller model runs faster and costs less. You can also use quantisation, which reduces the precision of the numbers the model uses (e.g., from 32-bit to 16-bit), making calculations faster and using less memory.
Finally, monitoring and optimisation are ongoing. You must track metrics like latency, throughput, and utilisation (how busy your GPUs are). Tools like OCI Monitoring and OCI Logging help you see where bottlenecks occur. If your GPU utilisation is below 50%, you are likely over-provisioned and wasting money. If latency spikes, you may need more caching or a more efficient model.
In summary, performance optimisation and cost management is about making smart trade-offs. You balance speed, accuracy, and price by using caching, batching, auto-scaling, and model selection. The goal is to deliver a fast, reliable user experience without exceeding your cloud budget.
Define Performance Requirements
Determine acceptable latency (e.g., under 2 seconds) and throughput (e.g., 100 requests per second). These numbers guide all subsequent choices. Without clear targets, you cannot measure success.
Select the Right Model
Choose between a large general-purpose model and a smaller distilled model based on the task complexity. Test both on a sample of real queries. The goal is the smallest model that still delivers acceptable accuracy.
Implement Caching
Set up a cache (e.g., OCI Cache with Redis) for repetitive queries. Decide on a cache key strategy (e.g., normalise user input) and a TTL. This reduces the number of expensive model invocations.
Configure Auto-Scaling
Set minimum, maximum, and target utilisation thresholds for GPU instances. Use OCI's auto-scaling service. This ensures you have enough capacity during spikes but do not pay for idle resources during lulls.
Monitor and Tune
Deploy monitoring dashboards for latency, utilisation, cache hit ratio, and cost. Review weekly. If utilisation stays low, downsize instances. If latency spikes, increase cache TTL or add more cache entries. Adjust scaling policies as usage patterns change.
An IT professional at a mid-sized e-commerce company is building a customer service chatbot powered by a generative AI model. The chatbot answers questions about orders, returns, and shipping. The company expects 10,000 conversations per day, but traffic spikes during holiday sales. The professional must ensure the chatbot responds in under two seconds while keeping the monthly cloud bill under a budget.
First, the professional selects a pre-trained model from Oracle Cloud Infrastructure's Generative AI service. They choose a medium-sized model, not the largest, because faster response time is more important than nuanced answers for simple queries. They deploy it in a single OCI region close to their customers to minimise network latency.
Next, they implement caching. They use OCI Cache with Redis to store answers to the most common questions, such as 'Where is my order?' or 'Return policy'. For these questions, the cache returns an answer in milliseconds without invoking the expensive model. They set a time-to-live (TTL) of one hour so that stale answers are refreshed periodically.
For handling the holiday spike, the professional configures auto-scaling. They set a minimum of two GPU instances and a maximum of ten. They create a scaling policy that adds a new instance when CPU or GPU utilisation exceeds 70% for five minutes. During Black Friday, the system automatically scales up to eight instances. After the sale, it scales back down to two, saving money on idle resources.
They also enable batching through the OCI Generative AI service's batch inference endpoint. Instead of sending one customer query at a time, the system groups requests from multiple users into a single batch. This improves throughput and reduces per-request cost. They monitor performance using OCI Monitoring dashboards:
Latency: average response time per request, aiming for under 1.5 seconds.
Throughput: requests per second, watching for drops that indicate a bottleneck.
Utilisation: GPU usage percentage, aiming for 60-80% to balance cost and performance.
Cache hit ratio: percentage of requests answered from cache, aiming for over 50%.
When utilisation stays below 40% for a week, they downsize the instance type. When latency spikes above two seconds, they add more caching for popular topics. The professional also sets budget alerts in OCI Budgets: if daily costs exceed a threshold, they receive an email to investigate.
Finally, they perform a quarterly review. They regenerate the cached answers with a newer model version if accuracy improves. They check if any model alternatives become available that offer better performance per dollar. This continuous cycle of monitoring, adjusting, and re-evaluating keeps the chatbot fast, reliable, and within budget.
The 1Z0-1127 exam tests your understanding of how to balance performance and cost in generative AI workloads. The questions are scenario-based: they describe a business situation and ask you to choose the best optimisation or cost management strategy. You will not be asked to write code or configure dashboards, but you must know which OCI services and features to use for each goal.
Key concepts the exam loves to test:
Caching: They will ask when to use a cache versus when to re-run the model. The correct answer is always to cache for repetitive, identical queries with a short TTL. A trap question might suggest caching for dynamic, personalised requests where each user's context is unique. That is wrong because caching would serve stale or irrelevant answers.
Auto-scaling: They test the difference between reactive scaling (adding instances when CPU hits a threshold) and predictive scaling (using ML to forecast demand). The exam expects you to know that reactive scaling is simpler and more common for most use cases. A trap asks you to choose a fixed number of instances for a workload with unpredictable spikes. The correct answer is to use auto-scaling with a minimum and maximum.
Batch inference: They present a scenario with many small, similar requests. The correct answer is to use batching to improve throughput. A trap suggests processing each request individually on separate GPUs, which is inefficient and expensive.
Model selection: They test your ability to choose between a large model and a smaller distilled model. The correct answer prioritises smaller models for real-time applications where latency is critical, and larger models for offline analysis where accuracy trumps speed.
Cost monitoring: They ask which OCI service to use for tracking spending. The answer is OCI Budgets and OCI Usage Reports. A trap might suggest OCI Monitoring, which tracks performance metrics, not costs.
Trap patterns to watch for:
'Always use the most powerful GPU' – this ignores cost. The correct answer is 'use the smallest instance that meets latency requirements'.
'Set up a cache with a TTL of one week' – too long, answers become stale. The correct answer is a short TTL, like one hour or one day, depending on how often the underlying data changes.
'Scale to a fixed number of instances based on historical peak' – this wastes resources during off-peak times. Auto-scaling is preferred.
'Ignore model size, just use the largest model for everything' – this is expensive and slow for simple tasks.
Definitions to memorise:
Latency: time from request to response.
Throughput: number of requests processed per second.
Cache hit: request answered from cache.
Cache miss: request that requires model inference.
Auto-scaling: automatic adjustment of compute resources based on demand.
Batch inference: processing multiple requests together.
Model distillation: training a smaller model to mimic a larger one.
Quantisation: reducing number precision to speed up computation.
The exam may also include questions about 'cold start' – the delay when a new instance launches. They want you to know that keeping a minimum number of instances running ('warm pool') reduces cold start latency at a small cost. Another topic: 'cost allocation tags' to track spending by department. Tags let you see which team or project drives the most cost.
In summary, focus on recognising the scenario, identifying the performance bottleneck (latency, throughput, or cost), and selecting the matching optimisation strategy. Always consider the trade-off between cost and performance.
Caching frequently requested AI responses reduces latency and compute costs by storing answers closer to the user.
Auto-scaling adjusts the number of GPU instances up or down based on real-time demand, preventing overspending on idle resources.
Choose the smallest model that meets your accuracy requirements to minimise cost and maximise speed.
Batch similar inference requests together to improve GPU utilisation and throughput.
Monitor latency, utilisation, and cache hit ratio to identify performance bottlenecks and cost waste.
Set budget alerts in OCI Budgets to receive notifications when cloud spending exceeds thresholds.
Use a warm pool of a few always-on instances to reduce cold start delays for critical workloads.
Model distillation and quantisation reduce model size, making inference faster and cheaper without major accuracy loss.
These come up on the exam all the time. Here's how to tell them apart.
Caching
Stores frequently used answers for instant retrieval
Reduces model invocation count by up to 80%
Increases complexity but lowers latency and cost
No Caching
Every request runs the full model
Higher latency and compute cost per request
Simpler to implement but expensive at scale
Auto-scaling
Automatically adjusts resources to demand
Pays only for resources used during spikes
Requires careful threshold configuration
Fixed Capacity
Runs a constant number of instances regardless of traffic
Wastes money during low-demand periods
Predictable performance but high baseline cost
Large Model (e.g., 70B parameters)
Higher accuracy for complex tasks
Slower inference and higher GPU cost per request
Requires expensive A100 or H100 GPUs
Small Distilled Model (e.g., 7B parameters)
Faster inference and lower cost
Accuracy may be insufficient for nuanced tasks
Runs on cheaper, smaller GPU instances
Real-time Inference
Returns answer immediately per request
Best for user-facing applications like chatbots
Higher latency if many concurrent requests hit limits
Batch Inference
Groups multiple requests for efficient processing
Best for offline jobs like content summarisation
Lower per-request cost but introduces waiting time
Mistake
Faster GPUs always give better performance for AI workloads.
Correct
Faster GPUs help, but performance is also limited by network latency, cache effectiveness, and model size. A smaller model on a modest GPU can outperform a huge model on the fastest GPU for simple tasks.
Beginners see GPU specs as the only factor, ignoring that the bottleneck is often elsewhere, like disk I/O or the model's size.
Mistake
Caching is only useful for identical requests, so it is rarely beneficial in AI chatbots.
Correct
Caching is extremely useful for frequently asked questions, common greetings, and static responses. Even with slight variations, you can normalise and cache normalised versions.
People underestimate how repetitive real-world user queries are. They think every chatbot query is unique.
Mistake
Auto-scaling always saves money because you only pay for what you use.
Correct
Auto-scaling saves money compared to running fixed capacity, but it can still waste money if you set the minimum instances too high or if scaling policies react too slowly and keep extra instances running longer than needed.
The concept of 'pay-as-you-go' sounds like it automatically optimises cost, but poor configuration can lead to over-provisioning.
Mistake
Larger models are always better because they are more accurate.
Correct
Larger models are more accurate for complex tasks but are slower and more expensive. For simple tasks, a smaller distilled model is often cost-effective and fast enough.
Beginners equate 'bigger' with 'better' without understanding the cost-performance trade-off.
Mistake
You can ignore cost management until the AI system is deployed and users start complaining.
Correct
Cost management must be planned from the start. Choosing the wrong model, instance type, or caching strategy during design can lead to runaway costs that are hard to fix later.
The 'build first, optimise later' mindset works in traditional software but fails with expensive GPU compute.
Reveal each answer, then mark whether you got it right. Score 60%+ to unlock the next chapter.
Caching stores the results of previous AI model requests so that if the same request comes again, you can return the stored answer instantly without re-running the expensive model. This matters because it drastically reduces response time and compute costs.
Start by listing the accuracy levels your users need. If your tasks are simple (like answering FAQs), a small distilled model is faster and cheaper. For complex reasoning or creative writing, you may need a larger model. Always test both on your actual data.
Latency is the time it takes for one request to get a response (like the time to boil a single egg). Throughput is how many requests you can handle per second (like how many eggs you can boil at once in a big pot). Both matter for performance.
You set a policy that monitors a metric like CPU utilisation. When utilisation goes above a threshold, OCI automatically launches new compute instances. When it drops below another threshold, it removes instances. You pay only for the time each instance runs.
OCI Budgets, OCI Usage Reports, and OCI Cost Analysis. You can set alerts that email you when spending exceeds a limit. OCI Monitoring tracks performance metrics like GPU utilisation, which indirectly affects cost.
A cold start happens when a new GPU instance is launched to handle a sudden spike in requests. It takes 30-60 seconds for the instance to be ready, causing delays. You reduce cold starts by keeping a small pool of instances always running (a warm pool).
Yes, if your chatbot receives many similar requests that can be grouped together. Batch inference processes multiple inputs in one go, improving GPU efficiency and reducing per-request cost. It works best when you can collect requests over a short window.
You've finished Performance Optimization and Cost Management. Continue through the 1Z0-1127 study guide to build a complete picture of the exam.
Done with this chapter?