The PMLE exam objective 5.2 — designing and implementing batch prediction pipelines — tackles the challenge of doing machine learning work at massive scale without breaking the bank or the schedule. A single prediction might take milliseconds, but when your business needs millions of predictions every day, you cannot send them one at a time; you need a system that can process them in efficient, organised chunks, and that is exactly what batch inference does.
Jump to a section
A simple way to picture Batch Prediction and Inference Optimization
A bakery's central oven is the core of its operation.
For a daily routine, the baker carefully prepares individual pastries as customers order them: a croissant here, a baguette there, each one baked fresh and handed over immediately. This is like online (real-time) inference, where a model predicts on a single request right when it arrives.
But once a week, the bakery receives a massive pre-order from a local hotel for 500 identical croissants. The baker doesn't bake them one-by-one as orders trickle in. Instead, they load all 500 croissants into the oven at once, bake them in a single batch, then let them cool on racks before packaging them for delivery. This is batch prediction: processing a large group of requests together, not individually, to save time and energy.
Optimising inference latency in this analogy means reducing the time from when the baker loads the tray to when the croissants are ready for packaging. The baker might preheat the oven (load and cache the model), use convection fans (parallel processing across multiple CPU cores), or split the batch across two ovens (distributed computing with multiple servers). The goal is never to make the first croissant come out faster than a single-order croissant would, but to ensure the whole batch finishes predictably and quickly, minimising the wait for the hotel delivery driver. This mirrors exactly how machine learning engineers optimise batch pipelines: they tune infrastructure, not the model itself, to get hundreds of thousands of predictions out the door in a reliable time window.
Machine learning inference is the process of using a trained model to make a prediction on new data. If your model is a recipe, inference is the act of following that recipe with fresh ingredients to produce a result. There are two main ways to perform inference: online (real-time) and batch (offline).
Online inference is like a vending machine: you put in a request, and within a second you get a single prediction back. It is designed for low latency (how fast a single prediction is made) and is used for things like fraud detection on a credit card transaction or a chatbot response. The model is always running, waiting for a request.
Batch inference is completely different. It is designed for high throughput (how many predictions can be made in a total amount of time). Instead of responding instantly to one request, batch inference collects many requests over a period of time — say, every hour or overnight — and then processes them all at once. This is much more cost-effective because you can turn on the computing power (the servers or virtual machines) just for that batch, then turn it off when done. It is also simpler: you do not need a complex serving infrastructure that never sleeps.
A 'batch prediction pipeline' is the sequence of steps that takes raw data, turns it into predictions, and delivers those predictions somewhere useful. Think of an assembly line in a factory.
Here are the typical stages:
Data extraction: Pull the raw data from a storage system like a database or a cloud data lake. For example, get all new customer sign-ups from the last hour.
Preprocessing: Clean and transform the raw data into the format the model expects. This might mean scaling numbers, converting text to numbers, or filling in missing values.
Inference: Send the preprocessed data to the model, which produces predictions. Because it is a batch, the model can work through hundreds of thousands of rows quickly.
Postprocessing: Convert the model's raw output (e.g., a number between 0 and 1) into something a human or another system can use (e.g., 'yes' or 'no').
Output storage/serving: Write the predictions to a database, a file, or send them to a message queue for other applications to consume.
Now, what is 'inference optimisation'? It is the art of making that batch pipeline run faster, cheaper, or with fewer computing resources. The goal is to minimise the time between starting the batch and getting all predictions back (known as batch latency), without exceeding the budget for compute power.
Key optimisation techniques include:
Batching size tuning: Increasing the number of requests processed together (the batch size) generally improves throughput until you hit memory limits. Too large, and the model runs out of memory; too small, and the overhead of starting each mini-batch wastes time.
Model quantization: Reducing the precision of the numbers in the model (for example, from 32-bit floating point to 8-bit integers). This makes the model smaller and faster, at the cost of a tiny dip in accuracy.
Parallelism: Using multiple machines or multiple CPU/GPU cores to process different parts of the batch at the same time. This is like having multiple factory lines running in parallel.
Hardware acceleration: Using specialised hardware like GPUs or TPUs (Tensor Processing Units) that are designed for the parallel math operations required by machine learning models.
Caching: Storing the results of preprocessing steps so that if the same data appears in future batches, you do not have to process it again.
Pipelining: Overlapping the stages of the pipeline. While the model is making predictions on batch number one, the preprocessing stage can already start work on batch number two. This keeps all parts of the system busy.
When you design a batch prediction system, you are making trade-offs. A faster batch might cost more because you use more expensive hardware or keep it running longer. A cheaper batch might run overnight and finish in several hours, which is fine if the predictions are not urgently needed. The PMLE exam expects you to understand these trade-offs and to be able to choose the right approach for a given business requirement.
Define the Batch Window and Frequency
Determine how often predictions are needed (hourly, daily, weekly) and the deadline by which they must be complete. This drives all other decisions, such as which hardware to use and whether you can use cheap spot VMs.
Prepare Input Data in a Cloud Storage Bucket
Export the raw data from a transactional database or a data warehouse into a file format compatible with the inference service (e.g., CSV, JSON Lines). Storing data in a distributed object store like Cloud Storage allows the batch service to read multiple chunks in parallel.
Configure the Batch Prediction Job
Define the machine type (CPU or GPU), the number of parallel workers (also called replicas), and the batch size (number of rows sent to the model per forward pass). This configuration directly controls the trade-off between speed and cost.
Submit the Job and Monitor Progress
Submit the batch prediction request using a command-line tool or API. Monitor logs and metrics like 'prediction throughput' and 'error rate'. If throughput is too low, you can stop the job and resubmit with more workers or a larger machine type.
Write Output Predictions to a Target System
After the job completes, the predictions are stored in the specified Cloud Storage bucket. A downstream script or service reads these predictions and populates a user-facing database, sends notifications, or triggers other business processes.
A streaming music service like Spotify wants to generate personalised playlists for 100 million users every Sunday morning. The playlist recommendations rely on a machine learning model that predicts which songs a user will like based on their listening history from the past week.
Here is how an IT professional — specifically a machine learning engineer — would design and implement this batch prediction pipeline step by step.
First, they define the business requirement: 'Generate a new playlist for every user by 06:00 AM every Sunday, using the listening data from Monday 00:00 to Saturday 23:59 of the previous week.' The tolerance for delay is 10 minutes. The batch must complete within a two-hour window (04:00 to 06:00) because the data processing and model training runs earlier in the night.
The engineer then chooses a batch processing framework. On Google Cloud, they might use Vertex AI Batch Prediction or Dataflow (a managed stream and batch processing service). They opt for Vertex AI Batch Prediction because it is purpose-built for deploying models and running inference at scale without managing servers.
The steps they execute are as follows:
Data preparation: Write a script that extracts the last week's listening logs from BigQuery (a cloud data warehouse). This script runs automatically at 03:00 AM Sunday. It joins user listening history with song metadata, producing a CSV file with 100 million rows.
Preprocessing transformation: The engineer uses a preprocessing step (a transform function) that converts categorical fields (like 'genre') into numeric IDs and scales numerical fields (like 'time of day listened'). This step is defined in the model's deployment configuration to run automatically before inference.
Batch inference request: The engineer submits a batch prediction job to Vertex AI, specifying the model, the input data location (Cloud Storage bucket), and the output data location. The platform automatically distributes the data across many machines.
Monitoring and optimisation: The engineer monitors the job. If it is going slower than expected, they might increase the number of replicas (parallel machines). If it finishes in 15 minutes but the budget is tight, they might reduce the machine type to save money for next week.
Postprocessing and delivery: When the job completes, the output is a file on Cloud Storage with 100 million user IDs and their corresponding recommended song IDs. The engineer runs a small script that formats this data and inserts it into a fast database like Cloud Spanner. Then, when the user opens the app on Sunday morning, the app queries this database and displays the fresh playlist.
In this scenario, the engineer never writes code to 'serve' the model to handle millions of simultaneous requests. Instead, they design a pipeline that efficiently processes all requests in a batch, using the cloud's ability to spin up many machines cheaply for a short time. The key skills are: choosing the right batch service, configuring the compute correctly, monitoring the job, and handling failures (if the job fails, a retry mechanism kicks in).
The PMLE exam tests your ability to distinguish between batch and online prediction, to design cost-effective batch pipelines, and to optimise them. You will see scenario-based questions where a company has a specific latency and cost requirement, and you must choose the right architecture.
Here are the exact concepts the exam loves to test:
Batch size vs. latency trade-off: They will give a scenario where increasing the batch size makes each individual prediction take slightly longer but dramatically improves total throughput. The correct answer is usually to increase batch size until you hit memory limits.
When to use batch vs. online prediction: If the business need is 'predictions must be available within 100 milliseconds', you choose online. If 'predictions must be ready within 4 hours', batch is correct. The trap is that they often describe a real-time need but recommend a batch solution in one of the distractors.
Vertex AI Batch Prediction capabilities: You need to know that Vertex AI accepts model artifacts stored in Cloud Storage, can handle CSV and JSON-Line input, and scales automatically based on the number of workers you configure.
Dataflow for batch inference: Sometimes they test that Apache Beam (the programming model behind Dataflow) can be used for batch prediction, and that it is best for complex preprocessing that needs to run across many machines.
Optimisation techniques: They ask literally 'Which of the following will reduce batch inference latency without degrading model accuracy?' Correct answers: 'using a larger machine type with more CPUs', 'enabling model parallelism'. Trap: 'using a smaller batch size' (that actually increases total latency for a large dataset).
Cost analysis: A question might give you the price of a machine per hour, the number of predictions, and ask you to choose the cheapest batch configuration that still meets the time constraint. You need to do simple arithmetic.
Common question patterns:
'You need to generate weekly churn predictions for 500,000 users. Predictions must be ready by 9 AM Monday. Which architecture is most cost-effective?' Correct: batch prediction with a job that runs overnight on preemptible/spot VMs (cheaper machines that can be shut down by the cloud provider).
'A model takes 2 seconds to warm up before it can make predictions. How does this affect batch design?' The correct answer is to send a small 'dummy' request to warm the model or to keep the model container running between batches to avoid cold starts.
'You are choosing between an online endpoint and a batch job. The dataset is 10 GB, the model is 500 MB, and you have a 30-minute deadline. What do you choose?' Correct: batch job, because sending 10 GB through an online endpoint one request at a time would take too long due to network overhead per request.
Key definitions to memorise:
Latency: time for a single prediction.
Throughput: number of predictions per unit time.
Batch size: number of predictions processed together in one forward pass.
Cold start: delay when loading a model into memory for the first time.
Vertex AI Batch Prediction: a managed service that takes input from Cloud Storage and writes output to Cloud Storage.
Preemptible/Spot VMs: low-cost compute instances that can be terminated at any moment; ideal for fault-tolerant batch jobs.
Batch prediction processes many requests together in a single job, maximising throughput while minimising cost, instead of handling requests one at a time.
The optimal batch size is the largest size that fits in the model's memory without causing an out-of-memory error; further increasing it does not improve throughput.
Preemptible or spot virtual machines are the most cost-effective choice for batch prediction because the job can tolerate interruptions.
Cold starts (model loading time) can be mitigated by sending a small warm-up request as part of the batch job configuration.
Model quantization (reducing numeric precision) speeds up inference with a negligible accuracy loss and is especially effective in batch settings.
For weekly or daily batch jobs, using a serverless service like Vertex AI Batch Prediction removes the need to manage infrastructure.
These come up on the exam all the time. Here's how to tell them apart.
Batch Prediction
Processes thousands to millions of requests together in one job.
Latency is measured from job start to job finish (minutes to hours).
Infrastructure is ephemeral; servers are torn down after the job.
Online Prediction
Processes one request at a time, returning a prediction immediately.
Latency is measured in milliseconds per request.
Infrastructure is always-on with load balancers and autoscaling.
Vertex AI Batch Prediction
Purpose-built for running model inference on large input files.
Minimal custom coding required (just provide model and input path).
Best for straightforward 'load data, predict, save results' scenarios.
Custom Batch Prediction with Dataflow
General-purpose stream and batch processing engine.
Requires writing a pipeline in Apache Beam (Python or Java).
Best when complex preprocessing or data joins are needed beyond simple inference.
Cold Start (No Model Warm-Up)
Model takes extra seconds to load from disk into memory at job start.
First batch of predictions may be significantly slower.
Overall job time increases by the loading time for each worker.
Warm Start (Model Already Loaded)
Model stays resident in memory between jobs or is pre-loaded.
No initial delay; predictions start immediately.
Requires keeping resources alive, which costs money during idle periods.
Mistake
Batch prediction is always slower than online prediction because you have to wait for the whole batch to finish.
Correct
Batch prediction is often faster at processing large volumes of data because it can use parallel computing and avoid the overhead of handling each individual request. It does have higher latency for the first prediction in the batch, but the total time to process 100,000 requests is much shorter than sending them one by one.
This mistake happens because people confuse 'individual prediction latency' with 'total batch throughput'. They think of the batch as a queue, but it is more like a factory assembly line.
Mistake
You cannot use a GPU for batch prediction because GPUs are only for real-time inference.
Correct
GPUs are actually even more efficient for batch prediction because they can process many predictions in parallel during each model forward pass. A single GPU can dramatically speed up a batch job. The real constraint is memory — the GPU must be able to hold the batch data.
This comes from the mistaken idea that GPUs are 'expensive' and only used for high-stakes real-time tasks. In fact, they are cost-effective for batch work because you only pay for the computation time, which is shorter.
Mistake
Batch prediction always requires the same infrastructure as real-time serving, just with a queue in front.
Correct
Batch prediction typically uses completely different infrastructure, often temporary resources that are torn down after the job finishes. Real-time serving requires always-on endpoints, load balancers, and autoscaling. Batch prediction uses job-based compute like Cloud Dataflow or a Vertex AI batch job, which spins up machines, runs the job, and shuts them down.
This confusion comes from the name 'prediction', which people associate with always-running servers. They do not realise the fundamental architectural difference between a 'service' and a 'job'.
Mistake
Optimising batch inference means making the model itself faster, such as by using a smaller model.
Correct
While model optimisation can help, batch inference optimisation is primarily about infrastructure choices: batch size, hardware type, parallelism, and pipeline design. You can optimise batch performance by grouping data efficiently, using larger machines, or caching preprocessing steps, all without touching the model's architecture.
Beginners tend to focus on the model (the algorithm) as the only lever. They overlook the fact that the data pipeline and compute configuration have a far bigger impact on batch job performance.
Mistake
Batch prediction cannot handle data that arrives in real time; you must always wait until the next scheduled batch window.
Correct
You can design a 'micro-batch' system that collects data for a short interval (e.g., 1 minute) and then processes it as a mini-batch. Services like Cloud Dataflow support this natively. It is not true real time, but it can provide near-real-time results with the efficiency of batching.
The word 'batch' implies 'big and slow' to many people. They do not realise that batching can be tuned to very small time windows for near-realtime use cases.
Reveal each answer, then mark whether you got it right. Score 60%+ to unlock the next chapter.
No. Batch prediction has too much overhead and latency for real-time applications. It is designed for use cases where users can wait minutes or hours for results, like generating weekly recommendations or monthly risk scores.
No, not if your model is already deployed in Vertex AI. You simply provide the input file location and the output file location in Cloud Storage, and the service handles the rest. For custom preprocessing, you need to provide a small preprocessing container.
The batch job will report an error. You typically need to restart it from the beginning, though Vertex AI gives you logs to identify the failed records. For critical jobs, you should implement retry logic and validate the input data before submission to reduce failures.
A GPU is faster for models that rely on deep learning or large matrix computations. A CPU is fine for simpler models like tree ensembles. Use a GPU if your model has many parameters and you want maximum throughput; use a CPU if cost is your primary concern and the model is small.
No. The machine learning model is loaded into memory at the start of the batch job and unloaded when the job finishes. This is a major advantage — you only pay for actual compute time, not for idle servers.
Yes, but you need to collect the streaming data into a storage bucket first and then trigger the batch job at regular intervals. Services like Dataflow can handle micro-batching, where data is processed in small batches every few minutes, combining the benefits of both approaches.
You've finished Batch Prediction and Inference Optimization. Continue through the PMLE study guide to build a complete picture of the exam.
Done with this chapter?