How do you get the massive amounts of raw data your machine learning model needs, from all its different sources, into a single, clean, reliably stored place that the model can actually use? That is exactly what data ingestion and storage solutions for ML solve. For your PMLE exam, understanding this process is critical because a model is only as good as the data it eats — and the exam will test you on how to design the pipeline that feeds it safely, efficiently, and at scale.
Jump to a section
A simple way to picture Data Ingestion and Storage for ML
Have you ever tried to coordinate a potluck dinner for fifty relatives, where each dish arrives at the right temperature and the dietary restrictions are met without a single phone call? That is the challenge of data ingestion and storage for machine learning. In the family scenario, you have multiple aunts bringing casseroles, cousins with salads, and your uncle’s famous barbecue brisket — all arriving at different times, in different containers, with varying shelf lives. Your job is to create a system that accepts every dish, checks it for freshness (no expired mayo!), records who brought what and when, and stores it in the right part of the kitchen — fridge for the cold dishes, oven for the hot ones, freezer for the ice cream. If you just pile everything on the counter, the brisket gets cold and the salad wilts. In data terms, you need a pipeline that ingests raw data from many sources (like streams, databases, and files), validates and cleans it, stores it in a central repository (like a data lake or warehouse), and makes it available in a consistent format for the machine learning model to ‘cook’ with. Your grandmother’s secret recipe never works if you use the wrong ingredients — similarly, a model fails if you feed it the wrong data.
So the next time you see the term ‘data ingestion’, think of that potluck: you are the host, orchestrating the arrival, inspection, and storage of countless dishes so the final feast (the trained model) is delicious.
Let’s strip away the fancy terminology and talk about what data ingestion and storage actually means for someone studying machine learning.
At its heart, data ingestion is the process of getting data from where it is created to somewhere you can use it for training or inference. Think of a machine learning model as a chef. A chef cannot cook without ingredients. The ingredients arrive from different farms (sensors, databases, user clicks, log files) in different forms (bags of flour, crates of eggs, jars of spices). Data ingestion is the system that receives all those ingredients, checks they are not spoiled, labels them, and places them in the pantry (storage) so the chef can grab them later.
Now, why does this matter for the PMLE exam? Because in real business scenarios, you are not just dealing with one single data source. You might have:
A stream of real-time data from a website showing user clicks.
A batch of daily transaction records from a database.
A collection of images uploaded to cloud storage.
Log files from thousands of servers.
Each of these arrives at different speeds (some trickle in constantly, some dump a huge pile once a day). The ‘injection’ part means you need a method to pull or receive this data. There are two main patterns you will encounter on the exam: batch ingestion and streaming ingestion.
Batch ingestion is like a scheduled delivery. You wait until a certain time (once an hour, once a day) and then load all the data that has accumulated since the last load. This is simpler and cheaper, but it means your model is always slightly behind the latest data. The exam loves to test this in scenarios where you are training a model that does not need to be updated in real time — for example, a weekly sales forecast model that uses all of last week’s transactions.
Streaming ingestion is like a conveyor belt. Data arrives continuously, in small pieces, and is immediately processed and stored. This is for use cases where milliseconds matter — like fraud detection on credit card swipes or a recommendation system that updates as a user browses. The exam will ask you to choose between these patterns based on latency requirements.
Once the data is ingested, it has to be stored somewhere. The storage layer for ML is typically one of two things: a data lake or a data warehouse.
A data lake is like a giant warehouse where you can dump anything — raw text, images, video, compressed files — without needing to structure it upfront. It is cheap and flexible, but it can become a swamp if you do not organise it carefully. In Google Cloud, Cloud Storage is the most common data lake service.
A data warehouse is more like a tidy filing cabinet. It expects data to be structured in rows and columns, with a schema defined in advance. This makes querying fast and efficient, but it is more expensive and less flexible. BigQuery is Google’s flagship data warehouse.
For ML workflows, you often use both: land raw data in a lake first, then transform and extract a smaller, cleaner dataset into a warehouse or directly into a feature store (a special database that serves features to models). The exam will test you on when to use each.
Another key concept is the feature store. A feature store is a centralised repository where you store and manage the features (input variables) that your model uses. Instead of every team rebuilding the same features from scratch, they can reuse, share, and version them. This is hugely important for production ML because it prevents data inconsistency (different models using different definitions of the same feature) and speeds up development.
Finally, you must consider data governance and security. In a real project, the data comes with rules about who can see it (think of medical records or credit card numbers). The exam will test you on using services like Cloud IAM (Identity and Access Management) to control access, and Cloud KMS (Key Management Service) to encrypt data at rest and in transit. You will also need to know about data versioning (keeping multiple snapshots of your data over time so you can reproduce an old model’s results) and data lineage (tracking where data came from and how it was transformed).
In simple terms: ingestion gets the data in, storage keeps it safe and organised, and together they form the foundation of any ML pipeline. Without this, the model is a chef with an empty kitchen.
Let’s say you are working for a company called ‘ShopStream’, an online fashion retailer. They want to build a machine learning model that predicts what items a customer is likely to buy next, so they can show personalised recommendations in real time. As the sole data engineer supporting this, you need to design the data ingestion and storage solution.
Your day starts with a meeting. The product manager tells you that the website generates two types of data:
User click events: every time someone clicks a product, it creates a tiny JSON record with the user ID, product ID, timestamp, and the page they were on. These come in continuously — thousands per second.
Historical purchase data: stored in a relational database, with years of past orders. This data is updated daily and you only need it once to train the initial model, then perhaps weekly retraining.
You decide on a hybrid approach. For the click events, you need streaming ingestion because the recommendations must be up-to-date within seconds. You configure a Cloud Pub/Sub topic (a message queue that collects incoming events) to receive the click data from the website’s front end. From there, you set up a Dataflow streaming job that cleans and transforms the raw events — adding a field for the product category, filtering out bot traffic, and aggregating events per user every few minutes. The cleaned data is then written to BigQuery (for training) and also to a feature store (for real-time prediction).
For the historical purchase data, you choose batch ingestion. Once a day, you run a Cloud Composer (Airflow) pipeline that extracts the new rows from the database, validates the schema, and dumps them into Cloud Storage as Parquet files. From there, you load them into BigQuery for the weekly retraining job.
Now, the storage part. You use Cloud Storage as your data lake: it holds the raw, unprocessed parquet files, the backup of the historical database, and any images of products that might be used for a future vision model. You use BigQuery as your data warehouse for analytics and training. And you use Vertex AI Feature Store for the cleaned, aggregated features so that both training and serving can access them with low latency.
A critical piece is security. Because ShopStream processes payment data, you must ensure that only authorised services and people can read or write the datasets. You create IAM roles and assign them carefully: your data scientists get read access to BigQuery training datasets but not to Cloud Storage raw logs, which contain customer IP addresses. You also set up encryption with Cloud KMS, using a customer-managed key so the company stays in control.
After deploying the pipeline, you monitor it using Cloud Logging and set up alerts. One day, you notice a spike in ingestion latency — the streaming jobs are falling behind. You look at the metrics and realise the Pub/Sub subscription has too few partitions. You scale up the number of partitions, and the latency drops back down. This is the kind of real-world troubleshooting that the PMLE exam tests indirectly through scenario questions.
By the end of the quarter, the recommendation model is live, using features from the feature store and fresh click data from the streaming pipeline. The business sees a 15% increase in average order value, and you have built an ingestion and storage architecture that can scale as the company grows.
The Google Professional Machine Learning Engineer exam focuses heavily on data ingestion and storage because it is the foundation of any ML system. You will see multiple scenario questions where you must choose the right combination of services and patterns. Here is exactly what they test and how they set traps.
First, they love testing the difference between batch and streaming ingestion. A typical question will describe a use case: ‘A company needs to train a model on customer transactions that arrive in real time. They want the model to update predictions every five minutes. What ingestion method should they choose?’ The answer is streaming, because even though the model updates every five minutes, the data must be ingested continuously to capture each transaction instantly. The trap is that some candidates see a five-minute window and think batch is fine. But batch would mean waiting five minutes to even see the data, then another batch window to process — too slow.
Second, they test your understanding of storage choices: data lake versus data warehouse. The exam might give you a scenario where the data contains unstructured text, images, and structured rows. They ask: ‘Which storage solution should you use as the primary landing zone for raw data?’ The answer is a data lake (like Cloud Storage), because it supports all formats without upfront schema. The trap is that candidates often misread and pick BigQuery (a warehouse) because it sounds more advanced. But BigQuery requires a defined schema — you cannot just dump arbitrary JSON into it easily.
Third, the exam tests feature store concepts. You will see questions like: ‘A team of data scientists each builds their own version of the same feature (e.g., ‘customer_lifetime_value’). They get different prediction results. What should you implement to solve this?’ The correct answer is a feature store, because it centralises feature definitions, versions, and computation. The trap is that candidates sometimes suggest a data warehouse or a shared spreadsheet, but the exam expects the specific architectural component designed for this — the feature store.
Fourth, they test data governance and security. You will be asked how to restrict access to sensitive data in a data lake. Look for answers involving IAM roles, Cloud KMS encryption, and audit logging. A common trap is picking a ‘firewall’ or ‘VPN’ — these are network-level controls, not granular data access controls. The exam wants you to name the correct Google Cloud service: Cloud IAM for permissions, Cloud KMS for encryption keys.
Fifth, they test data versioning and reproducibility. Questions will ask: ‘How do you ensure that you can re-run an old training job and get the exact same results?’ The answer is to snapshot the training data at a point in time, for example by storing it in Cloud Storage with a timestamped directory, and by versioning the code and pipeline configuration. The trap is suggesting that the model itself is versioned but forgetting that the training data must be versioned too.
Sixth, the exam tests your knowledge of data transformation patterns (ETL vs ELT). ETL means you transform the data before loading it into the warehouse; ELT means you load first, then transform. In Google Cloud, with BigQuery’s massive processing power, the preferred pattern is often ELT because BigQuery can handle the transformations efficiently. A question might ask: ‘You must ingest raw logs and quickly make them queryable. Should you use Dataflow for transformation first (ETL) or load to BigQuery then transform (ELT)?’ The correct answer is often ELT for performance reasons, but the trap is that candidates automatically pick ETL because it is the older, more common pattern.
Finally, they test your understanding of message queues and event-driven architectures. Pub/Sub is Google’s primary service for ingesting streaming data. Questions will ask about its properties: at-least-once delivery, push vs pull subscriptions, and ordering. The trap is that candidates confuse Pub/Sub with a database — it is a queue, not a store.
To prepare, memorise these key concepts: batch vs streaming, data lake vs data warehouse, feature store, Cloud Storage, BigQuery, Pub/Sub, Dataflow, Cloud Composer, IAM, KMS, and data versioning. Practise with scenario questions from Google’s official sample exam and build a simple pipeline with free tier resources to see how they interconnect.
Mistake
Data ingestion is just copying files from one place to another, so standard file transfer tools like FTP are sufficient.
Correct
Data ingestion involves validation, transformation, schema management, error handling, and often real-time streaming. Simple file copy does not handle schema evolution, data quality checks, or the scale needed for ML.
Beginners often think of 'ingestion' as synonymous with 'upload'. They have not experienced the complexity of multiple data formats, corrupted files, or latency requirements.
Mistake
A data lake and a data warehouse are essentially the same thing — just storage for data.
Correct
A data lake stores raw data in its native format (unstructured or semi-structured) and is optimised for cheap mass storage. A data warehouse requires a defined schema, indexes data for fast SQL queries, and is typically more expensive per gigabyte.
In conversation, both are often called 'big data repositories', so beginners assume they are interchangeable. They do not realise that using a warehouse for raw images would be both expensive and inefficient.
Mistake
You only need to worry about data ingestion once, when you first build the model. After that, the data is static.
Correct
In production ML, data ingestion is an ongoing process — new data arrives continuously (streaming or daily batches). You need a pipeline that keeps running, monitoring for schema changes, data drifts, and failures.
Many beginners think of ML as a one-time project (like a school assignment). They do not understand that production models require constant feeding with fresh data to stay accurate.
Mistake
If you use a feature store, you can skip data ingestion entirely because the features are already computed.
Correct
Feature stores are a form of storage and feature management. You still must ingest the raw data into the feature store (through batch or streaming pipelines) to compute and update the features. The feature store does not eliminate ingestion; it organises the output of ingestion.
The word 'store' makes it sound like a finished product, like buying groceries from a store. In reality, you still need to bring the raw ingredients to the store first.
Mistake
Data encryption is optional for ML training data because the data is only used internally.
Correct
Encryption is a mandatory security practice for any data that contains personally identifiable information (PII), financial data, or sensitive business information. Many industries (healthcare, banking) have legal requirements for encryption, and Google Cloud encrypts data at rest by default.
Beginners often assume that 'internal use' means 'safe from outsiders', forgetting that internal threats, accidental exposure, or compliance fines can occur without encryption.
You've finished Data Ingestion and Storage for ML. Continue through the PMLE study guide to build a complete picture of the exam.
Done with this chapter?