If you build a machine learning model without a central way to manage your data ingredients, you will end up with predictions that contradict each other, models that behave differently every time you retrain them, and a debugging nightmare that wastes weeks of work. This chapter explains Vertex AI Feature Store, a managed service that solves exactly that problem by acting as a single, consistent library of features for all your models. For the PMLE exam, understanding how and why to design a feature store is critical for the objective of ensuring consistency and reusability across your organisation's ML projects.
Jump to a section
A simple way to picture Managing ML Features with Vertex AI Feature Store
A busy restaurant kitchen during Friday night dinner service. The head chef needs to prepare three different dishes: a steak, a pasta, and a salad. Each dish uses some of the same ingredients – the steak and the salad both need the same special vinaigrette, the pasta and the steak both use the same freshly grated parmesan cheese, and all three need the same chopped herbs.
Without a central store, each station (grill, pasta, salad) keeps its own stash of these ingredients. The grill station keeps a jar of vinaigrette they made this morning, the pasta station has a different batch they made yesterday, and the salad station forgot to make any. When the head chef calls for vinaigrette, each station hands over a different version. One is too sour, one is too oily, and one is missing. The dishes come out inconsistent, and customers complain that the steak tastes different each time.
Now imagine a single, labelled shelf in the walk-in cooler, right by the pass. Every station knows: 'If the recipe calls for vinaigrette, get it from that shelf.' The shelf has one version, made fresh each morning by the prep cook, and every station uses the exact same jar. The parmesan is pre-grated and stored in a single container. The herbs are mixed in one batch. The head chef shouts 'vinaigrette!' and every station grabs from the same source. The dishes are consistent, the customers are happy, and nobody wastes time remaking ingredients they could just fetch. That central shelf is your feature store – a single source of truth for every ingredient your machine learning models need to produce reliable predictions.
Let's break down what a feature store actually is. A 'feature' is a specific, measurable piece of data that you give to a model to help it make a prediction. For example, if you are building a model that predicts house prices, features might include 'number of bedrooms', 'square footage', and 'distance to nearest school'. In a typical company, different teams build different models. One team might build a fraud detection model, another a customer churn model, and another a product recommendation system. All three models might need the same feature, say 'average transaction amount for this customer in the last week'.
Without a feature store, each team would write their own code to calculate that average transaction amount. They might get the raw data from different databases, use slightly different formulas, or update the value at different times. The result is that the fraud model might use a feature value of £150, while the churn model uses £120 for the same customer at the same moment. This inconsistency leads to unreliable predictions and makes it impossible to compare model performance across teams.
Vertex AI Feature Store is a managed Google Cloud service that acts as a central repository for features. It stores the feature definitions, the actual values (for both training and serving), and the metadata about how features are computed. It handles two main types of data. The first is 'online serving' – this is when your deployed model needs a feature value in milliseconds to make a real-time prediction, like deciding whether to approve a credit card transaction. The second is 'offline training' – this is when you are training a model on historical data, and you need to fetch millions of feature values at once.
Here is how it works in practice. A data engineer defines a feature (for example, 'customer_avg_spend_7d') and writes the code to compute it from raw transaction logs. This definition is registered in the feature store. The feature store then automatically computes the feature value for every customer every day and stores it. When the fraud team deploys their model, the model's serving code asks the feature store for the 'customer_avg_spend_7d' value for the specific customer making this transaction. The feature store fetches the pre-computed value and returns it in under 10 milliseconds. When the churn team wants to retrain their model, they run a batch job that queries the feature store for all historical values of that feature for the past year. They get a consistent dataset, computed the exact same way.
The key concept here is a 'single source of truth'. With a feature store, you know that every model using the 'customer_avg_spend_7d' feature is using the exact same definition and the exact same computed value. This eliminates the 'training-serving skew', which is when your model learns from data that looks different from the data it sees when making predictions. For example, if your training data computed the average over 7 calendar days but your serving code accidentally used the last 7 business days, the model's predictions would be wrong. The feature store enforces the same computation both times.
What did people use before feature stores? Teams would copy feature values into large CSV files or database tables, share them over email or network drives, and manually update them. This was slow, error-prone, and made it nearly impossible to track which model used which version of a feature. Feature stores automate this entire process, provide a clear audit trail of when features were computed and changed, and allow teams to share features without duplicating work. On the PMLE exam, expect questions that test your understanding of the trade-offs between online and offline serving, how to ensure consistency, and the role of feature stores in the ML lifecycle – from data preparation through model training to production inference.
Define Your Entity and Feature Group
Identify the entity your features describe, such as a customer_id or product_id. This step is crucial because the feature store uses the entity key to look up values. You then create a feature group to logically group related features (e.g., 'customer_transaction_features'). This groups features by source or use case, making them easier to manage and share.
Register Individual Features
For each feature you want to store (e.g., 'avg_spend_30d'), you register it within the feature group. You specify its data type (integer, float, string), a description, and the source table in BigQuery where the raw data lives. This registration step creates a metadata record that the feature store uses to understand how to fetch and serve the feature.
Write and Deploy the Feature Computation Code
You write a Python function that transforms raw data into the feature value. For example, a function that reads raw transaction rows and calculates a moving average over 30 days. You then deploy this function to Vertex AI Feature Store – it runs on a schedule you define (e.g., nightly) to compute new values from the latest raw data. This ensures the feature store stays updated without manual effort.
Configure Online and Offline Serving
You configure the feature store to enable online serving (for real-time predictions) and offline serving (for batch training). The online store uses Bigtable for low-latency lookups. The offline store uses BigQuery for scalable analytical queries. This separation is critical: you optimise each store for its specific workload, which is why you cannot use a single database for both.
Ingest Feature Values
The feature store automatically ingests the computed values into both the online and offline stores. For online, it stores only the latest value per entity. For offline, it appends the new value with a timestamp to the historical record. This step happens on your defined schedule, or you can trigger it manually for immediate updates.
Serve Features to Models for Training and Prediction
During training, you run a batch job that queries the offline store for all historical feature values. During prediction, your model serving code queries the online store for the latest feature value for a specific entity key. Both queries use the same feature definition, ensuring complete consistency. The feature store handles the API calls and returns the data within your latency requirements.
Monitor Feature Quality and Drift
After deployment, you periodically check the distribution of feature values in the online store. If the distribution changes significantly (drift), it may indicate data quality issues or concept drift in the underlying business process. The feature store's historical snapshots allow you to compare current distributions to past ones, which is a key capability for maintaining model performance over time.
Imagine you work as a machine learning engineer at a large online bank called 'SwiftSavings'. The bank has three main ML models in production. First, a fraud detection model that checks every transaction in real-time. Second, a loan approval model that runs when a customer applies for a personal loan. Third, a retention model that identifies customers likely to close their account next month. Each model needs some of the same features, like 'total account balance', 'number of recent transactions', and 'days since last login'.
Before adopting Vertex AI Feature Store, your team faced chaos. The fraud team stored their computed features in a Redis cache, the loan team saved theirs in a PostgreSQL database, and the retention team computed them fresh every night from a data warehouse using a script that sometimes broke. One Monday, the fraud model started flagging 10% of all transactions as suspicious for no reason. After three days of investigation, the team discovered that the loan model had accidentally overwritten a shared database field that the fraud model was reading. This kind of incident happened every few months, costing the bank thousands in lost revenue and customer trust.
The VP of Engineering mandated a move to Vertex AI Feature Store. Here is what the team did step by step. - They created a single feature store instance in Google Cloud. This is a container that holds all feature definitions and values. - The data engineering team defined a feature group called 'customer_activity'. They specified that every feature in this group should be computed nightly from the bank's transaction logs. They wrote a single piece of code that computes the features, and this code is now stored and versioned in the feature store. - Within the 'customer_activity' group, they registered specific features like 'transaction_count_7d', 'avg_spend_30d', and 'days_since_last_txn'. Each feature had a clear description, a data type (integer, float, string), and a source table. - They configured the feature store for online serving. This means it uses a high-speed Bigtable instance under the hood to return feature values in milliseconds. They also configured offline serving, which uses BigQuery to handle large batch queries for retraining. - Now, when the fraud model receives a transaction request, the serving wrapper does the following: it creates a key from the customer's ID and the timestamp, sends a request to the feature store for the features 'transaction_count_7d' and 'avg_spend_30d', gets the values back in under 5 milliseconds, and includes them in the prediction request. The loan model does the same thing when a customer submits an application. The retention team runs a weekly job that queries the feature store for all customer features from the last six months and uses them to retrain their model.
The result? No more inconsistent values. No more debugging data issues that turn out to be feature computation differences. The team now spends their time improving models instead of fighting data silos. The PMLE exam tests your ability to design this kind of system – understanding when to use online vs offline serving, how to monitor feature quality, and how to ensure that features are reusable and consistent across teams.
The PMLE exam objective 2.1 focuses specifically on designing and implementing a feature store for consistency and reusability. Here is exactly what you need to know to pass the questions on this topic.
The exam will test your understanding of three core scenarios. First, scenario A: a company has multiple teams developing models that use overlapping features. The correct answer always involves using a feature store to centralise feature definitions and ensure consistency. They will present a wrong answer where each team maintains their own feature computations – that is a trap. Second, scenario B: you are given a problem where models in production produce different predictions for the same input feature values. The right answer points to training-serving skew and recommends using a feature store to serve the same feature values during training and inference. Third, scenario C: a company wants to share a feature across projects or regions. The exam tests your knowledge of how Vertex AI Feature Store supports sharing across projects using IAM permissions.
Exam traps you must avoid:
Trap 1: 'Store features in a flat file on Cloud Storage for reuse.' This is wrong because it does not support low-latency serving and versions are hard to manage.
Trap 2: 'Use a single database table for both training and serving.' This is wrong because databases are not optimised for the high-throughput, low-latency reads needed for online serving – you need the specialised online store (Bigtable).
Trap 3: 'Manually copy feature definitions between projects.' This is wrong because it breaks consistency – the whole point of a feature store is to have a single source of truth.
Trap 4: 'Ignore timestamping features.' The exam expects you to know that features must be timestamped to support point-in-time lookups, which is how you avoid using future data when training a model.
Concepts the exam loves: - 'Online serving' vs 'offline serving'. Be able to explain the latency, throughput, and use case differences. - 'Feature ingestion' – the process of computing and storing feature values. Understand batch ingestion (e.g., nightly jobs) vs streaming ingestion (e.g., real-time events). - 'Feature monitoring' – the practice of tracking feature value distributions over time to detect drift. The exam might ask how a feature store can help with this by storing historical snapshots. - 'Point-in-time correctness' – the principle that when training a model, you should only use features that were available at the time of the prediction. A feature store that stores timestamps enables this. - 'Sharing features across projects' – you can use IAM roles to grant access to feature stores in different Google Cloud projects.
Key definitions to memorise:
Feature store: a managed repository for storing, sharing, and serving ML features at scale.
Feature group: a logical grouping of related features (e.g., 'customer_profile' features).
Feature value: the actual numeric or string value of a feature at a specific point in time.
Entity: a unique object that a feature describes (e.g., a customer ID, a product ID).
Exam questions might present a case study where a company has 500 features used by 20 models. The correct response will almost always involve using Vertex AI Feature Store to reduce duplication and ensure consistency. They will also test you on the trade-off: using a feature store introduces some latency for online predictions (because you have to fetch the feature from the store), but this is usually acceptable (under 10ms) compared to the benefits of consistency. Look for answers that mention 'single source of truth', 'feature reusability', and 'eliminating training-serving skew'.
A feature store is a single source of truth for feature definitions and values, ensuring all models use identical data.
Vertex AI Feature Store supports two serving modes: online (sub-10ms latency for real-time predictions) and offline (batch queries for model training).
Using a feature store eliminates training-serving skew by computing features the same way for both training and inference.
Features in a store are timestamped, enabling point-in-time lookups that prevent your model from using future data during training.
You can share features across Google Cloud projects using IAM permissions, which promotes reusability across teams.
Feature stores store only the latest value for online serving but keep full history for offline training.
You must write custom code to compute features; the feature store orchestrates and stores the results, but does not invent features.
Monitoring feature value distributions over time is a critical use case supported by the feature store's historical snapshots.
These come up on the exam all the time. Here's how to tell them apart.
Online Serving
Used for real-time predictions, with latency under 10 milliseconds.
Stores only the latest feature value for each entity.
Uses Bigtable as the underlying storage engine for low-latency reads.
Offline Serving
Used for model training, with high throughput for large batch queries.
Stores full historical record of feature values with timestamps.
Uses BigQuery as the underlying storage engine for scalable analytical queries.
Vertex AI Feature Store
Provides both low-latency (online) and high-throughput (offline) storage in one managed service.
Automatically handles feature versioning, point-in-time lookups, and metadata management.
Integrates natively with Vertex AI for model training and prediction workflows.
Regular Relational Database (e.g., Cloud SQL)
Designed for general-purpose transactional or analytical workloads, not ML feature serving.
Requires manual schema design for versioning and does not natively support point-in-time lookups.
May require additional custom code and infrastructure to serve features with ML latency requirements.
Batch Feature Ingestion
Computes and stores features on a fixed schedule (e.g., nightly).
Suitable for features that change slowly (e.g., customer demographic data).
Lower operational cost and simpler to implement than streaming.
Streaming Feature Ingestion
Computes and stores features in real-time as events occur.
Suitable for features that change rapidly (e.g., real-time transaction count).
Requires more complex infrastructure (e.g., Pub/Sub, Dataflow) and incurs higher cost.
Feature Store in Vertex AI
Fully managed by Google Cloud - no infrastructure to manage.
Integrated with Vertex AI Pipelines and Model Registry.
Supports Google Cloud-native services like BigQuery and Bigtable only.
Feature Store via Feast (Open Source)
Open source, can be self-hosted on any cloud or on-premises.
Requires manual setup and maintenance of infrastructure.
More portable across cloud providers, but lacks deep integration with any single platform.
Mistake
A feature store is just a database, like MySQL or PostgreSQL, that I can set up myself.
Correct
A feature store is a specialised system with separate storage engines optimised for online (low-latency) and offline (high-throughput) serving, along with versioning, point-in-time lookups, and IAM-based sharing. A regular database lacks these capabilities.
Beginners often think any data storage can act as a feature store because the term 'store' sounds generic. They underestimate the engineering complexity of serving features at ML scale.
Mistake
I can reuse features across models by just copying the CSV file I used for training.
Correct
Copying a CSV file means you lose the ability to serve that feature in real-time for online predictions, you have no version history, and you cannot enforce point-in-time correctness. A feature store handles all of this automatically.
People often start with small projects where copying a file seems fine. They don't realise that at production scale, this approach leads to versioning chaos and data inconsistency.
Mistake
The feature store's online store and offline store must contain the same feature values at all times.
Correct
The online store is typically a subset of the offline store, containing only the latest value for each entity. The offline store holds the full historical record, including daily snapshots. They are not identical in content.
It feels logical that 'the store should have one truth', but the engineering design prioritises speed for online access by keeping only the latest value, while storing history for training.
Mistake
You only need a feature store if your team is huge.
Correct
Even a single model with multiple features benefits from a feature store because it ensures consistency between training and serving. As soon as you have two models, the benefits multiply significantly.
People assume feature stores are only for 'enterprise' scenarios. They don't recognise that the main problem – training-serving skew – can happen even with one model.
Mistake
Vertex AI Feature Store can magically compute any feature from raw data without any code.
Correct
You still need to write custom code (usually a Python function) to compute the feature from raw data. The feature store provides the infrastructure to run that code on a schedule, store the results, and serve them – but it does not generate feature definitions automatically.
Beginners often overestimate the intelligence of managed services. They think 'managed' means 'no work'. In reality, the data engineering effort shifts from managing storage to defining features.
Reveal each answer, then mark whether you got it right. Score 60%+ to unlock the next chapter.
A feature store is specialised for ML features: it separates storage into an online store (low-latency for predictions) and an offline store (high-throughput for training), it automatically timestamps every value for point-in-time correctness, and it manages feature versions and metadata. A normal database lacks these features.
No. Vertex AI Feature Store provides APIs that your model's training pipeline and serving code can call directly. It handles the fetching automatically, so you don't need to manually copy or move data.
Yes. You can share a feature store across projects by configuring IAM permissions. This allows teams in different projects to access the same features while maintaining a single source of truth.
If the feature store is unavailable, your model will not be able to fetch required feature values and will likely return an error. To mitigate this, you should configure a fallback strategy, such as caching recently seen feature values on your serving instance, or using a lower-latency secondary store.
It depends on the feature. Frequently changing features (e.g., transaction counts) might be updated hourly. Stable features (e.g., account creation date) might be updated daily or weekly. Vertex AI Feature Store lets you define ingestion schedules per feature group.
Vertex AI Feature Store is a Google Cloud managed service, but the concept of a feature store exists on other platforms (e.g., Feast is an open-source alternative). For the PMLE exam, you only need to focus on Google Cloud's implementation.
You've finished Managing ML Features with Vertex AI Feature Store. Continue through the PMLE study guide to build a complete picture of the exam.
Done with this chapter?