Data Exploration, Validation, and Quality Assessment

Data exploration, validation, and quality assessment form the three-part ritual that every machine learning engineer performs before they let a single algorithm touch the data. Think of it as the prep work before cooking: you check the fridge, you check the expiry dates, and you check whether the ingredients actually taste good. Skipping this step is like building a house without inspecting the foundation.

Let’s define each term in plain language.

Data exploration means looking at your data to understand its shape, size, and obvious problems. You open the file, count the rows, look at the column names, and glance at the first few entries. You ask questions like: how many rows are there? How many columns? Are the numbers plausible or are there entries like 'age = 999' where age should be between 0 and 120? You make summary statistics — average, minimum, maximum, and the most common values. This step gives you a gut feeling for whether the data is roughly what you expected.

Data validation means checking that each piece of data meets the rules you defined in advance. If a column is supposed to contain email addresses, validation checks that every entry has an '@' symbol and a domain name. If a column is supposed to be a date, validation checks that no entry says 'potato' or '32/13/2024'. This step uses a schema — a formal description of what each column should contain. The schema is your rulebook.

Quality assessment goes deeper than validation. Quality assessment asks: is this data 'good enough' for machine learning? Even if every value passes validation, the data might still be useless. For example, if you are building a model to predict house prices and every house in your data was sold in the same neighbourhood, the quality is low because the data lacks variety. Quality assessment involves measuring completeness (are there missing values?), consistency (does the same address appear in different formats?), and distribution (do the numbers follow a realistic pattern?).

Why do we separate these three steps? Because they catch different kinds of problems. - Exploration catches the 'obviously wrong' problems — a missing column, a file that won’t open, a column full of NULL values. - Validation catches the 'technically wrong' problems — a date in the year 3000, a negative price for a product, a country code that doesn’t exist. - Quality assessment catches the 'subtly wrong' problems — a dataset that has no examples of a particular customer type, a column where 90% of the entries are the same value, a timestamp column that only records data during business hours (giving you a biased view).

In the real world, these three steps often blend together. You might explore the data, notice missing values, run a validation check to count them, then decide that the missing values are too numerous for quality purposes. But it is useful to think of them as a pipeline: first you look, then you check, then you judge.

What does this process replace? Before rigorous data validation became standard in machine learning, engineers would often train models on whatever data they had, trusting that the source was correct. That approach led to disasters: models that recommended products that didn’t exist, fraud detection systems that flagged every transaction because the input dates were corrupted, medical diagnostic tools that failed because the training data had typos in patient IDs. The three-step process is a safety net.

For the Google Professional Machine Learning Engineer exam, you need to know the specific Google Cloud tools that help with each step. For exploration, engineers use BigQuery to query large datasets and Dataflow to inspect streams of incoming data. For validation, they use TensorFlow Data Validation (TFDV) to automatically compare new data against a schema. For quality assessment, they compute statistics like mean, median, standard deviation, and null count using tools like Cloud Data Loss Prevention (DLP) or custom Python scripts.

The key insight is that data preparation is not a one-time task. Data changes over time. A column that always had five possible values might suddenly contain a sixth value after a software update. A quality metric that was fine last month might degrade because a sensor broke. Therefore, professional ML engineers set up automated monitoring — pipelines that re-run validation and quality checks every time new data arrives.

Beginners often think that data exploration is the 'fun' part and validation is the 'boring' part. In reality, validation saves you from the most embarrassing failures. Imagine presenting a model to your boss, only to discover that the model was trained on data where prices were recorded in Euros while the production system uses Dollars. Validation would have caught that unit mismatch. Quality assessment would have flagged the abnormally high price variance. Exploration might have shown you the column name 'price_eur' — but only if you looked. The three steps together protect you from human oversight.

Let’s walk through what a real data engineer or ML engineer does on a Tuesday morning at a mid-sized company that sells gardening equipment online. The company wants to build a model that predicts whether a customer will buy a lawnmower in the next month.

The engineer starts by pulling the last three months of customer data from BigQuery. This is a table with columns: customer_id, age, postcode, purchase_history (a JSON string), page_views (an integer), last_purchase_date, and loyalty_tier. The first step is exploration. The engineer runs a quick SQL query:

SELECT COUNT(*) FROM customer_data;

That returns 450,000 rows. Good. Next, they check for obvious nonsense:

SELECT MIN(age), MAX(age), AVG(age) FROM customer_data;

Minimum age is 0, maximum age is 150. Red flag. Age 0 is impossible for a customer — it could be a default value or a placeholder. The engineer also notices that the postcode column has 5,000 distinct entries, but some are 4 digits and some are 6 digits. The schema says they should all be 5-digit UK postcodes.

Now the engineer moves to validation. They define a schema using TensorFlow Data Validation. The schema specifies:

The engineer runs TFDV on a sample of the data. It reports that 2% of age values are below 18, 0.5% of postcodes fail the regex check, and 1% of last_purchase_date entries are in 2027 (impossible since today is 2025). These are validation failures. The engineer has decisions to make: delete the bad rows, replace (impute) the bad values with NULL, or go back to the source database to fix the root cause.

Next, the engineer performs quality assessment. They look at the distribution of page_views. Most customers have between 1 and 50 page views, but there is a spike of exactly 100,000 for 200 customers. That seems suspicious — maybe a bot or a logging error. The engineer also checks loyalty_tier. There are three tiers: Bronze, Silver, Gold. But 60% of rows are NULL, meaning no tier was recorded. That is a quality problem: the model cannot use a feature that is missing for most customers.

The engineer also runs a check for class imbalance. The target variable — whether a customer bought a lawnmower — shows that only 5% of customers purchased one. That is a highly imbalanced dataset. The engineer notes that the model will need special techniques (like upsampling or weighted loss) to avoid always predicting 'no purchase'.

Finally, the engineer documents all findings in a data quality report. They decide to:

The entire process took two hours on Tuesday and will be repeated next month when new data arrives. The model that results from this cleaned dataset will be far more reliable than if the engineer had skipped these steps.