# Data Quality, Cleaning, and Preparation for AI

> Chapter 5 of the Courseiva SALESFORCE-AI-ASSOCIATE curriculum — https://courseiva.com/learn/salesforce-ai-associate/data-quality-and-preparation

**Official objective:** 2.1 — Describe the importance of data quality and how to prepare data for AI models.

## Introduction

Without clean data, your AI model won't just be a little wrong — it will be dangerously, expensively, embarrassingly wrong. A model trained on messy data is like a GPS built with a map of the wrong city. This chapter explains why data must be clean and structured before it feeds into an AI system, so you can confidently answer exam questions about data readiness.

## The Restaurant Kitchen Analogy

A busy restaurant kitchen on a Friday night. The head chef shouts an order for a well-done steak with a side of mashed potatoes and a house salad.

The new prep cook grabs a steak from the fridge, slaps it on the grill, and microwaves a bag of frozen potato flakes. The salad? The lettuce is brown and wilted, so he just leaves it out. The steak is burnt on the outside and raw inside. The mashed potatoes are lumpy. A different waiter comes back with the ticket for "Table 4, medium-rare" and the cook thinks he's making another "well-done" because the ticket was smudged with ketchup and hard to read. He sends out a second wrong dish.

This is a data quality disaster. The head chef (the AI model) can only cook what it's given. If the ingredients are rotten (bad data), the recipe instructions are missing (missing data), or the ticket is illegible (inconsistent formatting), the output is guaranteed to be a mess. Preparing data for AI is like prepping a kitchen: you inspect every vegetable, measure every spice, and make sure every ticket is clear and complete before anyone starts cooking. You standardise how you chop onions so every batch cooks evenly. You label and date everything. Without this prep, your AI model is that overwhelmed cook sending burnt steak to the wrong table.

## Core explanation

Data quality is the measure of how suitable a dataset is for its intended purpose. For AI, high-quality data means the model can learn accurate patterns. Low-quality data teaches the model the wrong patterns, a problem known as "garbage in, garbage out" (GIGO). If you feed an AI system a thousand customer support tickets, but half of them are typed in all capitals with random punctuation (e.g., "HELP!!! MY ACCOUNT IS BROKEN????"), the model might learn that aggressive punctuation is a normal part of a support ticket. It will then fail to recognise polite inquiries.

Data cleaning is the practical process of fixing or removing incorrect, corrupted, improperly formatted, duplicate, or incomplete data within a dataset. Think of it as editing a messy spreadsheet before you can analyse it. Common cleaning tasks include:

- Removing duplicate records (e.g., the same customer listed twice with slightly different spellings)
- Fixing typos and misspellings
- Standardising formats (e.g., dates like "01/15/2024" vs. "15 Jan 2024" vs. "2024-01-15" all become one format)
- Handling missing values (deciding whether to fill them in, mark them, or delete those rows)

Data preparation is a broader process that includes cleaning but also covers transforming raw data into a structure the AI model can process. This often means converting text into numbers, because most AI models don't understand words — they understand numbers. This conversion is called "feature engineering". For example, a column of city names like "London", "Paris", "Tokyo" might be converted to three separate columns with a 1 or 0 in each, a technique called one-hot encoding.

Why does this exist? Early AI models were trained on small, manually curated datasets where data quality was less of an issue. As AI moved into business applications handling millions of records from dozens of sources (customer databases, web forms, purchase histories, chat logs), the data became messy and inconsistent. Organisations realised that 80% of a data scientist's time was spent cleaning and preparing data, not actually building models. Automated data cleaning tools and pipelines were developed to make this process repeatable and scalable.

What does it replace? It replaces the old assumption that you could just "dump data" into a system and get useful insights. In the past, business analysts might manually eyeball a report and catch obvious errors. AI does not have that common sense — it will happily learn the error as truth. Data preparation formalises the steps needed to make data trustworthy for automated decision-making.

A key concept is the "data pipeline". This is a series of steps that moves data from its raw source (like a database or a web form) through cleaning, transformation, and validation, until it is ready for the AI model. Each step is like a checkpoint in a factory assembly line, ensuring quality before the product moves to the next stage.

Another important idea is data profiling. Before you clean anything, you need to understand what you're working with. Profiling involves scanning the dataset to find patterns, missing values, outliers (data points that are far outside the normal range), and inconsistencies. For example, a profile might reveal that the "age" column contains values like 500 or -3, which are clearly errors.

For the AI Associate exam, you need to grasp that data quality directly impacts model accuracy and fairness. If the training data is biased (e.g., mostly male names in a hiring dataset), the AI will learn to favour male candidates. Cleaning alone cannot fix biased data if the bias is baked into the collection method. You also need to understand that AI models typically require data in a structured format (rows and columns) and that different model types can handle different levels of messiness, but none handle truly garbage data well.

## Real-world context

A typical scenario: Sarah works for a medium-sized insurance company that wants to build an AI model that predicts which customers are likely to cancel their policies next month (a process called churn prediction). The company has several data sources: a customer relationship management (CRM) system with contact info, a billing system with payment history, a claims database with claim records, and a call centre log of support calls.

Step one: Data collection and profiling. Sarah exports all relevant data into one large spreadsheet. She runs a data profiling tool and immediately spots problems:

- Customer names appear in the CRM as "John Smith" but in the billing system as "J. Smith" and "Smith, John"
- The claims database has many missing dates where the claim was filed
- The call centre log uses free text ("they were angry about the delay") instead of categorised tags
- Some customer IDs appear in the billing system but not in the CRM, meaning they might be orphans

Step two: Data cleaning. Sarah writes a set of rules to standardise the customer names, picking one format (e.g., "first name last name"). She decides to drop rows with missing claim dates because she cannot backfill them reliably. She converts the free text call notes into a few simple categories (e.g., "billing question", "complaint", "policy change") using a set of keywords.

Step three: Handling duplicates. The data profiling revealed several customers with two entries. Sarah uses a rule based on email address and phone number to merge duplicates, keeping the most recent information and flagging the older entry as archived.

Step four: Feature engineering. Sarah wants the AI model to consider customer behaviour over time. She calculates new columns (features) like "number of claims in the last 12 months", "average premium amount", and "number of support calls in the last 3 months". She also converts the category "policy type" (e.g., auto, home, life) into three separate columns using one-hot encoding.

Step five: Validation and splitting. Before feeding the data to the model, Sarah checks for any remaining anomalies: for example, a customer with a negative premium value (clearly a data entry error from the billing system). She removes those few records. Then she splits the cleaned dataset into three parts: a training set (70%), a validation set (15%), and a test set (15%). The training set teaches the model the patterns. The validation set helps tune the model's settings. The test set gives an honest final assessment of how well the model will perform on completely new data.

Throughout this process, Sarah documents every change she made. This is called data lineage. It is crucial for auditing and regulatory compliance. If regulators ask why a customer was offered a different price, Sarah can trace the data back to the original sources and the cleaning steps applied.

The model is trained on the clean data. It achieves a churn prediction accuracy of 85%. Without the cleaning and preparation, the accuracy would likely have been below 60% — worse than a random guess. The business saves millions by targeting retention offers to customers the model flags as high-risk.

## Exam focus

The AI Associate exam tests your understanding of data quality, cleaning, and preparation at a conceptual level. You will NOT be asked to write code or manually clean a dataset. The questions are multiple-choice, and they focus on why these processes matter and what the common pitfalls are.

Key topics that appear frequently:

- GIGO (garbage in, garbage out): The single most important concept. You must know that poor quality data leads to poor model performance. Expect at least one question that directly tests this.
- Bias in training data: If the data does not represent the real world fairly, the model's predictions will be unfair or inaccurate. For example, a hiring AI trained on data from a company that historically only hired men will learn to reject women.
- Common data quality issues: Missing values, duplicates, inconsistent formatting, and outliers are the big four. You should be able to identify them in a description of a dataset.
- Data preparation steps: Ordering matters. You need to know that cleaning generally comes before feature engineering, and that splitting into train/validation/test sets happens at the end.
- The impact on model accuracy: Clean data improves accuracy, but more importantly, clean data makes the model's behaviour more predictable and reliable.

Common traps the exam sets:

- They give a scenario where the data is clean but the model is still inaccurate. The correct answer is probably that the data lacks relevant features (i.e., the right information is missing from the start). Cleaning cannot fix a lack of useful data.
- They present an example of "noise" in the data (random errors) and list causes. The correct answer is often "data entry errors" or "sensor malfunctions", not "model architecture problems".
- They might ask about handling missing data. The trap answer is "delete all rows with missing data" — while sometimes correct, the better general answer is "investigate why data is missing first, then decide whether to delete or impute (fill in) values".
- They test the difference between structured data (tables with rows and columns) and unstructured data (text, images, audio). Most real-world AI uses structured, cleaned data for tabular models, but the exam may ask how to prepare unstructured data (e.g., text must be tokenised into numbers).

Definitions you must memorise:

- Data profiling: The first step to understand your dataset's structure and quality issues.
- Feature engineering: Creating new input columns from raw data to improve model learning.
- Training set: The data used to teach the model patterns.
- Test set: Data kept separate to evaluate the final model's performance.
- Imputation: Filling in missing values using statistical methods (mean, median, or a model's prediction).
- Normalisation: Scaling numeric values to a standard range (e.g., 0 to 1) so that one feature (like salary in thousands) does not dominate another (like age in single digits).

Study tip: When you see a question about "what is the most important step before training an AI model?", the correct answer is almost always "clean and prepare the data".

## Step by step

1. **Data Collection and Profiling** — Gather all raw data from different sources (databases, files, spreadsheets, APIs). Run a profiling tool to identify missing values, duplicates, inconsistent formats, and outliers. This is your first health check — you decide how bad the problem is before starting any cleaning.
2. **Standardisation** — Convert all data into a consistent format. For example, make sure all dates are in YYYY-MM-DD, all phone numbers follow the same pattern, and text fields use the same capitalisation. This prevents the AI from learning that 'New York' and 'new york' are different cities.
3. **Handling Missing Values and Duplicates** — Decide on a strategy for every missing value: either delete the row, fill with the mean/median, fill with a placeholder, or use a model to predict the missing value. Remove exact duplicate rows (same values in every column) and investigate near-duplicates (e.g., same name with a typo) to merge or drop them.
4. **Feature Engineering and Transformation** — Create new input columns (features) that might help the model, such as 'days since last purchase' from a date column. Convert categorical data (like country or product type) into numbers using methods like one-hot encoding. Scale numeric features to a comparable range (normalisation) so one feature does not dominate others.
5. **Splitting into Training, Validation, and Test Sets** — Randomly divide the final clean dataset into three portions: typically 70% training, 15% validation, and 15% test. Use only the training set to teach the model. Use the validation set to tune settings. Use the test set only once, at the very end, to measure how well the model generalises to unseen data. Never let the model see the test set during training.

## Comparisons

### Data Cleaning vs Data Preparation

**Data Cleaning:**
- Focuses on fixing errors: removing duplicates, correcting typos, handling missing values
- Is a subset of data preparation
- Does not create new features or transform data types

**Data Preparation:**
- Encompasses cleaning plus transformation steps like feature engineering and normalisation
- Is the broader process of making data ready for an AI model
- Includes splitting data into train/validation/test sets

### Training Data vs Test Data

**Training Data:**
- Used to teach the model patterns
- The model sees this data during training
- Largest portion of the dataset (typically 70%)

**Test Data:**
- Used to evaluate final model performance on unseen data
- The model never sees this data during training
- Smallest portion of the dataset (typically 15%)

### Imputation (filling missing values) vs Deletion (removing rows with missing values)

**Imputation (filling missing values):**
- Preserves data size and can reduce bias
- Uses statistical methods like mean or median to fill gaps
- May introduce artificial values that do not reflect reality

**Deletion (removing rows with missing values):**
- Simple and avoids introducing artificial data
- Can lose valuable information and introduce bias
- Removes entire rows, reducing dataset size

### Structured Data vs Unstructured Data

**Structured Data:**
- Organised in rows and columns like a spreadsheet
- Easy for tabular AI models to process directly
- Examples: customer records, sales transactions

**Unstructured Data:**
- No predefined format: text, images, audio, video
- Requires extensive preprocessing (e.g., text tokenisation, image resizing)
- Examples: support chat logs, photos, recorded calls

## Diagram

_This flowchart shows the data preparation pipeline from raw sources through profiling, cleaning, feature engineering, and splitting into training, validation, and test sets._

```mermaid
flowchart TD
    A[Raw Data Sources] --> B[Data Profiling]
    B --> C{Issues Found?}
    C -->|Yes| D[Data Cleaning]
    C -->|No| E[Move to Preparation]
    D --> F[Standardisation & Deduplication]
    F --> G[Feature Engineering]
    G --> H[Normalisation]
    H --> I[Split Dataset]
    I --> J[Training Set 70%]
    I --> K[Validation Set 15%]
    I --> L[Test Set 15%]
    J --> M[Train AI Model]
    K --> N[Tune Model]
    L --> O[Evaluate Final Model]
```

## Common misconceptions

- **Misconception:** More data always makes an AI model more accurate, even if the data is messy. **Reality:** More low-quality data amplifies errors. A small, clean, relevant dataset is far better than a huge, messy one. (People are used to the idea that 'more information is better' from everyday life, not realising that AI amplifies patterns — including bad ones.)
- **Misconception:** Data cleaning is a one-time task you do at the start of a project, then ignore. **Reality:** Data cleaning and preparation is an ongoing process. New data arrives with new errors, business rules change, and models need retraining on fresh clean data regularly. (Beginners think of software projects as 'build once, deploy forever', but AI is a living system that needs continuous data care.)
- **Misconception:** If I remove all the bad rows, my data is clean enough. **Reality:** Removing rows can introduce bias. For example, if you delete all rows with missing income data, you might remove low-income individuals who are less likely to report income, making your model biased toward wealthier people. (It feels like the safest, most objective action, so beginners default to it without thinking about the consequences to fairness.)
- **Misconception:** Data quality only matters for the AI model's final predictions — training is the important part. **Reality:** Data quality affects every single step: training accuracy, validation honesty, and final deployment reliability. Bad data can also cause the model to fail silently, making confident but wrong predictions. (People naturally focus on the exciting 'AI magic' (training) and ignore the boring prep work.)
- **Misconception:** An AI model can fix messy data on its own once you start training. **Reality:** AI models learn the patterns you give them. They have no ability to detect or fix data errors unless specifically programmed to do so (e.g., with outlier detection features). By default, they treat errors as valid patterns. (This misconception comes from sci-fi ideas of AI as all-powerful and self-correcting, when in reality it is a pattern-matcher with no common sense.)

## Key takeaways

- Garbage in, garbage out means that an AI model is only as good as the data you train it on.
- Data cleaning removes errors, duplicates, and inconsistencies, but it cannot fix fundamentally biased or irrelevant data.
- Feature engineering transforms raw data (like text dates) into numeric representations that AI models can process.
- Always split your cleaned dataset into training, validation, and test sets before training to get an honest measure of performance.
- Missing data must be investigated first — do not automatically delete rows without understanding why data is missing.
- Data profiling is the first step in any AI project; you cannot fix what you have not measured.
- Bias in training data leads to unfair and inaccurate model predictions that can cause real-world harm.
- Normalisation ensures that features with larger numeric ranges do not unfairly dominate the model's learning.

## FAQ

**What does 'garbage in, garbage out' mean in AI?**

It means that if you train an AI model on bad data (incomplete, inaccurate, or biased), the model's outputs will also be bad, regardless of how sophisticated the model is.

**Do I need to be able to write code to clean data for the AI Associate exam?**

No. The exam tests your understanding of the concepts and processes of data cleaning and preparation. You will not be asked to write or read code.

**What is the difference between data cleaning and data preparation?**

Data cleaning is a subset of data preparation. Cleaning fixes errors and removes duplicates. Preparation is broader and includes cleaning plus steps like feature engineering, normalisation, and splitting data into training and test sets.

**How much of my data should be used for training vs testing?**

A common split is 70% for training, 15% for validation, and 15% for testing. The exact percentages can vary, but you always need separate sets to avoid overfitting (where the model memorises the training data instead of learning general patterns).

**Can I delete all rows with missing values?**

You can, but it is often a bad idea. Deleting rows can remove valuable information and introduce bias. It is better to investigate why the data is missing and use a strategy like imputation (filling with an average or a predicted value) where appropriate.

**What is feature engineering and why does it matter?**

Feature engineering is the process of creating new input variables from raw data to help the AI model learn better patterns. For example, extracting the day of the week from a date column is feature engineering. It matters because good features can drastically improve model accuracy without changing the model itself.

## Check your understanding

1. **What does 'GIGO' stand for and why is it relevant to AI?**

   Answer: GIGO stands for 'garbage in, garbage out', meaning that an AI model trained on poor quality data will produce poor quality predictions.

2. **What is the name of the process that converts categories like 'Red', 'Blue', 'Green' into numeric format for an AI model?**

   Answer: That process is called one-hot encoding, where each category becomes a new column with a 1 or 0 value.

3. **What is the primary purpose of a validation set in the data splitting process?**

   Answer: The validation set is used to tune the model's settings (hyperparameters) and choose the best version of the model without touching the final test set.

4. **If your dataset contains many duplicate customer records, what should you do before training an AI model?**

   Answer: You should merge or remove the duplicate records so that the model does not give double weight to those customers and learn a skewed pattern.

5. **Why is it important to investigate missing data before deciding how to handle it?**

   Answer: Different types of missing data (random vs. systematic) require different handling; blindly deleting rows can introduce bias, and random imputation might not work for non-random missing patterns.

---

Interactive version with quiz and diagrams: https://courseiva.com/learn/salesforce-ai-associate/data-quality-and-preparation