Exploratory Data Analysis and Visualization

Exploratory data analysis – often shortened to EDA – is the process of looking at raw data before you build any machine learning model. Think of it as meeting your date before agreeing to marry them. You need to know what you are working with. EDA has three big goals: find patterns, spot anomalies, and understand relationships between different pieces of data.

Let us break down the jargon. Data is just information – numbers, text, dates, images, anything you can collect. A dataset is a collection of that data, usually organised into rows and columns like an Excel spreadsheet. Each column is called a feature (sometimes a variable). Each row is a single observation or data point. For example, if you have a dataset of houses, the features might be "number of bedrooms", "square footage", and "price". Each row is one specific house with its specific values for those features.

When you start EDA, the first thing you do is check the shape of your data. How many rows and columns? Are there missing values – cells that are empty because the data was not recorded? Missing data is common and can break your model if you ignore it. You also look for duplicates – rows that are exactly the same as another row, which can accidentally make the model think certain patterns are more common than they really are.

Next, you examine the data types. Is a column full of numbers (integers or floats), text (strings), dates, or categories (like "red", "blue", "green")? The machine learning algorithm needs to know this because it cannot work with text directly. You will often need to convert categories into numbers using techniques like one-hot encoding, where each unique category gets its own column with a 1 or 0.

After that, you look at distributions. A distribution shows how often each value appears. For example, if you are looking at ages in a dataset, a distribution chart will show you if most people are in their 30s or if the ages are spread out evenly. This matters because a skewed distribution – where values cluster at one end – can mislead your model. A histogram is the common chart for this. It divides the range of values into bins (like 0-10, 10-20, 20-30) and counts how many data points fall into each bin.

Now, anomalies. These are data points that do not fit the normal pattern. A house that costs one dollar is likely an anomaly (maybe a typo). An age of 200 years is impossible. Anomalies can also be subtle – a sudden spike in website traffic at 3am might be a bot attack, or it might be a data collection error. Spotting these early saves you from training a model on nonsense.

Relationships between features are where the real value lies. You want to know: if the number of bedrooms increases, does the price usually increase too? That is called a correlation. If two features move in the same direction (both go up together), they are positively correlated. If one goes up while the other goes down, they are negatively correlated. A scatter plot is the classic visualisation for this – you put one feature on the x-axis (horizontal) and another on the y-axis (vertical), and each data point is a dot on the chart. If the dots form a rough line, there is a correlation. If they are scattered like confetti, there is no strong relationship.

You also check for multicollinearity – when two features are so strongly correlated that they essentially tell you the same thing. For example, "total square footage" and "number of rooms" might be highly correlated because bigger houses usually have more rooms. Some machine learning models get confused by this, so you might need to remove one of the two features.

Visualisation is the tool that makes EDA practical. You use charts and graphs to see patterns that are invisible in a table of numbers. Common types include:

Why does EDA exist? Before machine learning became mainstream, data analysts would manually look at spreadsheets and draw graphs by hand (or with early software like Excel). The process was slow and limited to small datasets. Today, with massive datasets containing millions of rows, you need automated tools and programming libraries like Python's pandas, matplotlib, and seaborn to explore data efficiently. EDA replaces guesswork with evidence. Instead of assuming what the data looks like, you prove it to yourself visually and numerically.

What does EDA replace? It replaces the dangerous practice of feeding data directly into a model without understanding it. That is called "garbage in, garbage out" – if you put bad data in, you will get bad predictions out. EDA is the quality check that prevents garbage from entering your machine learning pipeline.

Imagine you work as a data scientist at an online clothing retailer called StyleShip. The company wants to build a machine learning model that predicts whether a customer will return a purchased item within 30 days. This is a binary classification problem – the outcome is either "return" or "keep". The business goal is to reduce return rates by identifying high-risk purchases early.

You are given a dataset with 500,000 rows, each representing a single purchase. The features include: customer age, purchase amount, item category (dress, shoes, accessories), day of the week, whether the customer used a discount code, the customer's previous return rate, and the item colour. Your first move is to open the dataset and start EDA.

You begin by checking the shape – 500,000 rows and 15 columns. Good, you have plenty of data. Next, you check for missing values. You find that 12% of rows have a missing value for "customer age". That is a significant chunk. You need to decide whether to drop those rows, fill them with an average age, or use a more sophisticated imputation method (guessing the missing value from other features). You also find 200 duplicate rows – identical purchases recorded twice. You delete those.

Now you look at data types. The "item category" is text – "dress", "shoes", "accessories". You convert it to numbers using one-hot encoding, creating three new columns: is_dress, is_shoes, is_accessories, each containing 0 or 1. The "purchase date" is stored as a date string. You extract the day of the week from it and make that a separate numeric feature (Monday=0, Sunday=6).

You then generate visualisations. You make a histogram of purchase amounts. It shows a right-skewed distribution – most purchases are between £20 and £80, but there is a long tail of very expensive items up to £2,000. Those expensive items are rare but could be important for returns. You make a box plot of customer age. It shows the median age is 34, with most customers between 22 and 52. But there are a few ages listed as 0 and one as 150 – clearly data entry errors. You remove those anomalous rows.

Next, you explore relationships. You make a scatter plot of purchase amount versus previous return rate. You see a faint positive correlation – customers who previously returned more items tend to purchase slightly more expensive items. That is interesting but weak. You then create a heatmap of correlations between all numeric features. You notice a strong correlation between "purchase amount" and "used discount code" – customers who use discount codes tend to spend less. That makes sense.

You also look for patterns by category. You group the data by item category and compute the return rate for each. You find that shoes are returned 25% of the time, dresses 18%, and accessories only 8%. That is a crucial insight – the model should give extra weight to shoe purchases. You create a bar chart to present this to your team because visuals are easier to understand than tables.

Finally, you identify a subtle anomaly. The dataset shows that purchases made on Sundays have a return rate of 35% compared to 15% on other days. At first you think it is a pattern, but then you dig deeper. You discover that Sunday purchases are a very small sample – only 200 rows out of 500,000 – and those were all from a single promotional event for high-end shoes. The high return rate is not a day-of-week effect; it is an artifact of that specific promotion. Without EDA, you would have built a model that over-predicted returns for all Sunday purchases, which would be wrong.

All of these discoveries happened before you trained a single machine learning model. They directly influenced how you cleaned the data, which features you used, and which patterns the model should focus on. That is the real-world power of EDA.