How do you get the right data into your machine learning model in a way that it can actually understand? That's the fundamental problem that data engineering solves: it's the process of finding, collecting, and moving data from where it lives to where your model can use it. For the MLS-C01 exam, mastering this is critical because before you can build a clever algorithm, you need to know the difference between a database, a data lake, and a streaming service, and how to pick the right tool for the job.
Jump to a section
A simple way to picture Data Engineering Foundations for Machine Learning
Have you ever tried to cook a complex meal for guests without prepping the ingredients first? You'd be chopping onions while the sauce burns, searching for the paprika while the meat overcooks. It's chaos. That's what building a machine learning model without data engineering is like. Your data is the raw ingredients. Data engineering is the meal prep. Before you even think about turning on the stove (training your model), you need to decide what to cook (the business problem), go to the shops (identify data sources), and bring home the groceries (ingest the data). You wouldn't just grab a random mix of items off the shelf; you'd choose specific vegetables, meats, and spices based on your recipe. Some ingredients come fresh from the farmers' market (real-time streaming data), some come from the tinned goods aisle in the cupboard (batch data from a database), and some might be a pre-made sauce from a specialty store (data from a third-party API). Then, you need to wash, chop, and organise everything on your counter (data cleansing and structuring) so that when you start cooking, you can just grab what you need. Without this prep, your dinner party (machine learning project) will be a stressful disaster, and your guests (the model's users) will leave hungry.
Data engineering for machine learning is the entire process of getting data ready for analysis. It's the boring but absolutely essential plumbing that makes all the fancy AI possible. Think of it as building a reliable water supply system for a city: you need to find the water source (identify data sources), build pipes to bring the water to the treatment plant (data ingestion), and clean the water so it's safe to drink (data preparation).
Let's break down the key parts. First, you have to identify data sources. This means figuring out where your data actually lives. In a company, data can be scattered everywhere. Here are the most common types you will see on the MLS-C01 exam: - Databases: These are like organised filing cabinets. A relational database (like Amazon RDS or Aurora) stores data in tables with rows and columns, like a spreadsheet. A NoSQL database (like Amazon DynamoDB) is more flexible and can store things like user profiles or shopping cart contents without a rigid structure. - Data Lakes: This is a giant, centralised repository that holds all your data in its raw, original format. Think of it as a massive warehouse where you dump everything—structured data (like a CSV file), semi-structured data (like JSON logs from a web server), and unstructured data (like images or video). Amazon S3 is the most common data lake on AWS. - Data Warehouses: These are like a high-end, curated library. The data is cleaned, structured, and optimised for analysis and reporting. Amazon Redshift is a popular data warehouse. The key difference from a data lake is that a warehouse contains processed, ready-to-query data, while a lake holds the untouched original. - Streaming Data: This is data that is generated continuously, like a never-ending river. Examples include clickstream data from a website, sensor readings from an IoT device, or financial market tickers. AWS services like Amazon Kinesis and Amazon Managed Streaming for Apache Kafka (Amazon MSK) are used to capture and process this real-time data. - Third-Party APIs (Application Programming Interfaces): Sometimes you need data that your company doesn't own, like weather data or social media sentiment. An API is a set of rules that allows different software programs to talk to each other. You can write code to call an API and pull data into your system.
Once you know where your data is, you need to ingest it. This is the process of moving the data from its source to a storage or processing system (usually an S3 data lake). There are two main patterns: 1. Batch Ingestion: Data is collected and transferred in large, scheduled chunks. For example, you might run a job every night at 2 AM that copies all the day's sales transactions from your database to S3. This is simpler, cheaper, and suitable for non-time-sensitive tasks. 2. Streaming Ingestion: Data is ingested continuously as it is created, with minimal delay. For a ride-sharing app, you need to stream the locations of all available cars in real-time to match them with customers. This is more complex and expensive but necessary for real-time applications.
AWS provides specific services for each of these tasks. For batch ingestion, you might use AWS Glue (a serverless ETL service) or AWS Data Pipeline. For streaming, you would use Amazon Kinesis or Amazon MSK. The exam loves to test which service is appropriate for which scenario.
Why does all this matter? Because a machine learning model is only as good as the data it's trained on. If you feed it messy, incomplete, or incorrectly sourced data, you will get a useless model (a concept known as 'garbage in, garbage out'). Data engineering provides the foundation: it ensures you have the right data, in the right place, at the right time, and in the right format.
Identify Business Problem and Required Data
Before any technical work, you must understand what your ML model needs to predict (e.g., customer churn). This defines what data you need: historical customer behaviour, transaction logs, support tickets. You cannot build a pipeline for 'all data'.
Catalogue Available Data Sources
Survey the organisation to find where the needed data resides. This could be relational databases (RDS), log files on S3, streaming event streams (Kinesis), or external APIs. Document the format, size, and update frequency of each source.
Design the Ingestion Strategy (Batch vs. Stream)
Decide on the ingestion pattern. For a daily sales forecast, batch is appropriate. For a fraud detection system, you need streaming. This decision drives which AWS service you select (Glue vs. Kinesis).
Build and Configure the Ingestion Pipeline
Use the chosen AWS services to pipeline the data. For batch, you might create an AWS Glue job that reads from a database and writes Parquet files to S3. For streaming, you set up a Kinesis Data Stream to capture events from an application.
Implement Security and Monitoring
Apply IAM roles to control access to the pipeline. Enable encryption (using AWS KMS) for data at rest in S3 and in transit. Set up CloudWatch alarms to alert on failures (e.g., a failed Glue job or a Kinesis stream lagging).
Validate and Monitor the Pipeline
After deployment, run the pipeline and check the output data for correctness. Monitor for schema changes in the source data. Schedule regular reconciliations to ensure all expected data was ingested. This step is often overlooked but is critical for reliability.
Let's walk through a realistic example: 'ShopStream', a UK-based online clothing retailer, wants to build a machine learning model to predict next week's demand for each product so they can manage their inventory better.
Here is what the IT professional (a data engineer) actually does, step by step:
First, they must identify all the data sources that are relevant. This is never as simple as 'it's in the database'. For ShopStream, the data is spread across several systems:
The main relational database (Amazon RDS for PostgreSQL) holds historical sales transactions, customer IDs, and product details.
The website generates a firehose of event data (clicks, page views, add-to-cart actions) that is currently dumped into Amazon S3 as JSON log files.
Marketing uses a third-party tool (e.g., Google Analytics) that has aggregated traffic data accessible via an API.
The warehouse management system has a separate database with stock levels and supplier lead times.
The data engineer must catalogue all these sources. They will probably use AWS Glue to crawl these sources and create a data catalog—a central metadata repository that describes what data exists and where it is.
Next, they need to design the ingestion pipeline. Because demand forecasting is not a real-time problem (they don't need to know about a sale that happened 5 seconds ago), batch processing is appropriate. They decide to run a nightly batch job. Here is the action plan:
For the relational database, they will use AWS Database Migration Service (DMS) to continuously replicate changes into S3, or they will run a regular SQL query dump.
For the S3 click logs, they will use AWS Glue ETL jobs to transform the raw JSON into a structured Parquet format and join it with the sales data.
For the Google Analytics data, they will write a Python script that calls the API every night and writes the result to S3. This might run on AWS Lambda (serverless compute) or a scheduled Amazon EC2 instance.
All of this data will land in a staging area in S3, organised by date and source (e.g., s3://shopstream-data-raw/sales/2024/05/21/).
Finally, they must handle edge cases. What if the API call fails? They set up a retry mechanism and an alert. What if the database schema changes? They configure AWS Glue to automatically detect schema evolution. What if there is a huge spike in traffic one day? They ensure their ingestion pipeline can auto-scale using services like AWS Glue's auto-scaling or Amazon Kinesis for the streaming path if they decide to add real-time features later.
The end result is a clean, reliable pipeline that delivers fresh, structured data into an S3 data lake every single day. Only now can the data scientist on the team begin building the actual demand forecast model.
The MLS-C01 exam tests your understanding of data engineering in a very specific way. It is not about memorising AWS service features line-by-line; it is about scenario-based selection. You will be given a business problem and asked to choose the best combination of services to source and ingest data.
Here's what you need to focus on for the exam: - Differentiating Batch vs. Streaming (Real-Time): This is the most common trap. The question will describe a scenario with time sensitivity. If it says 'every hour', 'daily', or 'nightly', it is batch. If it says 'sub-second latency', 'real-time', or 'immediate analysis', it is streaming. The exam will try to trick you by mentioning 'Amazon Kinesis' in a clearly batch scenario—the correct answer will be something like 'AWS Glue' or 'AWS Data Pipeline'. - Knowing the Core Ingestion Services: You need to know the specific purpose of these services: - Amazon Kinesis: For streaming data (Data Streams for real-time, Data Firehose for near-real-time loading into S3/Redshift). - Amazon MSK: For streaming data if the company already uses or requires Apache Kafka. - AWS Glue: For batch ETL (Extract, Transform, Load). It is serverless and has a built-in Data Catalog. - AWS Data Pipeline: An older batch orchestration service. Often the correct answer when complex workflows or on-premises data is involved. - AWS DMS: Specifically for migrating databases to AWS. If the question says 'migrate an on-premises Oracle database', DMS is the answer. - Traps around Cost and Scale: The exam loves to test your ability to choose the cheapest viable option. For example, for a small, daily data load from a single database, a simple scheduled AWS Lambda function writing to S3 might be the best answer, not a complex AWS Glue setup. But if the data volume is massive, AWS Glue's framework would be more appropriate. - Security and Governance: Questions may ask about encrypting data in transit or at rest. The default correct answer often involves using AWS KMS (Key Management Service) for encryption and IAM roles for permissions. Never share access keys. - Schema on Read vs. Schema on Write: The exam tests this subtle but important concept. A data lake (S3) uses 'schema on read'—you define the structure only when you read the data. A data warehouse (Redshift) uses 'schema on write'—the structure must be defined before you load the data. A question might describe a scenario where data formats change frequently; a data lake would be the better choice.
In summary, when you see a question on data sources and ingestion, immediately identify the time constraint (batch vs. stream) and the data source type (database, API, on-premises). Then, eliminate services that don't match that pattern. The correct answer will almost always be the simplest service that meets the specific latency and volume requirements.
Data engineering is the foundation of any machine learning project: garbage data in equals garbage predictions out.
Identify the data source type (database, data lake, streaming, API) before choosing an ingestion method.
Batch ingestion (AWS Glue, Data Pipeline) is for scheduled, non-time-sensitive data; streaming ingestion (Kinesis, MSK) is for real-time data.
A data lake (Amazon S3) stores raw data with no predefined schema; a data warehouse (Amazon Redshift) stores cleaned data with a rigid schema.
Amazon Kinesis is the core AWS service for streaming data ingestion, with Data Streams for real-time and Data Firehose for near-real-time loading.
AWS Glue provides both a serverless ETL engine and a data catalog to manage metadata across various sources.
These come up on the exam all the time. Here's how to tell them apart.
Data Lake (Amazon S3)
Stores raw data in its native format (CSV, JSON, Parquet).
Uses 'schema-on-read': structure is defined when you query the data.
Ideal for exploratory analysis, data science, and storing large volumes of cheap storage.
Data Warehouse (Amazon Redshift)
Stores cleaned, transformed, and structured data.
Uses 'schema-on-write': structure must be defined before loading data.
Ideal for business intelligence, reporting, and fast SQL queries on curated data.
Batch Ingestion (AWS Glue)
Processes data in large, scheduled chunks (e.g., nightly).
Best for non-time-sensitive tasks like daily sales reports.
Typically cheaper and simpler to manage.
Streaming Ingestion (Amazon Kinesis)
Processes data continuously as it is generated.
Best for real-time use cases like fraud detection or live dashboards.
More complex and expensive but provides low-latency insights.
Relational Database (Amazon RDS)
Stores data in rigid tables with rows and columns and pre-defined schemas.
Best for transactions and complex queries (JOINS, aggregations).
Requires schema changes to be managed and deployed explicitly.
NoSQL Database (Amazon DynamoDB)
Stores data as key-value pairs or documents with flexible schemas.
Best for high-speed, high-volume read/write operations at scale.
Allows schema evolution automatically; you can add new attributes without downtime.
Mistake
A data lake is the same as a data warehouse.
Correct
A data lake stores raw, unprocessed data in its original format, while a data warehouse stores cleaned, transformed, and structured data optimised for querying and reporting.
The terms are often used interchangeably in casual conversation, leading beginners to think they are synonyms. The exam explicitly tests the difference regarding schema flexibility and data purpose.
Mistake
All data ingestion needs to happen in real-time using a streaming service like Amazon Kinesis.
Correct
Most business use cases, especially for machine learning training, only require batch ingestion (daily or hourly). Streaming is only needed for real-time analytics or low-latency predictions.
Beginners are excited by 'real-time' and over-apply it. The exam punishes you for choosing an expensive, complex streaming solution when a simple batch job will do.
Mistake
AWS Glue is the only service for batch data ingestion.
Correct
AWS Glue is one option, but there are many others including AWS Data Pipeline, AWS DMS, AWS Lambda, and even simple shell scripts run on an EC2 instance. The 'best' choice depends on scale, complexity, and cost.
AWS markets Glue heavily, so beginners assume it is always the right tool. The exam tests your ability to pick the most appropriate and cost-effective service for the specific scenario.
Mistake
Once data is in the data lake (S3), it is clean and ready to use.
Correct
Data in a data lake is typically raw and 'dirty' (duplicates, missing values, inconsistent formats). It must be processed and transformed (ETL) before being used for machine learning.
The name 'lake' implies a pristine body of water. In reality, an enterprise data lake is often a 'data swamp' full of messy, unprocessed data. This misconception leads to expecting clean data from a raw source.
Mistake
An API is a database.
Correct
An API is an interface that allows two applications to communicate. The data behind the API might come from a database, but the API itself is not the storage system. You write code to call it.
Beginners hear 'get data from an API' and confuse the method of retrieval with the storage location. The exam tests this by asking how to integrate an external data source.
Reveal each answer, then mark whether you got it right. Score 60%+ to unlock the next chapter.
Kinesis Data Streams is for real-time, custom processing where you write consumers to read and process data as it arrives. Kinesis Data Firehose is a managed service that automatically loads streaming data into destinations like S3, Redshift, or Elasticsearch with no coding required.
Use AWS Glue if you need a serverless, scalable solution with automatic schema discovery and integration with the Glue Data Catalog. Use a simple Python script on EC2 or Lambda if your ETL is small, simple, and you want to avoid the overhead of managing Glue's Spark environment.
A data lake is a central repository (usually Amazon S3) that stores all your raw data in its native format. It's important for ML because data scientists need access to the raw, unprocessed data to explore and experiment without constraints from a rigid data warehouse schema.
Use AWS Database Migration Service (DMS) for a one-time migration or continuous replication. Alternatively, export the data to a CSV or Parquet file, upload it to S3, and then use AWS Glue or Athena to process it.
No. Streaming is more complex and expensive. Use batch when the business requirement does not need real-time analysis (e.g., daily reports). Use streaming only when you need sub-second or near-real-time insights (e.g., fraud detection, live dashboards).
The Glue Data Catalog is a central metadata repository that stores information about your data sources, such as table schemas, locations, and formats. It allows services like Athena, Redshift Spectrum, and EMR to easily discover and query your data without manual schema definitions.
You've finished Data Engineering Foundations for Machine Learning. Continue through the MLS-C01 study guide to build a complete picture of the exam.
Done with this chapter?