AWS Data Stores and Ingestion Pipelines

Before we dive into the specific services, let us understand the big picture. In any machine learning project, you start with data—lots of it. This data comes from many places: website clicks, sensor readings, financial transactions, or social media feeds. But before you can train a model, you need to store that data somewhere secure, scalable (meaning it can grow as you get more data), and cost-effective. That is where AWS Data Stores come in. AWS offers several types of data stores, each designed for a specific kind of data and access pattern. The three main categories are object storage, block storage, and database storage. Let us break each one down.

Object storage is best exemplified by Amazon Simple Storage Service (S3). Think of S3 as an infinite filing cabinet. You put files (called objects) into folders (called buckets). Each object can be anything—a CSV file, an image, a video, a log file—and it can be up to 5 terabytes in size. S3 is designed to hold massive amounts of data, petabytes (one petabyte is one million gigabytes) and beyond. It is also highly durable, meaning AWS guarantees that your data will not be lost. S3 replicates your objects across multiple devices in at least three different physical locations within the same region (a region is a specific geographic area, like US East (N. Virginia) or Europe (Ireland)). This is called 99.999999999% durability. For machine learning, S3 is the most common place to store training datasets, because you can access the data from any machine learning service like Amazon SageMaker without needing to set up complex connections. S3 has different storage classes. Standard is for frequently accessed data. Standard-Infrequent Access (S3 Standard-IA) costs less per gigabyte but charges a fee if you retrieve the data, so it is good for data you access only a few times a month. Glacier and Glacier Deep Archive are for long-term archival, where you might only retrieve data once a year or less. These have lower storage costs but higher retrieval costs and longer retrieval times (minutes to hours).

Block storage is different. This is for data that needs to be accessed at the level of individual bytes, like the operating system of a virtual machine. The main AWS block store is Amazon Elastic Block Store (EBS). EBS volumes are like virtual hard drives that you attach to an Amazon Elastic Compute Cloud (EC2) instance. They are very fast and offer consistent, low-latency performance. For machine learning, you might use EBS to store intermediate outputs of model training or to hold a local copy of a dataset that your code reads repeatedly. However, EBS volumes are tied to a specific EC2 instance in a particular Availability Zone (a data centre location), so they are less flexible for sharing data across many different services compared to S3.

Database stores are optimised for querying—asking questions about the data and getting answers quickly. AWS offers many database types: relational databases like Amazon RDS (for structured data with relationships, such as customer orders and products), NoSQL databases like Amazon DynamoDB (for key-value or document data that needs high-speed reads and writes without a fixed schema), and graph databases like Amazon Neptune (for data where relationships between entities are important, such as social networks or recommendation engines). For ML workflows, you often pull data from databases to create training datasets, or you store model predictions back into databases for real-time applications.

Now, how does data actually get from its source into these data stores? That is where ingestion pipelines come in. Ingestion means bringing data in. A pipeline is a series of steps that transforms and moves the data. AWS provides several managed services to build these pipelines so that you do not have to write complex code from scratch.

Amazon Kinesis Data Firehose is one of the easiest ways to load streaming data (data that arrives continuously, like Twitter feeds or stock prices) into S3, Redshift, or Elasticsearch. It can also transform the data on the fly using AWS Lambda (a serverless compute service that runs small pieces of code). For example, you might use Firehose to take raw JSON log data from a web app, convert it to Parquet format (a columnar format that is efficient for queries), and then save it to S3 every 60 seconds or every 1 MB of data.

AWS Glue is a serverless data integration service that can crawl your data sources (understand their schema—the structure of the data), clean and transform the data, and then load it into a target data store. Glue is especially useful for extract, transform, load (ETL) jobs in machine learning. Suppose you have customer data in DynamoDB and order data in RDS. Glue can join these datasets, handle missing values, and output a single CSV file ready for model training in S3.

Amazon Data Firehose (previously called Kinesis Data Firehose) is now integrated with Amazon S3, Amazon Redshift, and Amazon OpenSearch Service. It does not store data permanently; it delivers it to destinations you specify. For example, if you are monitoring website traffic, Firehose can take every click event, convert it to JSON, and send it to an S3 bucket for later analysis.

Amazon Kinesis Data Streams is a more low-level service for building custom streaming applications. It stores data for up to 365 days and allows multiple applications to consume the same stream simultaneously. You would use Kinesis Data Streams if you need real-time processing with custom code, like detecting fraud as transactions happen.

AWS Database Migration Service (DMS) is used to move data from one database to another. For example, if you have an on-premises Oracle database and you want to copy its data to Amazon RDS for MySQL so you can use it for ML, DMS can do that with minimal downtime.

The choice of ingestion pipeline depends on factors: is the data arriving in real-time or in batches (at scheduled intervals)? How much data is there? Does the data need transformation before it lands in the store? For the MLS-C01 exam, remember that for real-time streams, think Kinesis or Firehose. For scheduled batch ETL, think Glue or AWS Step Functions (a service to coordinate multiple AWS services). For one-time data migration, think DMS.

Finally, all these storage and pipeline services work together in a typical ML workflow. The data resides in operational databases (RDS, DynamoDB) or object storage (S3). Glue or Firehose moves it to a data lake (usually S3) where it is cleaned and prepared. Then SageMaker reads the data from S3 to train a model. The model is stored back in S3 as a model artifact, and predictions can be written back to DynamoDB or RDS for use by applications. Understanding which service fits which role is exactly what the exam tests.

Let us imagine you work for a retail company called ShopFast, and you are building a machine learning system to predict customer churn (which customers are likely to stop buying). The company captures data from multiple sources: a PostgreSQL database containing customer profiles and purchase history, clickstream logs from the website, and customer support ticket text. Your task is to design the data storage and ingestion pipeline so that data scientists can train a churn prediction model using Amazon SageMaker.

Step 1: Identify the source systems. The PostgreSQL database is running on-premises (in the company's own data centre). The clickstream logs are generated every second from the web server, and they are currently saved as plain text files on an FTP server. The support ticket texts are stored in a MongoDB database (a NoSQL document database). You need to bring all of these into AWS.

Step 2: Choose the appropriate AWS Data Stores. Because the final dataset needs to be a single, clean, tabular format with features like 'total purchases last 30 days', 'days since last login', 'support ticket volume', you decide to land all raw data in Amazon S3. S3 is cheap, durable, and easily accessible by SageMaker. You create an S3 bucket called 'shopfast-ml-raw-data' with three prefixes (folders): /customer_db, /clickstream, /support_tickets. For the PostgreSQL data, you use AWS Database Migration Service (DMS) to do an initial full copy, then set up ongoing replication so that changes are continuously streamed to S3 as CSV files. For the clickstream logs, you decide to use Amazon Kinesis Data Firehose. You configure a Firehose delivery stream that reads the log data from a custom HTTP endpoint (your web server pushes logs there), then uses a Lambda function to convert the raw text into JSON and filter out irrelevant entries (like page load times). Firehose then delivers the processed data to the S3 bucket every 5 minutes, organised by date partitions (e.g., /clickstream/year=2025/month=04/day=10). For the MongoDB support tickets, you write a small Python script that runs on an EC2 instance once a day to extract new tickets from the last 24 hours and writes them as JSON files to S3 using the AWS SDK.

Step 3: Build the ETL pipeline to transform raw data into a training dataset. You set up an AWS Glue job that runs nightly. The Glue job first crawls the S3 raw data to automatically recognise the schemas of the CSV and JSON files. Then it performs transformations: it joins the customer data with the clickstream data by customer ID, computes aggregate features like ‘number of logins in the last week’, and joins with support ticket counts. It also handles missing values (e.g., if a customer has no clickstream data, it fills zero). The output is a Parquet file written to a separate S3 bucket called 'shopfast-ml-training-data'. Parquet is used because it compresses well and is efficient for querying large datasets.

Step 4: Use the prepared data for training. The data scientists use Amazon SageMaker. They point a processing job to the S3 location 'shopfast-ml-training-data' and run a script that splits the data into training (80%) and testing (20%) sets. The model is trained using SageMaker's built-in XGBoost algorithm (a popular gradient boosting algorithm for classification). The trained model artifact is stored back in S3 under a 'models' prefix.

Step 5: Serve predictions. Once the model is deployed as an endpoint in SageMaker, a custom application for customer service agents queries the endpoint via an API. The input data (customer features) comes from the same operational PostgreSQL database, transformed on the fly using a Lambda function. The predicted churn score is stored in an Amazon DynamoDB table for fast retrieval by the agent's dashboard.

This real-world scenario shows that the IT professional does not write most of the data pipeline code from scratch. Instead, they configure AWS managed services (DMS, Firehose, Glue, Lambda) to do the heavy lifting. The key decisions are choosing the right data store for each stage (raw S3, clean S3, operational DynamoDB), selecting the ingestion method (batch vs stream), and ensuring transformation steps are performed in the correct order (usually before training, not during inference). The exam expects you to know which service fits which part of this workflow.