Data engineering is the practice of designing and building systems that collect, store, transform, and make data usable for analysis and decision-making. In the AWS cloud, these systems allow organisations to handle massive amounts of information – from website clicks to sensor readings – without needing to manage physical servers. For the DEA-C01 exam, you must understand the core value proposition of AWS data engineering: it replaces slow, error-prone manual work with automated, scalable, and secure pipelines that turn raw data into business gold.
Jump to a section
A simple way to picture AWS Data Engineering Overview and Core Concepts
A restaurant kitchen is a system for transforming raw ingredients into finished dishes customers can eat and enjoy. The kitchen is not a single magic machine; it is a collection of specialised stations, each with a specific job, connected by a careful workflow. The chef de partie at the grill station is an Amazon S3 bucket, holding raw beef and vegetables in their original state, organised by shelf (folder). The sous chef who takes those ingredients, chops them, marinates them, and combines them is AWS Glue, performing Extract, Transform, Load (ETL) operations to clean and prepare the data. The expo chef who calls out orders and ensures the right plate goes to the right table is Amazon Simple Queue Service (SQS), managing the flow of work. The head chef who decides the menu and plans the kitchen layout is the Data Engineer, designing the pipeline. The dishwasher who keeps things clean and running is Amazon CloudWatch, monitoring and logging every step. If the grill chef runs out of steak, the inventory system must trigger a reorder, just like an Amazon EventBridge event that starts a new data processing job. The entire kitchen runs on a shared set of equipment – ovens, hobs, sinks – which is the AWS global infrastructure of compute, storage, and networking. Without this organised kitchen, cooking would be chaos: raw ingredients would rot, orders would be lost, and customers would leave hungry. Data engineering is exactly that – creating a clean, reliable, scalable kitchen for data so that the business (the restaurant) can serve its customers (analysts, data scientists, executives) the freshest, most accurate insights on time, every time.
A restaurant kitchen's success depends on each station executing its role perfectly and passing results to the next station without delay or error. This maps directly to the AWS Well-Architected Framework pillars: operational excellence (the chef's standardised recipes), security (locking the walk-in fridge), reliability (a backup generator), performance efficiency (a high-output convection oven), and cost optimisation (buying ingredients in bulk from a wholesaler).
Data engineering sits at the foundation of modern data-driven organisations. Before a data scientist can train a machine learning model or a business analyst can create a sales dashboard, someone must first collect the raw data from various sources, clean it of errors and inconsistencies, convert it into a useful format, and store it in a place where it can be easily queried. That 'someone' is the data engineer, and the set of tools and processes they use is data engineering.
In the traditional, pre-cloud world, data engineering was painful. Companies bought physical servers, installed operating systems, configured databases, and wrote custom scripts to move data around. Every time a new data source appeared – say, a new mobile app or a third-party marketing platform – the engineer had to manually provision more hardware, update firewall rules, and patch software. Scaling meant ordering more hardware and waiting weeks for delivery. This approach was slow, expensive, and brittle.
AWS data engineering changes this completely by providing a suite of managed services. A 'managed service' means AWS handles the underlying infrastructure – the servers, networking, patching, and backups – so you can focus on working with data. The core building blocks of AWS data engineering include:
Storage: Amazon S3 (Simple Storage Service) is the primary data lake. A 'data lake' is a central repository that holds all your data, both structured (like sales records in a spreadsheet) and unstructured (like images or video logs), in its original format. S3 is infinitely scalable and highly durable.
Compute: AWS Glue is a serverless ETL service. ETL stands for Extract, Transform, Load. You extract data from a source (like a database or an S3 bucket), transform it (clean it, remove duplicates, convert data types), and load it into a target (like another S3 bucket or a data warehouse). AWS Glue also provides a Data Catalog, which is a central metadata repository. 'Metadata' is data about data – for example, the column names, data types, and table locations.
Data Warehousing: Amazon Redshift is a fast, fully managed data warehouse. A 'data warehouse' is a specialised database designed for analytical queries (aggregations, joins, complex calculations) on huge datasets. Unlike a transactional database that handles thousands of small writes per second (like an e-commerce order system), Redshift is optimised for reading and analysing large volumes of data.
Orchestration: AWS Step Functions and Amazon Managed Workflows for Apache Airflow (MWAA) let you coordinate multiple steps in a data pipeline. An 'orchestrator' ensures that Step B runs only after Step A finishes successfully, and if Step A fails, it can send an alert or run a fallback job.
Messaging and Streaming: Amazon Kinesis and Amazon Simple Queue Service (SQS) handle real-time data. Kinesis can capture streaming data like clickstreams or IoT sensor readings. SQS is a message queue that decouples components – one service sends a message to a queue, and another service picks it up and processes it when it is ready. This makes the system resilient and scalable.
The value of this ecosystem is that an organisation can build a complete data pipeline – from ingestion to storage to transformation to analysis – without ever provisioning a single physical server. This is the 'serverless' model: you pay only for the resources you consume, and AWS automatically scales everything up and down based on demand.
For the DEA-C01 exam, you specifically need to understand the purpose of each of these core services and how they fit together into a data engineering lifecycle. The exam will ask you to identify the correct service for a given scenario, such as choosing between Kinesis Data Firehose (for loading streaming data directly into S3/Redshift) and Kinesis Data Streams (for custom real-time processing by your own application). You also need to know the difference between a data lake (raw data, any format, S3) and a data warehouse (cleaned, structured data, Redshift).
Data engineering replaces ad-hoc, manual scripts written by a single person (a 'data wrangler') with a robust, repeatable, auditable pipeline managed by multiple teams. It replaces fragile, single-server databases with distributed, fault-tolerant systems. It replaces the question 'can we get this data by Friday?' with the confident answer 'yes, and it will run automatically every hour.'
Define Business Requirements
You start by understanding what the business needs: how frequently data must be updated (real-time vs daily), what types of data sources exist (databases, logs, APIs), and who will consume the data (analysts in SQL, data scientists for ML). This step drives every architectural decision.
Design the Data Architecture
You decide on a data lake (S3) for raw storage, a data warehouse (Redshift) for structured analytics, or both. You choose batch (AWS Glue) or streaming (Kinesis) ingestion based on latency needs. The architecture must also consider security, cost, and AWS Well-Architected pillars.
Set Up Data Ingestion
You configure services to pull data from sources. For example, you use AWS Database Migration Service (DMS) for persistent database replication, Kinesis Data Firehose for streaming logs, or AWS Transfer Family for SFTP-based file uploads. All raw data lands in the S3 data lake.
Build the Transformation Layer
You create AWS Glue ETL jobs written in Python or PySpark to clean, deduplicate, standardise, and enrich the data. The jobs run on a schedule (e.g., every hour) and write the cleaned output to a 'curated' S3 zone, updating the Glue Data Catalog with new table schemas.
Load into Consumption Layer
The transformed data is loaded into Amazon Redshift for fast SQL queries. This can be done via a COPY command directly from S3, or using Redshift Spectrum to query data without moving it. You also create materialised views in Redshift for frequently used aggregations to speed up queries.
Implement Orchestration and Monitoring
You use AWS Step Functions to orchestrate the pipeline: trigger ingestion, wait for Glue job completion, then trigger Redshift load. You add CloudWatch alarms for errors and duration metrics, and set up dashboards in CloudWatch or Grafana to monitor pipeline health and SLAs.
Test, Iterate, and Document
You run end-to-end tests with sample data to validate the pipeline outputs. You document the data lineage (where data came from, how it was transformed) using the Glue Data Catalog and AWS Lake Formation. You schedule regular reviews to optimise cost and performance as data volumes grow.
Imagine you work as a data engineer for a mid-sized e-commerce company called 'ShopFast'. ShopFast sells products online and wants to understand customer behaviour to improve sales and inventory management. Currently, data is scattered everywhere: sales transactions are in a PostgreSQL database, customer website clicks are logged as text files on a web server, and inventory levels are tracked in a spreadsheet. Analysts spend 80% of their time manually copying data, cleaning it in Excel, and creating reports. The business wants a single, reliable source of truth that updates every hour.
Here is what you, the data engineer, would actually do step-by-step using AWS:
Set up the data lake: You create an Amazon S3 bucket called 'shopfast-data-lake'. This will be the central landing zone for all raw data. You create folders (prefixes) like 'raw/sales/', 'raw/clickstream/', and 'raw/inventory/'.
Ingest the sales data: You configure AWS Database Migration Service (DMS) to continuously replicate the PostgreSQL sales database into an S3 bucket in Parquet format (a columnar storage format that is faster for analytics). You also set up a scheduled AWS Glue job that runs every hour to move new records from a staging table to S3.
Ingest the clickstream data: The web server logs are sent to Amazon Kinesis Data Firehose. Firehose automatically buffers the data and writes it to the S3 'raw/clickstream/' folder as new files arrive every 5 minutes. This is a 'streaming' ingestion.
Transform the data: You create an AWS Glue ETL job written in PySpark (Python for Spark) that runs every hour. This job reads the raw data from the S3 data lake, cleans it – removes rows with null customer IDs, standardizes date formats, converts currency columns to decimal – and writes the clean output to a new S3 bucket called 'shopfast-curated-data'. The job updates the AWS Glue Data Catalog so that the new tables are visible to query engines.
Load into a data warehouse: You set up an Amazon Redshift cluster. You use a Redshift COPY command (or a feature called Redshift Spectrum to query data directly from S3) to load the transformed data from the curated bucket into Redshift tables. Now analysts can run SQL queries like 'SELECT product_id, SUM(revenue) FROM sales GROUP BY product_id ORDER BY revenue DESC LIMIT 10' and get results in seconds.
Orchestrate the pipeline: You build a AWS Step Functions state machine that triggers the Glue job, waits for it to complete, then runs the Redshift COPY operation. If any step fails, the state machine sends an alert via Amazon SNS (Simple Notification Service) to the team's Slack channel.
Monitor and maintain: You enable Amazon CloudWatch alarms to track the duration of the Glue job, the number of bytes written to Redshift, and the error rate. You set up a dashboard so the team can see the 'health' of the pipeline.
The result: analysts can now run queries in Amazon Athena (a serverless query service that directly reads S3 data) or Redshift in minutes instead of hours. The CEO can open QuickSight (AWS's business intelligence tool) and see real-time dashboards of sales by region. The company stops losing data when a web server crashes. The data engineer no longer spends weekends fixing broken scripts – because the pipeline is automated and resilient. This is the concrete value of AWS data engineering.
The DEA-C01 exam tests your understanding of the 'why' behind AWS data engineering as much as the 'how'. In the 'Domain 1: Data Engineering Fundamentals' section, which includes objective 1.1, you will see scenario-based questions that require you to distinguish between core concepts. The exam loves to test your ability to match a business requirement to the correct AWS service or architectural pattern.
Here are the specific concepts, question types, and traps you must master:
Data Lake vs Data Warehouse: The exam will present a scenario (e.g., 'A company wants to store raw JSON files from IoT sensors for future machine learning. Which storage solution is most cost-effective?'). The correct answer is nearly always Amazon S3 as a data lake. A trap answer might be Amazon Redshift, which is a data warehouse for cleaned, structured data. Remember: data lake = S3, raw, all formats; data warehouse = Redshift, structured, optimised for queries.
ETL vs ELT: ETL (Extract, Transform, Load) transforms data before loading; ELT (Extract, Load, Transform) loads raw data first, then transforms it inside the target (e.g., Redshift or Snowflake). AWS Glue is typically ETL; Redshift Spectrum enables ELT. The exam may ask which approach reduces time-to-insight for raw data. Know the trade-offs: ETL is more predictable but slower to start; ELT is faster for initial load but can clog the warehouse with raw garbage.
Batch vs Streaming: Batch processing (hourly, daily) using AWS Glue vs streaming (real-time, seconds latency) using Amazon Kinesis or Amazon MSK (Managed Streaming for Apache Kafka). A question might say: 'A financial trading firm needs sub-second alerts on stock price changes. Which service should they use?' The answer is Kinesis Data Streams or MSK, not Glue. The trap is to pick Glue because it is the ETL workhorse.
The AWS Well-Architected Framework: The exam will indirectly test the five pillars (Operational Excellence, Security, Reliability, Performance Efficiency, Cost Optimisation) in data engineering contexts. For example: 'A data pipeline currently runs on a single EC2 instance. To improve reliability, what should be added?' The answer involves making the pipeline stateless and using a managed service like AWS Glue or Step Functions with retries.
Data Lineage and Governance: Concepts like the AWS Glue Data Catalog and AWS Lake Formation. The exam may ask: 'Which service provides a central metadata repository so that users can discover datasets?' Answer: AWS Glue Data Catalog. A common distractor is Amazon Athena, which uses the Data Catalog but is not the catalog itself.
Cost Optimisation: The exam will test whether you know that S3 Intelligent-Tiering automatically moves data between storage classes to save costs, or that using Spot Instances for transient ETL workloads reduces compute costs by up to 90%. A question might describe a nightly batch job that must finish within 6 hours. The cheapest compute option with the condition that it can be interrupted is Spot Instances.
Traps to watch for: The exam loves to include a 'RDS' (Relational Database Service) option as a trap for data warehousing questions. RDS is for online transaction processing (OLTP) – like an e-commerce site – not for analytics. If the scenario mentions 'complex queries on petabytes of data', Redshift is the answer, not RDS. Another trap: confusing Amazon EMR (Elastic MapReduce, for big data processing with Hadoop/Spark frameworks) with AWS Glue (ETL service). Glue is serverless and simpler; EMR gives more control but is more complex to manage.
To prepare, drill on these specific scenario patterns. In the real exam, you will see multiple-choice questions where three answers are clearly wrong and two are plausible. The correct one will match the exact wording of the requirement, not the buzzword. Always look for keywords: 'raw' points to S3, 'clean' or 'analytics' points to Redshift, 'real-time' points to Kinesis, 'serverless ETL' points to Glue.
Data engineering on AWS replaces manual, fragile data processing with automated, scalable, and secure pipelines using managed services like Amazon S3, AWS Glue, and Amazon Redshift.
A data lake (Amazon S3) stores raw data in any format, while a data warehouse (Amazon Redshift) stores structured, cleaned data optimised for analytical queries.
ETL (Extract, Transform, Load) transforms data before storage using services like AWS Glue, whereas ELT (Extract, Load, Transform) transforms data inside the data warehouse.
Batch processing is used for scheduled, periodic data loads (e.g., hourly Glue jobs), while streaming processing (e.g., Amazon Kinesis) handles real-time, continuous data ingestion.
The AWS Well-Architected Framework's five pillars (Operational Excellence, Security, Reliability, Performance Efficiency, Cost Optimisation) guide the design of robust data pipelines.
AWS Glue Data Catalog is the central metadata repository that stores table definitions and locations, enabling services like Athena and Redshift Spectrum to query S3 data.
Orchestrating data pipelines with AWS Step Functions or Amazon MWAA ensures that tasks run in the correct order, with error handling and retries, making pipelines reliable.
Choosing between managed services like Glue (serverless, simpler) and EMR (more control, cluster-based) depends on the complexity and customisation needs of the data transformation tasks.
These come up on the exam all the time. Here's how to tell them apart.
Data Lake
Stores raw data in native format (JSON, CSV, Parquet, images)
Schema is applied on read (schema-on-read)
Cost-effective for large volumes of any data type on Amazon S3
Data Warehouse
Stores processed, structured data in tables
Schema is defined before loading (schema-on-write)
Optimised for complex SQL queries on aggregated data in Amazon Redshift
AWS Glue (ETL)
Serverless, no infrastructure to manage
Best for simple to moderate ETL transformations on a schedule
Pricing per Data Processing Unit (DPU) per second
Amazon EMR (Big Data)
Runs on a cluster of EC2 instances you manage
Best for complex custom processing with Spark, Hadoop, Hive
Pricing per EC2 instance per hour (can use Spot to reduce costs)
Batch Processing
Data is processed in chunks at scheduled intervals (hourly, daily)
Higher latency (minutes to hours)
Cheaper per unit of data, simpler to manage with AWS Glue
Streaming Processing
Data is processed continuously as it arrives (sub-second to seconds)
Very low latency (real-time)
More complex to build, higher cost per unit of data with Amazon Kinesis
Amazon S3 (Object Storage)
Stores files (objects) like JSON, images, logs
Unlimited storage, pay per GB stored per month
Not a database; no SQL queries natively (needs Athena)
Amazon RDS (Relational Database)
Stores structured data in tables with rows and columns
Maximum storage limits per instance type, managed scaling
Full SQL support, optimised for transactions (CRUD operations)
Amazon Redshift (Data Warehouse)
Columnar storage, fast for large aggregations and joins
Designed for analytics and reporting
Higher cost per query but vastly faster on huge datasets
Amazon RDS (Transactional Database)
Row-based storage, fast for single-row lookups and inserts
Designed for transactional workloads (OLTP)
Lower cost for small workloads but slower on large analytical queries
Mistake
Data engineering and data science are the same job with different titles.
Correct
Data engineering is focused on building and maintaining the infrastructure and pipelines that collect, store, and prepare data. Data science uses that prepared data to build models and generate insights. They are distinct roles; a data engineer enables the data scientist.
Newcomers see both roles working with data and assume they are interchangeable. The exam distinguishes the job responsibilities, so blurring them causes wrong answers on questions about who does what.
Mistake
A data lake and a data warehouse are just different names for the same thing.
Correct
A data lake stores raw data in its native format (structured, semi-structured, unstructured). A data warehouse stores processed, structured data optimised for SQL queries. They serve different purposes in a pipeline.
The terms are casually misused in blog posts and by vendors. The exam specifically tests the distinction, and mixing them up leads to choosing the wrong storage service.
Mistake
ETL is always better than ELT because it cleans data upfront.
Correct
ETL and ELT are different approaches with different trade-offs. ETL is better when you need strict governance and schema-on-write. ELT is better when you want speed of ingestion and schema-on-read flexibility. Neither is universally superior.
Many tutorials teach ETL as the standard, leading beginners to believe it is the only correct pattern. The exam tests whether you understand the trade-off between upfront transformation and faster loading.
Mistake
AWS Glue is just another name for Amazon EMR.
Correct
AWS Glue is a serverless ETL service that abstracts away the underlying infrastructure. Amazon EMR is a big data platform that runs frameworks like Apache Spark and Hadoop on a cluster of EC2 instances. Glue is simpler and more integrated with the Data Catalog; EMR gives more control and is better for complex, custom processing.
Both can run Spark jobs, so beginners think they are interchangeable. The exam tests their different pricing models (pay per DPU vs pay per EC2 instance) and use cases (simple transforms vs complex machine learning pipelines).
Mistake
If you store data in S3, you cannot run SQL queries on it.
Correct
Amazon Athena is a serverless service that lets you run SQL queries directly on data in S3. You pay per query. Redshift Spectrum also allows querying S3 data from within a Redshift cluster. S3 is not just a storage blob; it is a queryable data source.
Beginners think 'file storage' means 'no query capability'. This misconception causes them to unnecessarily move data to a database when simple Athena queries would suffice, increasing cost and latency.
Reveal each answer, then mark whether you got it right. Score 60%+ to unlock the next chapter.
A data pipeline is an automated sequence of steps that moves data from one or more sources (like databases or apps) to a destination (like a data warehouse), transforming it along the way. In AWS, you build pipelines using services like AWS Glue, Kinesis, and Step Functions.
Yes, you typically need skills in Python or PySpark for writing ETL logic in AWS Glue or Amazon EMR. You also need to know SQL for querying data in Redshift and Athena. The DEA-C01 exam does not ask you to write code, but assumes you understand the purpose of these languages.
AWS Glue is an ETL service that transforms and moves data. Amazon Athena is a serverless query service that lets you run SQL on data already stored in S3. They work together: Glue prepares the data, and Athena queries it.
No, Amazon S3 is an object storage service, not a database. It stores files (objects) like JSON, Parquet, or images. However, you can run SQL queries on S3 data using Athena, and it is the foundation of a data lake.
Serverless means you do not manage or provision servers. AWS automatically handles the compute resources. For example, AWS Glue runs ETL jobs without you starting or stopping any EC2 instances. You pay only for the duration and resources your job uses.
Redshift is a columnar data warehouse optimised for analytical queries (aggregations, joins) on massive datasets (hundreds of gigabytes to petabytes). RDS (like PostgreSQL) is a row-based database optimised for transaction processing (many small reads/writes). Redshift is far faster for reporting.
It is a central metadata repository that stores table definitions, schemas, and locations for your data in S3. Services like Athena, Redshift Spectrum, and EMR use the Catalog to discover and query data without you manually specifying file formats and paths.
Use batch processing when updates are hourly or daily and latency is not critical, using AWS Glue. Use streaming processing when you need sub-second to minute-level updates, using Amazon Kinesis or MSK, for example in fraud detection or real-time dashboards.
You've finished AWS Data Engineering Overview and Core Concepts. Continue through the DEA-C01 study guide to build a complete picture of the exam.
Done with this chapter?