For exam objective 2.2 — Monitor data pipeline operations and implement data quality checks — you must understand how to ensure your data is trustworthy and your pipelines are healthy. This concept matters because every business decision made from bad data is worse than no decision at all, and AWS charges you real money if your pipeline silently fails for hours. If you are studying for the DEA-C01, you must know not just what services exist, but how they work together to catch data problems before they cost your company millions.
Jump to a section
A simple way to picture Data Quality, Monitoring, and Alerting with AWS Services
Ever been to a busy restaurant where the food comes out perfectly every time, and when something goes wrong—like a burnt steak—the manager catches it before it even reaches your table? How do they do that? It’s not luck, and it’s not just a great chef. It’s a system of constant quality checks, real-time monitoring, and instant alerts.
Here’s how: Imagine the kitchen is your data pipeline. Every dish is a piece of data flowing through your pipeline—from the fridge (raw data storage) to the plate (your final report or application). The head chef isn’t personally tasting every single dish; that would take forever. Instead, the chef has a set of rules written on a whiteboard: “Every steak must be medium-rare, not well-done” (data quality rule 1). “Every order must be plated within 10 minutes of being fired” (latency check). The chef also has a sous-chef whose only job is to walk the line and spot-check dishes (automated data quality checks).
But even with that, things can slip: a ticket gets buried (data backlog), or a grill runs too hot (processing error). That’s where the monitoring system comes in. The chef has a buzzer that sounds if any dish stays on the pass for more than 2 minutes (alert for data freshness issue). Another buzzer goes off if the temperature in the walk-in fridge rises above 40°F (alert for data storage corruption). Every 30 minutes, the chef reviews a printed report of all dishes served, times, and any complaints (monitoring dashboard). If an alert sounds—say the fridge buzzer goes off—the chef drops everything, calls the repair person, and quarantines any potentially spoiled ingredients (incident response).
This is exactly how AWS handles data quality, monitoring, and alerting. You define rules for what “good” data looks like. AWS services constantly measure your data against those rules. If something breaks a rule, an alert fires, and your team knows instantly. Without this system, you’d be serving cold steak to your customers—running business reports on bad data—and you wouldn’t even know it until someone got sick.
Data quality means ensuring your data is accurate, complete, consistent, and timely. In a world where data flows from hundreds of sources—user clicks, IoT sensors, bank transactions—errors are inevitable. A record might be missing a field, a date might be in the wrong format, or a database might get corrupted mid-stream. Monitoring means constantly watching the health of the systems that move and store data: CPU usage, disk space, error rates, and data volume. Alerting means sending a notification (email, SMS, or an API call to another system) when something crosses a defined threshold, like “more than 5% of records failed validation” or “the data pipeline hasn’t received new data in 10 minutes.”
AWS provides several services to manage this. Amazon CloudWatch is the central hub for monitoring any AWS service. It collects metrics (like CPU utilisation, request count, or latency) and logs (text records of events, such as error messages or API calls). You can set up a CloudWatch Alarm that triggers when a metric goes outside a defined range—for example, if the number of failed database connections exceeds 10 in 5 minutes. CloudWatch Logs can also search for specific patterns in log data using a feature called Logs Insights, which lets you write SQL-like queries to find anomalies, like “Count of 4xx HTTP errors in the last hour.”
For data quality specifically, AWS Glue DataBrew (a visual data preparation tool) and AWS Glue (a serverless ETL service) include built-in data quality checks. You define rules like “Column ‘email’ must contain an ‘@’ symbol” or “Column ‘age’ must be between 0 and 120.” These checks run as part of your ETL jobs, and they can fail the entire job if a certain percentage of rows fail, or they can quarantine bad rows to a separate folder for later inspection. AWS Glue also has a feature called Data Quality in AWS Glue (part of the Glue Studio visual interface) where you can set up “data quality rule sets” without writing code, drag-and-drop style.
Another service is AWS DataZone, which helps with data governance—ensuring data is properly catalogued, discovered, and that access is controlled. Part of that is data quality: DataZone uses AWS Glue Data Quality to run checks and then displays quality scores (e.g., “95% completeness”) on your data catalogue so analysts can decide if a dataset is trustworthy before they use it.
Amazon EventBridge is your alerting backbone. It can listen for CloudWatch alarms or for custom events (like “a Glue job failed with a data quality error”) and then route that event to a target—like sending an email via Amazon SNS (Simple Notification Service), triggering an AWS Lambda function to fix the problem automatically, or forwarding to an incident management tool like PagerDuty. This decouples the detection (CloudWatch) from the response (Lambda, SNS, EventBridge).
What does this replace? In the old days, companies had a person manually checking spreadsheets for missing data, or they ran a script every night at 3 AM that generated a report of errors, which someone looked at the next morning. By then, bad data had already flowed into the database and corrupted reports used by executives. By the time they fixed it, the damage was done. AWS automates all of this in real time, so you catch problems as they happen, not hours or days later.
For the DEA-C01 exam, you also need to understand the concept of “freshness” in monitoring. Freshness is a measure of “how old is the most recent data in my pipeline?” If your pipeline ingests data every minute, but your CloudWatch alarm shows the last successful ingestion was 5 minutes ago, you know something is wrong. You might set an alert for freshness-based metrics, especially for streaming data (like from Amazon Kinesis or Amazon MSK). You’ll also see concepts like “completeness” (what percentage of expected records actually arrived) and “accuracy” (how well the data matches known reference values, like comparing against a master customer list).
Finally, another key service is AWS Data Pipeline (for older batch workflows), but the exam focuses heavily on Glue and CloudWatch. The common scenario is: “You have a Glue ETL job that runs hourly. You want to be notified if more than 2% of rows fail a quality check. What should you use?” The answer typically involves CloudWatch metrics from Glue, a CloudWatch alarm, and an SNS topic for the notification.
Define data quality rules for your dataset
Before you can monitor quality, you must decide what 'good' looks like. Write down rules like: 'order_id must not be null', 'order_amount must be greater than 0', 'email must contain @'. In AWS Glue Data Quality, you can create these rules in a graphical interface or as a JSON rule set. This step is crucial because it determines what your pipeline will accept or reject.
Attach the rule set to the ETL job in Glue
In your Glue job (either in the visual editor or in code), you add a data quality transform that reads the incoming data frame, applies the rules, and either passes the data through (if it passes) or fails the job (if it doesn't). You decide the threshold: do you fail the job if any single bad row exists, or only if more than 1% of rows are bad? This step is where you put your quality rules into action.
Emmit custom quality metrics to CloudWatch
In the same Glue job script, after the quality check, you push a custom metric using the CloudWatch PutMetricData API. For example: 'RowsPassed' and 'RowsFailed'. This allows you to track quality over time and set up CloudWatch alarms on the ratio. Without this step, the quality check happens, but you have no visibility into how often it fails.
Create a CloudWatch Alarm on the bad row percentage metric
Go to CloudWatch > Alarms > Create alarm. Select your custom metric (RowsFailed/(RowsPassed+RowsFailed) * 100). Set a threshold: e.g., greater than 5 for 1 consecutive period of 5 minutes. The alarm will transition to ALARM state when the condition is met. This is your detection layer.
Configure the alarm action to send a notification via SNS
When you create the alarm, or edit it later, you add an SNS topic as the alarm action. The SNS topic must already exist and have subscribers (your email, a Lambda function, etc.). When the alarm goes into ALARM state, it publishes a message to that topic. This is your notification layer. Without this, the alarm exists but nobody hears it.
Test the alerting system with a deliberate failure
Intentionally corrupt a test dataset (e.g., set all amounts to null) and run the Glue job. Verify that: (1) the job fails or writes zero rows, (2) the CloudWatch metric shows the alarm state, (3) you receive the SNS notification (email, SMS). This step proves your system works before it's needed in production.
Set up a CloudWatch dashboard for high-level pipeline health
Create a dashboard that shows key metrics from all your pipelines: number of successful/ failed jobs, average data quality pass rate, freshness of last load, and Lambda invocations (if you use auto-remediation). Share it with your team. This step provides continuous monitoring so you see trends before they become problems.
Imagine you work for an online retailer called “ShopFast.” You have a data pipeline that takes order data from a website (stored in Amazon DynamoDB), runs it through an AWS Glue ETL job to clean and transform it, and then loads it into Amazon Redshift (a data warehouse) for business analysts to run sales reports. Every day, the CEO expects a report titled “Yesterday’s Sales by Region,” and the marketing team uses it to decide which products to advertise. If the report shows zero sales for Spain because the data pipeline broke—and nobody noticed—the marketing team might stop all ads in Spain, losing thousands of pounds in revenue.
Here is what your role as a data engineer looks like in practice:
You start by setting up data quality rules in AWS Glue Data Quality. For example: “The ‘amount’ column must be a positive number,” “The ‘order_date’ column must not be null,” and “The ‘country_code’ must be 2 characters long and exist in our ISO country list.” You create a rule set called “order_entry_quality_v1” and attach it to your Glue ETL job.
In the Glue job script (Python or Scala), you add a few lines of code that call the Glue Data Quality API to apply these rules to the in-memory data frame before writing to Redshift. The code essentially says: “Check these 15 rules against the data. If fewer than 98% of rows pass all rules, then fail the job and do not write any data to Redshift.” This is a guardrail that prevents bad data from ever reaching the warehouse.
Next, you configure CloudWatch to monitor the Glue job. Glue automatically publishes metrics like “Number of rows read,” “Number of rows written,” “Job duration,” and “Error count.” You create a custom CloudWatch metric called “BadRowPercentage,” which your Glue job emits after each run. You then create a CloudWatch Alarm on that metric: if BadRowPercentage exceeds 2% for any single run, trigger a state change to “ALARM.”
You connect that alarm to an Amazon SNS topic. The SNS topic is subscribed to by a Lambda function that sends a Slack message to the #data-pipeline channel, and also to your personal mobile number via SMS. The message says: “Alert: ShopFast order pipeline – bad row percentage is 5.3%. Check Glue job ID abc123 logs.” The on-call data engineer sees the alert within 30 seconds and opens the Glue job logs in CloudWatch Logs to find the specific rows that failed the rule (e.g., all rows from a test server that accidentally went live, injecting orders with null amounts).
You also set up a daily batch monitoring dashboard in CloudWatch Dashboards. This dashboard shows: throughput of all ETL jobs (rows per second), freshness of the data in Redshift (last successful load timestamp), and the count of rows that failed quality checks over the last 7 days. You share this dashboard with the data quality manager and the head of analytics. Every morning, they glance at it and can see if any pipeline is trending towards failure.
Finally, you set up EventBridge rules to catch specific errors. For instance, if a Glue job fails with a Python error (not a data quality failure), EventBridge can trigger a Lambda function that restarts the job automatically after 5 minutes (with a maximum retry count of 3). If the job still fails after 3 attempts, EventBridge sends a higher-priority alert to the on-call engineer using a different SNS topic with a more urgent message.
This entire system catches problems at multiple layers: data quality checks catch bad data before it enters the warehouse, CloudWatch monitors the health of the job itself, and EventBridge orchestrates alerts and auto-fixes. Without this, one engineer would have to manually log in and check every job, which is impossible for 50 pipelines running at all hours.
The DEA-C01 exam tests your understanding of how these services hang together in a real pipeline, not just their individual features. They will give you a scenario—like “A pipeline runs hourly, and suddenly data stops arriving”—and then ask you which combination of services and configurations you would use to detect and alert on that failure. They love to test your ability to choose between similar services: CloudWatch vs. EventBridge vs. SNS vs. SQS.
Key concepts the exam tests directly:
CloudWatch Alarms: You must know the difference between a “static” alarm (e.g., metric > threshold) and an “anomaly detection” alarm (uses machine learning to set a dynamic threshold based on past behaviour). They will ask you which type to use for seasonal data (like higher sales on weekends). The answer is often anomaly detection.
CloudWatch Metric Math: You can combine multiple metrics in a single alarm (e.g., “Request count minus error count”). The exam loves questions about “how to calculate availability as a percentage from CloudWatch metrics” (it’s done via Metric Math in the alarm configuration).
Glue Data Quality vs. DIY custom validation: They will ask when to use Glue’s built-in DQ rules vs. writing your own validation in Spark code. The correct answer is: use Glue DQ for simple rules (null check, type check, range check); write custom code for complex rules (like checking referential integrity across two different data sources).
“Freshness” metric: A common question gives you a data freshness metric threshold of 5 minutes, and asks you to design an alarm. The trap is that freshness is measured in seconds or minutes, not percentage. They will give you a percentage-based answer to see if you notice.
Monitoring vs. Observability: The exam distinguishes “monitoring” (tracking known metrics) from “observability” (exploring unknown issues using logs and traces). They may ask which AWS service is best for observability—the answer is AWS X-Ray for traces, and CloudWatch Logs Insights for unstructured log analysis.
EventBridge event sources: They love to test whether an event can come from an AWS service (like “when an EC2 instance terminates”) or from a custom application. The answer is both, but they will ask “which AWS service allows you to filter events by content” – that’s EventBridge with event patterns.
SNS fan-out vs. SQS polling: You need to know that SNS pushes messages to multiple subscribers (like Lambda and email), while SQS requires a consumer to poll for messages. For alerting, you want SNS because you need immediate push notification, not polling.
Logs retention and exporting: The exam expects you to know that you can export CloudWatch Logs to Amazon S3 for long-term storage by using a Lambda function or a subscription filter. They might ask “how to keep logs for 7 years” – answer is export to S3 with lifecycle policies.
Traps they set:
They will describe a pipeline where the data quality check passes but the pipeline is still broken (e.g., Glue job runs successfully but writes zero rows because source was empty). They want you to say “set a CloudWatch alarm on ‘number of rows written’.”
They will present a scenario where you need alerts, but they mention “SQS with a dead-letter queue” as an option. The correct answer for real-time alerts is SNS, not SQS.
They will list CloudWatch Logs subscription filters as a method to send logs to Lambda—this is true but slower than EventBridge for the same use case. They want you to choose EventBridge for near-real-time responses.
Key definitions to memorise: - “Data quality rule” – a declarative statement about the expected state of a dataset. - “Metric” – a time-ordered set of data points measuring a specific value. - “Alarm” – a state machine that transitions between OK, ALARM, and INSUFFICIENT_DATA based on a metric. - “Freshness” – the time since the last successful (or attempted) data update. - “Completeness” – the percentage of expected records that arrived. - “Accuracy” – how well data matches a trusted source.
CloudWatch is the central hub for all monitoring metrics and logs in AWS; every other service (Glue, Lambda, etc.) sends its native metrics to CloudWatch automatically.
Data quality checks in Glue can be defined visually without code using Glue Data Quality rules, but if you need complex logic (like cross-table referential integrity), you must write custom Python/Spark code in the ETL job.
A CloudWatch Alarm does not notify anyone by itself - you must link it to an SNS topic (which then sends SMS, email, or triggers a Lambda function) or to an EventBridge rule for automated actions.
Freshness is measured in time units (seconds/minutes/hours) and is a critical metric for streaming pipelines — never express freshness as a percentage on the exam.
Amazon EventBridge differs from Amazon SNS because EventBridge can filter events based on content (event patterns) and can route to many target types, while SNS simply pushes a message to all subscribers without content filtering.
When designing an alerting system, always consider the order: detection (CloudWatch metric) → decision (Alarm or EventBridge rule) → notification (SNS topic) → action (Lambda, Slack, S3, etc.).
Data quality rule sets in AWS Glue can be reused across multiple jobs, which reduces duplication and ensures consistent quality checks across pipelines.
CloudWatch Logs Insights is a query engine (like SQL) that you pay for based on the amount of log data scanned, so you should limit the time range and use specific filters to reduce costs.
These come up on the exam all the time. Here's how to tell them apart.
CloudWatch Alarm
Lets you set a static or dynamic threshold on a single metric (or metric math).
Alarm states: OK, ALARM, INSUFFICIENT_DATA. You can configure actions per state change.
Best for metrics that are numeric and continuous (e.g., CPU, count of errors).
EventBridge Rule
Can match on event patterns (e.g., 'source: aws.glue', 'detail: state: FAILED').
No concept of alarm states – it either matches an event or doesn't; actions run immediately when pattern matches.
Best for discrete events that occur at unpredictable times (e.g., 'a file was uploaded to S3' or 'a job failed').
Amazon SNS (Simple Notification Service)
Push-based: sends messages immediately to all subscribed endpoints (email, SMS, Lambda, HTTP).
Designed for broadcasting the same message to many consumers simultaneously.
No message retention by default (immediately delivered or lost if consumer is unavailable).
Amazon SQS (Simple Queue Service)
Pull-based: messages are stored in a queue; consumers must poll to retrieve them.
Designed for decoupling between applications (one producer, one consumer, or a group).
Messages are stored up to 14 days by default; consumer can come back later.
AWS Glue Data Quality (Built-in Rules)
No coding required; drag-and-drop rule creation in Glue Studio.
Only supports simple rules: null checks, type checks, uniqueness, range checks ( >, <, between).
If the rule fails, the entire job fails by default (no built-in quarantine).
Custom Data Validation in Spark Code
Requires writing Python or Scala code in your Glue script (using PySpark or Scala Spark).
Can implement any logic: cross-field comparisons, referential integrity across tables, regex pattern matching.
Can route bad rows to an error folder or separate table without failing the whole job.
Data Freshness Metric
Measures the age of the most recent data: 'time since last successful update'.
Expressed in time units (seconds, minutes, hours).
Useful for streaming or near-real-time pipelines where data should arrive every X minutes.
Data Completeness Metric
Measures the percentage of expected records that actually arrived.
Expressed as a percentage (e.g., 99.5% of expected rows).
Useful for batch pipelines where you have a known count of expected records (e.g., 1M rows from a CRM dump).
Mistake
Monitoring and alerting are the same thing; if you set up CloudWatch, you automatically get alerts.
Correct
Monitoring is collecting and visualising metrics (CloudWatch dashboards). Alerting is a specific action triggered when a metric crosses a threshold (CloudWatch Alarm + SNS + action). You must deliberately create alarms; CloudWatch does not alert by default.
Beginners assume the word 'monitoring' includes alerts because in common parlance 'monitoring' often implies 'watching for problems.' In AWS, they are separate steps, and missing the alarm configuration is a common mistake on the exam.
Mistake
If my Glue ETL job has data quality checks and fails, the data is automatically quarantined to a separate location.
Correct
By default, if a Glue data quality check fails, the entire job fails and no data is written anywhere. You must explicitly add code to redirect bad rows to an 'error' folder or a separate table if you want quarantine behaviour. Glue does not have a built-in quarantine feature.
The word 'quality check' sounds gentle, like a filter. But Glue treats quality failures as job failures by default, which is very strict. Beginners expect a 'soft fail' option that just moves bad data aside without stopping the job.
Mistake
CloudWatch Logs Insights is the same as CloudWatch Logs; it just adds a search box.
Correct
CloudWatch Logs Insights is a separate query engine that uses a SQL-like language to analyse logs. It is not 'just a search box.' It can aggregate, count, and plot log data over time. It also incurs separate cost based on data scanned.
Because both products have 'CloudWatch Logs' in the name, beginners assume they are the same interface with minor additions. They do not realise Logs Insights is a fundamentally different compute/service for querying.
Mistake
Amazon EventBridge is only for scheduling events, like cron jobs.
Correct
EventBridge has two halves: a scheduler (for cron jobs) and an event router (for reacting to events from AWS services or custom apps). The event-based routing is more commonly used for alerting and automation than the scheduler.
The word 'EventBridge' sounds like a scheduler ('bridge' -> 'calendar'?). The scheduler feature is prominent in the console, but the exam focuses more on event-driven architecture.
Mistake
You cannot create a CloudWatch alarm for a custom metric unless you use a third-party library.
Correct
You can emit custom metrics from any application (running on EC2, Lambda, or on-premises) by using the AWS SDK or the CloudWatch PutMetricData API. You do not need any special library—just an IAM role with permissions. Many AWS services (like Glue) also allow you to emit custom metrics directly from the job script.
Newcomers think custom metrics are 'advanced' or require special software, because the AWS console hides the API call. The exam expects you to know that any code can emit metrics.
Reveal each answer, then mark whether you got it right. Score 60%+ to unlock the next chapter.
Use CloudWatch Alarms for metric-based alerts (CPU > 80%). Use EventBridge for event-based alerts (e.g., 'a Glue job has failed' or 'a new file landed in S3'). They can work together: an alarm can trigger EventBridge, and EventBridge can trigger actions. The exam tests the difference.
Yes, by installing the CloudWatch Agent on your on-premises servers (Windows or Linux). The agent sends metrics and logs to CloudWatch. The exam expects you to know this concept, especially for hybrid data pipelines that span AWS and on-premises.
Yes. Glue Data Quality charges based on the number of rows evaluated and the number of data quality rules used. It is not free, but it is often cheaper than writing custom validation scripts. The exam may ask about cost optimisation, but the focus is on functionality, not pricing.
It means the alarm has not yet received enough data points to evaluate whether the metric is above or below the threshold. This is common when you first create the alarm or if the metric stops flowing. It is not an error—it is a temporary state. The exam expects you to recognise it as a possible alarm state.
Use Amazon CloudWatch to monitor the pipeline (e.g., Glue job success/failure metrics), create a CloudWatch Alarm on the 'JobRunState' metric (terminal failure = 1), and link that alarm to an Amazon SNS topic that sends an SMS or email. For faster response, replace SNS with EventBridge + Lambda to automatically restart the job.
Yes, but it requires more steps. You can create a metric filter in CloudWatch Logs that counts occurrences of a specific string (e.g., 'ERROR'), and then create a CloudWatch Alarm on that metric. This is useful for app-level logs where errors are in text, not numeric metrics. The exam tests metric filters.
They are often used interchangeably, but in AWS Glue, 'data quality' refers to the built-in features (rule sets, profiling), while 'data validation' is a broader term that can include custom logic. The exam uses 'data quality' for the specific Glue feature.
You've finished Data Quality, Monitoring, and Alerting with AWS Services. Continue through the DEA-C01 study guide to build a complete picture of the exam.
Done with this chapter?