This chapter covers EventBridge Pipes and event filtering, two powerful features for building event-driven architectures on AWS. EventBridge Pipes provide a simple, cost-effective way to connect event sources to targets with built-in filtering, enrichment, and transformation—all without writing custom code. For the DVA-C02 exam, understanding Pipes and filtering is crucial as it appears in approximately 5-8% of questions, often in scenarios requiring decoupling, real-time processing, or data transformation. You will need to know when to use Pipes versus EventBridge Event Buses, how filtering syntax works, and how to configure enrichment with Lambda or Step Functions.
Jump to a section
Imagine a factory assembly line where raw materials arrive on conveyor belts from different suppliers. Each supplier may send materials in different containers—some in crates, others in barrels, some with extra packaging that needs to be removed. The factory has a central processing station that only accepts materials in standardized bins. To handle this, the factory installs a 'Pipe' between each supplier's conveyor and the central station. The Pipe automatically picks up each raw material unit, inspects it, filters out any defective or unwanted items (e.g., broken parts, wrong size), transforms the material into the standardized bin format (e.g., removes packaging, adds a label), and then places it onto the central station's conveyor. The Pipe can also enrich the material by adding information from a separate database (e.g., a serial number lookup) before placing it. If the central station is busy, the Pipe holds the material temporarily in a buffer until the station is ready. If anything fails, the Pipe sends the failed item to a designated scrap bin for later analysis. This assembly line is fully automated—no manual intervention needed once configured. The factory manager can monitor each Pipe's throughput, error rates, and buffer usage from a central dashboard. In AWS EventBridge Pipes, the suppliers are event sources (SQS, DynamoDB Streams, Kinesis, etc.), the central station is a target (Event Bus, Step Functions, API Gateway, etc.), and the Pipe itself handles filtering, enrichment, and transformation automatically.
What is EventBridge Pipes?
EventBridge Pipes is a serverless point-to-point integration service that allows you to connect event sources to targets with built-in filtering, enrichment, and transformation. Unlike EventBridge Event Buses, which are designed for event routing across multiple consumers, Pipes are optimized for simple, high-throughput, single-source-to-single-target patterns. They are particularly useful for streaming data from sources like Amazon SQS, DynamoDB Streams, Kinesis Data Streams, or Amazon MQ to targets such as EventBridge Event Buses, Step Functions, API Gateway, Kinesis Firehose, or S3.
How Pipes Work Internally
When you create a Pipe, you specify a source, a target, and optionally a filter and an enrichment. The Pipe continuously polls the source for new events (e.g., messages from an SQS queue, records from a DynamoDB stream). It applies any configured filter to discard events that do not match the criteria. If an enrichment (Lambda function or Step Functions state machine) is defined, the Pipe sends the filtered event to the enrichment for processing, which can transform the event payload or add data from external sources. The enriched event is then sent to the target. If no enrichment is defined, the filtered event goes directly to the target. The Pipe handles batching automatically—it can batch multiple events from the source and send them as a batch to the target, depending on the target's capabilities.
Key Components and Configuration
Source: The source of events. Supported sources: SQS (standard and FIFO), DynamoDB Streams, Kinesis Data Streams, Amazon MQ (ActiveMQ and RabbitMQ), and self-managed Kafka. For SQS FIFO, ordering is preserved. For streams, records are processed in order per shard/partition.
Filter: A JSON-based filter pattern that determines which events are passed through. The filter syntax is the same as EventBridge event patterns. Multiple filters can be combined with OR logic. If no filter is specified, all events pass through.
Enrichment: Optional Lambda function or Step Functions state machine that receives the event, processes it, and returns a modified event. The enrichment can call other AWS services (e.g., DynamoDB GetItem) to enrich the data.
Target: The destination for the (filtered and enriched) event. Supported targets: EventBridge Event Bus, Step Functions state machine, API Gateway (REST or HTTP), Kinesis Data Streams, Kinesis Firehose, SQS queue, SNS topic, Lambda function, and more.
Batch size: The number of records to read from the source at once. Default is 10 for SQS and 100 for streams. Maximum is 10,000 for streams.
Maximum record age: The maximum age of a record before it is discarded. Default is 1 day (86400 seconds) for streams.
Retry policy: If the target is unavailable, the Pipe will retry sending the event. The default retry count is 3, with a maximum of 5. The retry interval is exponential backoff starting at 1 second.
Dead-letter queue: Events that cannot be delivered after all retries can be sent to an SQS queue or SNS topic for further analysis. This is optional but recommended.
Filtering Syntax and Examples
Event filtering uses the same pattern syntax as EventBridge event rules. The filter is a JSON object where each key corresponds to a field in the event. You can use comparison operators like $or, $and, $not, $eq, $prefix, $wildcard, $exists, $numeric (for numeric comparisons), and $anything-but. For example:
{
"source": ["aws.ec2"],
"detail-type": ["EC2 Instance State-change Notification"],
"detail": {
"state": ["running", "stopped"]
}
}This filter passes only EC2 instance state change events where the new state is either 'running' or 'stopped'. Multiple filters can be specified in an array, and they are combined with OR logic.
Enrichment with Lambda or Step Functions
Enrichment allows you to transform or augment events before they reach the target. You can configure a Lambda function or a Step Functions state machine as the enrichment. The enrichment receives the event as input and must return a JSON object that will be sent to the target. If the enrichment fails (e.g., Lambda throws an error), the Pipe can be configured to send the original event to the target or discard it. The enrichment is synchronous—the Pipe waits for the enrichment to complete before sending the event to the target.
Interaction with Related Services
EventBridge Event Buses: Pipes can send events to an Event Bus, which then routes them to multiple targets based on rules. This is a common pattern for centralizing events from multiple sources.
Step Functions: Pipes can directly invoke a Step Functions state machine as a target, starting a new execution for each event (or batch). This is useful for orchestrating complex workflows.
Lambda: Pipes can invoke a Lambda function as a target, either synchronously or asynchronously. For asynchronous invocation, the Pipe handles retries and DLQ.
SQS: Pipes can read from SQS and write to another SQS queue, effectively acting as a message filter/transformer.
DynamoDB Streams: Pipes can process DynamoDB stream records, filter them, and send them to various targets for change data capture (CDC).
Performance and Scaling
Pipes are designed to handle high throughput. For SQS sources, the Pipe polls the queue with long polling (20 seconds) and can process up to 1000 messages per second per Pipe. For Kinesis and DynamoDB Streams, the Pipe reads from each shard in parallel, scaling automatically as the number of shards increases. There is no limit on the number of Pipes you can create. Pricing is based on the number of events processed (after filtering) and any enrichment invocations.
Monitoring and Troubleshooting
You can monitor Pipes using CloudWatch metrics: Invocations, InvocationsFailed, TargetInvocationFailed, EnrichmentFailed, FilteredEvents, DroppedEvents. CloudWatch Logs can be enabled to capture detailed logs for each event processed. If events are being dropped, check the filter pattern, the DLQ configuration, and the target's permissions (resource-based policies).
Create a Pipe Source
First, identify the event source. For example, an SQS standard queue. The Pipe will poll this queue using long polling (20 seconds by default). Ensure the SQS queue has appropriate permissions allowing EventBridge to receive messages. The queue must exist in the same AWS account. For streams (Kinesis, DynamoDB), the Pipe reads from the stream's shards. The source must be configured with an IAM role that grants `sqs:ReceiveMessage`, `sqs:DeleteMessage`, and `sqs:GetQueueAttributes` (or equivalent for other sources). The Pipe will start polling as soon as it is created.
Define Filtering Criteria
Create a JSON filter pattern to select only relevant events. For example, to process only orders with amount > 100, use: `{"detail": {"amount": [{"numeric": [">", 100]}]}}`. The filter is applied before enrichment and targeting. If multiple filters are provided, they are ORed. Events that do not match any filter are silently discarded (counted as `FilteredEvents` metric). Filters can reduce cost and downstream load. Test your filter pattern using the EventBridge test console.
Configure Enrichment (Optional)
Choose an enrichment: a Lambda function or Step Functions state machine. The enrichment receives the filtered event and returns a new event. For example, a Lambda function can look up customer details from DynamoDB and add them to the event. The enrichment must be in the same AWS account. Ensure the enrichment's resource policy allows EventBridge to invoke it. The Pipe waits for the enrichment to complete; if it fails, you can choose to send the original event to the target or drop it. Enrichment adds latency and cost.
Set Target and Output
Select a target, e.g., an EventBridge Event Bus. The target receives the filtered/enriched event. Configure the target's input transformer (optional) to modify the event payload before delivery. For example, you can extract only specific fields. Ensure the target's resource policy allows EventBridge to send events. For targets like Lambda or SQS, the Pipe uses synchronous invocation and retries on failure. Configure a dead-letter queue for failed events.
Set IAM Permissions and Create Pipe
Create an IAM role that the Pipe will assume. The role needs permissions for the source (e.g., `sqs:ReceiveMessage`), enrichment (e.g., `lambda:InvokeFunction`), and target (e.g., `events:PutEvents`). Attach the role to the Pipe. Then create the Pipe using AWS Management Console, CLI, or CloudFormation. Example CLI: `aws pipes create-pipe --name my-pipe --source "arn:aws:sqs:us-east-1:123456789012:my-queue" --target "arn:aws:events:us-east-1:123456789012:event-bus/default" --role-arn "arn:aws:iam::123456789012:role/pipe-role"`.
Monitor and Troubleshoot
After creation, the Pipe immediately starts processing. Monitor CloudWatch metrics for failures. If events are not reaching the target, check the filter pattern—maybe it's too restrictive. Check the enrichment function for errors. Verify IAM permissions. Use CloudWatch Logs for detailed event processing logs. If the source is SQS, ensure messages are not being deleted before processing (the Pipe deletes after successful delivery). For streams, check iterator age—if it's high, the Pipe may be falling behind.
Enterprise Scenario 1: Real-Time Order Processing
A large e-commerce company uses DynamoDB as its order database. Every order insertion generates a stream record. They need to process orders in real-time: filter out test orders, enrich with customer loyalty data from another DynamoDB table, and then send the enriched order to a Step Functions workflow for fulfillment. They use EventBridge Pipes with DynamoDB Streams as source, a Lambda function for enrichment (fetches loyalty tier), and a Step Functions state machine as target. The Pipe handles batching (100 records per batch) and retries. In production, they process 10,000 orders per second across multiple shards. A misconfiguration: they forgot to set a filter for test orders, causing unnecessary Lambda invocations and cost. They added a filter pattern {"dynamodb": {"NewImage": {"isTest": [{"exists": false}]}}} to exclude test orders.
Enterprise Scenario 2: Centralized Log Aggregation
A SaaS company collects application logs from multiple microservices into separate SQS queues (one per service). They want to centralize logs into a single EventBridge Event Bus for routing to a log analytics pipeline and a security monitoring system. They create one Pipe per SQS queue, each filtering for error-level logs (using a filter on the log level field), and send them to the central Event Bus. The enrichment is not needed. Each Pipe processes up to 1000 messages per second. A common issue: if the target Event Bus is throttled (default limits 10,000 events/second per bus), events get dropped. They increased the Event Bus quota and added a DLQ to capture failures. They also monitor TargetInvocationFailed metric.
Scenario 3: IoT Device Telemetry
An IoT company receives device telemetry via Kinesis Data Streams. They need to filter out low-priority readings (temperature < 30°C) and transform the payload to include device metadata from a DynamoDB table. They use a Pipe with Kinesis source, a Lambda enrichment that queries DynamoDB for device location, and target is Kinesis Firehose for S3 storage. The Pipe processes 500 shards. A performance consideration: the enrichment Lambda must be fast to avoid backpressure. They set the Lambda concurrency to match the number of shards (500). If the Lambda throttles, events are retried, causing latency. They optimized by using DynamoDB Accelerator (DAX) for low-latency lookups.
DVA-C02 Exam Focus on EventBridge Pipes and Filtering
The DVA-C02 exam tests your ability to choose the right integration pattern and configure filtering correctly. The relevant objective is Domain 1 (Development), Objective 1.6: "Integrate AWS services into an application." Specifically, you must know when to use EventBridge Pipes versus EventBridge Event Buses, SQS, or Kinesis Data Firehose. Common exam scenarios: (1) You need to process messages from an SQS queue with filtering and transformation before sending to a Step Functions workflow. (2) You need to capture changes from DynamoDB Streams and route them to different targets based on the type of change. (3) You need to filter out certain events before they reach a Lambda function to reduce invocations.
Most Common Wrong Answers
Using EventBridge Event Buses instead of Pipes: Candidates often choose Event Bus for point-to-point integration. However, Event Buses are for publish/subscribe patterns with multiple consumers. Pipes are simpler and cheaper for single-source-to-single-target with filtering/enrichment.
Configuring filter on the source (e.g., SQS message attribute) instead of Pipe filter: Some think they can use SQS message filtering, but SQS does not support content-based filtering. Pipe filter is the correct place.
Using Lambda as target and writing filtering logic inside: While possible, it's less efficient and more expensive than using Pipe's built-in filter. The exam wants you to recognize the serverless-first approach.
Forgetting to configure a dead-letter queue: The exam often presents a scenario where events fail and are lost. The correct answer includes setting up a DLQ.
Specific Numbers and Terms
Batch size defaults: SQS=10, streams=100.
Maximum retry count: 5 (default 3).
Maximum record age: 86400 seconds (1 day).
Filter syntax uses $or, $and, $numeric, etc.
Supported sources: SQS, DynamoDB Streams, Kinesis, Amazon MQ, self-managed Kafka.
Supported targets: Event Bus, Step Functions, API Gateway, Kinesis, Firehose, SQS, SNS, Lambda, etc.
Edge Cases
FIFO queues: Pipes support SQS FIFO, but only one Pipe can poll a FIFO queue at a time. Multiple Pipes on the same FIFO queue will cause throttling.
Ordering: For FIFO sources, events are processed in order. For standard sources, ordering is not guaranteed.
Enrichment failure: If enrichment fails, you can choose to send the original event to the target or drop it. The exam may ask which behavior is configured.
Input transformer: You can transform the event payload before sending to the target using a template. This is different from enrichment.
How to Eliminate Wrong Answers
If a question asks for a way to filter and transform messages from SQS before sending to a Step Functions state machine, eliminate options that use Lambda for filtering (costly) or Event Bus (overkill). The correct answer is EventBridge Pipes with a filter and optional enrichment. If the question mentions enrichment, ensure the option includes Lambda or Step Functions. If it mentions ordering, check for FIFO support.
EventBridge Pipes provide a simple, serverless way to connect one event source to one target with built-in filtering, enrichment, and transformation.
Supported sources: SQS, DynamoDB Streams, Kinesis, Amazon MQ, self-managed Kafka. Supported targets: Event Bus, Step Functions, API Gateway, Kinesis, Firehose, SQS, SNS, Lambda, and more.
Filter syntax uses JSON pattern matching with operators like $or, $numeric, $prefix, etc. Multiple filters are ORed.
Enrichment is optional and can be a Lambda function or Step Functions state machine. It must return a JSON payload.
Batch size defaults: 10 for SQS, 100 for streams. Maximum retry count is 5 (default 3). Maximum record age is 86400 seconds.
Pipes use IAM roles for permissions. The role must have permissions for source, enrichment, and target.
Common exam trap: using Event Bus instead of Pipe for a simple source-to-target pattern. Choose Pipe for point-to-point with filtering/enrichment.
Always configure a dead-letter queue for failed events to avoid data loss.
These come up on the exam all the time. Here's how to tell them apart.
EventBridge Pipes
Point-to-point: one source, one target.
Built-in filtering, enrichment, and transformation.
Polls sources like SQS, DynamoDB Streams, Kinesis.
Simpler to set up for single-flow patterns.
Supports FIFO ordering from SQS FIFO.
EventBridge Event Buses
Publish/subscribe: many sources can send to one bus, many targets can receive.
No built-in enrichment; uses rules for filtering and input transformation.
Receives events via PutEvents API, not polling.
Better for event routing to multiple consumers.
Does not support FIFO; ordering is best-effort.
Mistake
EventBridge Pipes can only send events to EventBridge Event Buses.
Correct
Pipes support many targets including Step Functions, API Gateway, SQS, SNS, Lambda, Kinesis, Firehose, and more. Event Bus is just one option.
Mistake
Pipes can filter events based on SQS message attributes.
Correct
Pipes filter based on the event payload (JSON body), not on SQS message attributes. The filter is applied after receiving the message and parsing the body.
Mistake
You need to write custom code to transform events in Pipes.
Correct
Pipes have a built-in input transformer that can modify the event payload using a template. No code needed. For complex transformations, you can use Lambda enrichment.
Mistake
Pipes guarantee exactly-once delivery.
Correct
Pipes provide at-least-once delivery. Events may be delivered more than once in rare cases (e.g., retries). For exactly-once, use FIFO sources and idempotent targets.
Mistake
Pipes can read from multiple sources at once.
Correct
Each Pipe has exactly one source. To read from multiple sources, you need multiple Pipes.
Reveal each answer, then mark whether you got it right. Score 60%+ to unlock the next chapter.
EventBridge Pipes are for point-to-point integration: one source (e.g., SQS queue) to one target (e.g., Step Functions) with optional filtering and enrichment. EventBridge Event Buses are for publish/subscribe: many sources can send events to a bus, and many targets can receive events via rules. Use Pipes when you need a simple, direct flow with built-in processing. Use Event Buses when you need to route events to multiple consumers.
Yes, Pipes can filter events from SQS based on the message body content. You specify a JSON filter pattern that is applied to the event payload. Events that do not match are discarded. This is more efficient than consuming all messages and filtering in a Lambda function.
Enrichment is an optional step where a Lambda function or Step Functions state machine processes the event and returns a modified payload. Use it to add data from other sources (e.g., database lookups) or to transform the event format. For simple transformations, use the input transformer instead.
Yes, if the source is an SQS FIFO queue, Pipes preserve the order of messages. However, only one Pipe can poll a FIFO queue at a time to maintain ordering. For streams like Kinesis, ordering is preserved per shard.
Configure a dead-letter queue (DLQ) on the Pipe. Events that fail after all retries are sent to the DLQ (SQS queue or SNS topic). You can then process these events later. Without a DLQ, failed events are lost.
No, each Pipe has exactly one target. To send to multiple targets, you can send to an EventBridge Event Bus and then use rules to route to multiple targets, or create multiple Pipes.
For SQS, the maximum batch size is 10 (default). For streams (Kinesis, DynamoDB), the maximum batch size is 10,000 records. You can configure the batch size when creating the Pipe.
You've just covered EventBridge Pipes and Event Filtering — now see how well it sticks with free DVA-C02 practice questions. Full explanations included, no account needed.
Done with this chapter?