SNS Message Filtering and Fan-Out

Amazon SNS (Simple Notification Service) message filtering allows subscribers to a topic to receive only a subset of messages based on message attributes. Without filtering, every message published to an SNS topic is delivered to every subscriber. This can lead to unnecessary processing, increased costs, and complexity. Filter policies are JSON documents attached to a subscription that define which messages the subscriber wants to receive.

In a typical fan-out pattern, a single SNS topic might have multiple SQS queues, Lambda functions, and email endpoints as subscribers. For example, an e-commerce system publishes an "OrderPlaced" event. The inventory service needs only orders containing certain product categories, the shipping service needs only orders with a specific shipping method, and the analytics service needs all orders. Without filtering, each subscriber would receive all messages and must filter them internally, wasting compute and network resources. SNS filtering shifts this responsibility to the message broker, improving efficiency.

When a message is published to an SNS topic, SNS evaluates the message's attributes against each subscription's filter policy. The message attributes are key-value pairs where the key is a string and the value is of type String, String.Array, or Number. The filter policy consists of rules that define matching criteria. The evaluation follows these steps:

{
  "event_type": ["order_placed", "order_shipped"],
  "priority": [{"numeric": [">=", 5]}]
}

This policy matches messages where event_type is either "order_placed" or "order_shipped" AND priority is a number >= 5.

4. Matching Logic: - Exact match: The attribute value must exactly equal one of the values in the array. - Prefix match: Use {"prefix": "start"} to match attributes that start with the given string. - Suffix match: Use {"suffix": "end"} to match attributes that end with the given string. - Numeric range: Use {"numeric": ["op1", value1, "op2", value2]} where operators are <, <=, >, >=, =. - Anything-but match: Use {"anything-but": ["value1", "value2"]} to match any value except those listed. - Exists match: Use {"exists": true} or {"exists": false} to check attribute presence.

You can attach a filter policy to a subscription using the AWS Console, AWS CLI, or SDK. Example CLI command:

aws sns subscribe --topic-arn arn:aws:sns:us-east-1:123456789012:MyTopic --protocol sqs --notification-endpoint arn:aws:sqs:us-east-1:123456789012:MyQueue --attributes '{"FilterPolicy": "{\"event_type\":[\"order_placed\"]}"}'

To update an existing subscription's filter policy:

aws sns set-subscription-attributes --subscription-arn arn:aws:sns:us-east-1:123456789012:MyTopic:abc123 --attribute-name FilterPolicy --attribute-value '{"event_type": ["order_placed"]}'

To verify, use get-subscription-attributes:

aws sns get-subscription-attributes --subscription-arn arn:aws:sns:us-east-1:123456789012:MyTopic:abc123

Fan-out is the ability of a single SNS topic to deliver messages to multiple subscribers simultaneously. This is a core messaging pattern in AWS. Combined with filtering, fan-out becomes a powerful routing mechanism.

In a fan-out architecture:

Within an attribute, multiple values are OR'd. For example:

{
  "event_type": ["order_placed", "order_shipped"]
}

Matches if event_type is either "order_placed" OR "order_shipped".

Across attributes, conditions are AND'd. For example:

{
  "event_type": ["order_placed"],
  "priority": [{"numeric": [">=", 5]}]
}

Matches if event_type is "order_placed" AND priority >= 5.

To achieve OR across attributes, you need multiple subscriptions with different filter policies. For example, to match event_type = "order_placed" OR priority > 5, create two subscriptions: one with policy for event_type, another for priority.

Filtering adds minimal latency (sub-millisecond) to message delivery. The evaluation is done at the SNS service side. There is no limit on the number of messages filtered per second beyond the general SNS throughput limits (which are very high).

Scenario 1: Multi-Service Event Routing in E-commerce

An online retailer uses a single SNS topic for all order-related events. Different microservices subscribe to this topic: Inventory Service, Shipping Service, and Analytics Service. The Inventory Service only cares about events where the product category is 'electronics' to update stock. The Shipping Service only cares about events where shipping method is 'express'. The Analytics Service wants all events. Without filtering, each service would receive every event and filter internally, wasting compute and increasing latency. With SNS filtering, each subscription has a filter policy: Inventory subscription has {"category": ["electronics"]}, Shipping subscription has {"shipping_method": ["express"]}, Analytics has no filter. When an order is placed, the publisher sends a message with attributes category and shipping_method. SNS evaluates and delivers only relevant messages to each queue. This reduces SQS message volume, Lambda invocations, and processing costs. In production, this setup handles millions of events per day with sub-second delivery latency.

Scenario 2: Alerting System with Multiple Channels

A DevOps team monitors infrastructure metrics. They have an SNS topic that receives alerts from CloudWatch Alarms. Subscribers include an email list for critical alerts, a Slack webhook for warnings, and an SQS queue for all alerts for archival. They want: email only for alerts with severity 'critical', Slack only for severity 'warning' or 'critical', and SQS for all. They configure filter policies: Email subscription has {"severity": ["critical"]}, Slack has {"severity": ["warning", "critical"]}, SQS has no filter. This ensures that only the appropriate channels receive the right alerts. Misconfiguration often occurs when the email subscription accidentally receives all alerts because no filter was set, flooding inboxes.

Scenario 3: Real-time Data Pipeline with Filtering

A company streams IoT sensor data to an SNS topic. Each sensor publishes messages with attributes device_type and reading_type. Downstream, they have different consumers: a Lambda function that processes temperature readings for HVAC control, a Kinesis Firehose for all data to S3, and an SQS queue for error readings. Filter policies ensure that the Lambda only gets temperature readings, Firehose gets everything, and the SQS queue gets only messages with reading_type = 'error'. This pattern is scalable and cost-effective, as it reduces unnecessary processing. Common mistakes include using the wrong attribute name or forgetting to include attributes in the message, causing all messages to be dropped for filtered subscriptions.