Cloud Pub/Sub Patterns

Cloud Pub/Sub is a fully managed, real-time messaging service that allows you to send and receive messages between independent applications. It provides asynchronous communication, meaning the sender (publisher) does not wait for the receiver (subscriber) to process the message. This decoupling is essential for building scalable, reliable systems.

- Pull Subscription: The subscriber initiates requests to fetch messages. Two sub-types: - Streaming Pull: Uses a bidirectional gRPC stream for low-latency delivery. Recommended for high-throughput scenarios. - Synchronous Pull: The subscriber sends individual Pull requests via HTTP or gRPC. Suitable for low-throughput or legacy systems. - Push Subscription: Pub/Sub sends messages to a webhook endpoint (HTTPS). The endpoint must respond with a 200 OK within the ackDeadline. Push is ideal for Cloud Functions or App Engine.

When a subscriber receives a message, it must ACK within the ackDeadline. If it does not, the message is resent. The default ackDeadline is 10 seconds, but you can modify it per subscription (10 to 600 seconds). For pull subscribers, you can also extend the deadline by sending ModifyAckDeadline requests while processing.

Subscriptions can filter messages based on attributes. For example, a subscription might only receive messages where event_type = 'order_placed'. Filtering reduces the number of messages delivered to subscribers.

If a message repeatedly fails to be processed (e.g., subscriber errors), it can be moved to a dead letter topic after exceeding a maximum number of delivery attempts (default 5, configurable 5-100). This prevents poison messages from blocking the subscription.

You can create a snapshot of a subscription's state (acknowledged/unacknowledged messages) and later seek to that snapshot to replay messages. Useful for debugging or reprocessing.

# Create a topic
gcloud pubsub topics create my-topic

# Create a pull subscription
gcloud pubsub subscriptions create my-sub --topic=my-topic

# Publish a message
gcloud pubsub topics publish my-topic --message="Hello" --attribute="key=value"

# Pull messages
gcloud pubsub subscriptions pull my-sub --auto-ack

# Create a push subscription
gcloud pubsub subscriptions create my-push-sub --topic=my-topic --push-endpoint=https://example.com/push

# Enable exactly-once delivery on a push subscription
gcloud pubsub subscriptions update my-push-sub --enable-exactly-once-delivery

# Set ack deadline
gcloud pubsub subscriptions update my-sub --ack-deadline=60

# Set message retention duration
gcloud pubsub topics update my-topic --message-retention-duration=2d

Use Cloud Monitoring metrics: - pubsub.googleapis.com/subscription/num_undelivered_messages - pubsub.googleapis.com/subscription/ack_message_count - pubsub.googleapis.com/topic/publish_request_count

Alerts can be set for backlog growth or high publish latency.

An online retailer uses Pub/Sub to decouple the order placement service from downstream processors (inventory, shipping, billing). When a customer places an order, the order service publishes a message to orders topic with attributes like event_type=order_placed and user_id. Three subscriptions exist: inventory-sub (push to a Cloud Function that deducts stock), shipping-sub (pull by a Compute Engine worker that generates labels), and billing-sub (push to a Cloud Run service that charges the customer). The system processes thousands of orders per second. The challenge: the billing service sometimes fails (e.g., payment gateway timeout). Without a DLQ, the same message would be retried indefinitely, blocking other messages. By configuring a dead letter topic (orders-dlq) with max delivery attempts of 5, failed messages are moved aside, allowing the subscription to continue processing. The operations team monitors the DLQ and reprocesses failed messages after fixing the issue.

A media company ingests clickstream data from millions of users. Each click event is published to a clicks topic. A Dataflow streaming pipeline reads from a pull subscription, aggregates data in 1-minute windows, and writes to BigQuery. The pipeline uses exactly-once sinks to avoid duplicates. The Pub/Sub subscription has a message retention of 1 day to allow for reprocessing if the pipeline fails. The team uses Cloud Monitoring to alert if the subscription backlog exceeds 10 million messages. They also use ordering keys (user session ID) to ensure events from the same session are processed in order. Misconfiguration: initially, they used a 10-second ack deadline, but the Dataflow windowing caused processing delays. They increased it to 300 seconds to prevent premature redelivery.

A smart building company collects sensor data from thousands of IoT devices. Each device publishes to a sensors topic using the MQTT bridge (IoT Core, now deprecated, but similar). A push subscription sends data to a Cloud Function that stores readings in Firestore. The push endpoint must be a public HTTPS URL with a valid SSL certificate. The team uses message filtering to create separate subscriptions for temperature vs. humidity sensors (using attributes). They set a short ack deadline (10s) because the Cloud Function runs quickly. For devices with intermittent connectivity, they use a pull subscription with a longer retention (7 days) to ensure no data loss. Common mistake: forgetting to enable message ordering for time-series data, causing out-of-order writes.