Real-Time Analytics with Pub/Sub and Dataflow

Google Cloud Pub/Sub is a fully managed, real-time messaging service that allows you to send and receive messages between independent applications. It decouples senders (publishers) from receivers (subscribers) by introducing an intermediary topic. Publishers send messages to a topic, and subscribers receive messages from a subscription attached to that topic. Pub/Sub guarantees at-least-once delivery of messages, meaning a message may be delivered more than once, but it will never be lost. Messages are retained for up to 7 days if not acknowledged.

When a publisher sends a message to a topic, Pub/Sub assigns a unique message ID and stores the message redundantly across multiple zones. The message is then fanned out to all subscriptions on that topic. Each subscription maintains its own queue of messages. Subscribers pull messages from the subscription (pull mode) or receive them via a webhook (push mode). After processing, the subscriber sends an acknowledgment (ACK) to remove the message from the subscription queue. If the subscriber does not ACK within the acknowledgement deadline (default 10 seconds, configurable up to 600 seconds), the message is redelivered. This ensures at-least-once delivery. The maximum message size is 10 MB, and the default message retention is 7 days.

Dataflow is a fully managed service for executing Apache Beam pipelines. Apache Beam is an open-source, unified programming model that allows you to define both batch and streaming data processing jobs with the same code. Dataflow handles the execution, scaling, and fault tolerance automatically. It is ideal for ETL (Extract, Transform, Load), real-time analytics, and event-driven applications.

Dataflow pipelines are defined as a Directed Acyclic Graph (DAG) of transforms (PTransforms) applied to collections of data (PCollections). For streaming, Dataflow uses a technique called "windowing" to group unbounded data into finite windows (e.g., fixed, sliding, session). Each window is processed when its watermark (an estimate of event time completeness) passes a threshold. Dataflow supports exactly-once processing guarantees for streaming, using checkpointing and commit logs. It uses Google Cloud Storage for temporary storage and can automatically scale workers (up to 1000) based on the backlog of unprocessed data.

A common pattern is to use Pub/Sub as a source for Dataflow pipelines. Dataflow subscribes to a Pub/Sub subscription (pull mode) and processes messages in real time. The pipeline can then write results to BigQuery, Cloud Storage, or another sink. Dataflow automatically acknowledges messages after successful processing, ensuring exactly-once semantics if configured. The integration supports both streaming and batch (using Pub/Sub's seek feature to replay messages).

To create a Pub/Sub topic and subscription:

gcloud pubsub topics create my-topic
gcloud pubsub subscriptions create my-sub --topic=my-topic

To run a Dataflow streaming pipeline with Pub/Sub source:

gcloud dataflow jobs run my-streaming-job \
    --gcs-location gs://my-bucket/templates/streaming_beam_pipeline \
    --parameters inputSubscription=projects/my-project/subscriptions/my-sub \
    --region us-central1

To verify the pipeline is running:

gcloud dataflow jobs list --region us-central1 --filter="STATE:RUNNING"

For monitoring, use the Dataflow monitoring interface in the console, which shows step-by-step throughput, watermark lag, and system lag.

A financial services company processes millions of credit card transactions per second. They use Pub/Sub to ingest transaction events from point-of-sale systems. Each transaction is published to a transactions topic. A Dataflow streaming pipeline subscribes to the topic, enriches the transaction with customer profile data from Cloud Bigtable, and applies a machine learning model to score fraud risk. High-risk transactions are published to a fraud-alerts topic, which triggers a Cloud Function to notify the customer via SMS. The pipeline uses sliding windows of 5 minutes to aggregate transaction counts per card. Dataflow's autoscaling handles traffic spikes during holiday seasons. The company configures Pub/Sub's exactly-once delivery to avoid duplicate fraud alerts. A common misconfiguration is setting the acknowledgment deadline too low, causing redeliveries and duplicate processing. The solution uses a deadline of 60 seconds to account for ML inference time.

A manufacturing plant has thousands of IoT sensors sending temperature, pressure, and vibration data every second. Each sensor publishes to a Pub/Sub topic with a sensor ID attribute. A Dataflow pipeline reads from the subscription, groups data by sensor ID using session windows (gap duration 10 seconds), and calculates moving averages. The results are written to BigQuery for long-term storage and to a real-time dashboard via Data Studio. Dataflow's watermark helps handle out-of-order data due to network delays. The plant uses Pub/Sub's ordering keys to ensure sensor data is processed in sequence. A key challenge is managing message volume: each sensor sends 86,400 messages per day, so with 10,000 sensors, the pipeline must handle 864 million messages daily. Dataflow's autoscaling scales to 200 workers during peak. Misconfiguration of window size (too small) can cause excessive output, while too large a window delays insights.

An e-commerce platform tracks user clicks, page views, and purchases. Events are published to a Pub/Sub topic. A Dataflow pipeline performs sessionization (session windows with 30-minute gap) to group user activity. It then computes metrics like conversion rate and average session duration. The output is written to BigQuery for ad-hoc analysis and to a Pub/Sub topic for real-time personalization. The company uses Pub/Sub's filter feature to route only purchase events to a separate subscription for inventory updates. A common issue is that late-arriving data (e.g., from mobile devices offline) can cause session splits. Dataflow's allowed lateness (set to 1 hour) merges late events into the correct session. Without this, sessions are fragmented, leading to inaccurate metrics.