You are designing a system that ingests high-velocity event streams from IoT devices using Pub/Sub. Each event must be processed exactly once to update a Firestore database. However, due to the distributed nature, at-least-once delivery is guaranteed by Pub/Sub. Which design pattern should you use to achieve exactly-once processing?
Trap 1: Use a Cloud Function with a retry policy to ensure delivery, and…
Bigtable row keys can help but the primary pattern is idempotency.
Trap 2: Use Cloud Dataflow with exactly-once processing mode and write to…
Dataflow exactly-once still requires idempotent writes to the sink.
Trap 3: Use a Cloud Run service to pull messages and write to Firestore;…
Firestore does not automatically deduplicate based on document ID.
- A
Use a Cloud Function with a retry policy to ensure delivery, and deduplicate using a Cloud Bigtable row key.
Why wrong: Bigtable row keys can help but the primary pattern is idempotency.
- B
Use Cloud Dataflow with exactly-once processing mode and write to Firestore using a custom sink.
Why wrong: Dataflow exactly-once still requires idempotent writes to the sink.
- C
Use a Cloud Run service to pull messages and write to Firestore; rely on Firestore's built-in deduplication using document IDs.
Why wrong: Firestore does not automatically deduplicate based on document ID.
- D
Make the message processor idempotent by using a unique event ID as the Firestore document ID, and perform upsert operations.
Idempotent processing with upsert ensures exactly-once effect.