PDE Designing Data Processing Systems Practice Question

You are designing a Dataflow pipeline that joins two unbounded PCollections from different sources. Which transform should you use?

Answer choices

Why each option matters

Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.

Correct answer & explanation

✓

CoGroupByKey

CoGroupByKey performs a key-based join of multiple PCollections. It can handle unbounded streams with appropriate windowing.

Answer analysis

For each option: why learners choose it and why it is or isn't the right answer here.

✗
ParDo
Why it's wrong here
ParDo applies a per-element user function and has no mechanism to match keys across two unbounded PCollections. It is tempting because DoFns can perform arbitrary logic, but it is the right choice for element-wise transformation, filtering or enrichment, not for key-based correlation of two streams.
✗
Flatten
Why it's wrong here
Flatten merges multiple PCollections of the same type into one, preserving no key relationship, so no join occurs. It is tempting because it combines collections, but it is designed for concatenating homogeneous inputs, not correlating records across two sources by a shared key.
✓
CoGroupByKey
Why this is correct
CoGroupByKey performs a relational join of two or more PCollections sharing a common key type, emitting grouped values per key. It is the designated transform for joining unbounded PCollections, unlike side inputs or per-element lookups that cannot correlate streams.
✗
GroupByKey
Why it's wrong here
GroupByKey groups values per key within a single PCollection; it cannot correlate records from two separate sources. It is tempting because grouping is a join building block, but it is the correct choice only after a CoGroupByKey or when aggregating one collection's values by key.

Go deeper