← PDE·Google Cloud

PDE Storing the Data Practice Question

A data engineer is designing a Cloud Storage bucket for a machine learning training pipeline. The pipeline writes many small Parquet files (about 1 MB each) from a Dataflow job, and a downstream training job reads them sequentially. The engineer wants to minimize the number of storage operations and improve read throughput. The bucket uses Standard storage class and has no lifecycle rules. What should the engineer do?

⚠ Common exam trap

The trap here is focusing on storage class or versioning as a performance fix, when the real bottleneck is the number and size of objects being read.

Answer choices

Why each option matters

Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.

Correct answer & explanation

✓

Configure the Dataflow job to write larger Parquet files, for example by using a longer window or by grouping writes into 128 MB to 512 MB files.

The core issue is many small files, which increases the number of Cloud Storage operations and reduces read throughput because each file requires separate open, metadata, and read operations. Writing larger Parquet files, such as 128 MB to 512 MB, reduces object count and allows the training job to read contiguous data more efficiently. This is a common Dataflow and Cloud Storage optimization: adjust the pipeline to produce fewer, larger shards. Other options address durability, cost class, or billing, not the operation-count and throughput problem.

Answer analysis

Option-by-option breakdown

For each option: why learners choose it and why it is or isn't the right answer here.

✗
Change the bucket storage class to Nearline and add a lifecycle rule to transition objects after 30 days.
Why it's wrong here
Nearline storage is designed for data accessed less than once per month and has lower storage cost but higher access cost and retrieval fees. The training pipeline reads the data frequently, so Nearline would increase cost and introduce retrieval latency. It also does nothing to reduce the number of small objects or improve read throughput. The problem is about file size and operation count, not about storage class cost optimization.
✗
Enable Requester Pays on the bucket so that the training job pays for egress and operations.
Why it's wrong here
Requester Pays shifts the cost of access charges to the requester, but it does not change the number of operations or the file sizes. The training job still performs the same number of GET requests on small files, and read throughput remains limited by per-file overhead. This option changes billing responsibility, not performance. It is also typically used for public datasets where the bucket owner does not want to pay for others' access.
✗
Enable Object Versioning on the bucket and set a lifecycle rule to delete noncurrent versions after 7 days.
Why it's wrong here
Object Versioning helps with recovery from accidental overwrites or deletions, but it does not reduce the number of small objects or improve read throughput. In fact, keeping noncurrent versions increases the number of objects stored and can increase storage costs and listing operations. The training job still reads the same number of small files. This option addresses durability, not the performance and operation-count problem described.
✓
Configure the Dataflow job to write larger Parquet files, for example by using a longer window or by grouping writes into 128 MB to 512 MB files.
Why this is correct
Writing larger Parquet files reduces the number of objects, which lowers the number of storage operations (PUT and GET) and improves sequential read throughput because the training job opens fewer files and can read larger contiguous ranges. Parquet is columnar; larger files also improve compression and encoding efficiency. The engineer can adjust the Dataflow pipeline to batch writes, use a larger shard size, or repartition before writing. This directly addresses both the operation count and read performance.

Quick reference

AWS S3 Storage Class Comparison

Storage Class	Min Duration	Retrieval	Use Case
S3 Standard	None	Immediate	Frequently accessed data
S3 Standard-IA	30 days	Immediate	Infrequent access, rapid retrieval
S3 One Zone-IA	30 days	Immediate	Non-critical infrequent data
S3 Intelligent-Tiering	None	Immediate–hours	Unknown or changing access patterns
S3 Glacier Instant	90 days	Milliseconds	Archive with instant retrieval
S3 Glacier Flexible	90 days	Minutes–hours	Archive, flexible retrieval
S3 Glacier Deep Archive	180 days	Hours	Long-term compliance archive

Go deeper

PDE Storing the Data Practice Question

Why each option matters

Option-by-option breakdown

AWS S3 Storage Class Comparison

Related to this question