← PDE·Google Cloud

Question 192 of 499

Designing data processing systems →hardMultiple ChoiceObjective-mapped

Quick Answer

The answer is to enable Cloud Storage as a shuffle destination by setting `dataproc:dataproc.shuffle.direct` to `true` and ensuring appropriate IAM permissions. This is correct because the high disk I/O on local SSDs indicates that shuffle data is being written to ephemeral local storage, creating a bottleneck—especially on preemptible VMs where data is lost on preemption. By routing shuffle data directly to Cloud Storage, you bypass local disk contention and leverage Google Cloud’s high-throughput object storage, which dramatically improves shuffle performance for data-heavy batch jobs. On the Google Professional Data Engineer exam, this scenario tests your understanding of Dataproc’s shuffle architecture and the trade-offs between local disk and cloud-native storage; a common trap is assuming more workers or faster SSDs will fix shuffle slowness, when the real fix is offloading shuffle to Cloud Storage. Memory tip: “Shuffle to the cloud, not the disk—direct is the trick.”

PDE Designing data processing systems Practice Question

This PDE practice question tests your understanding of designing data processing systems. Read the scenario carefully and evaluate each option against the stated constraints before committing to an answer. After answering, compare your reasoning against the explanation and wrong-answer breakdown below. Once you have made your selection, read the full explanation to reinforce the concept and understand why each distractor is designed to mislead on exam day.

A large e-commerce company is migrating its on-premise Hadoop cluster to Google Cloud using Dataproc for batch processing. The cluster processes daily sales data from multiple sources, generates aggregated reports, and performs ad-hoc analysis. The migration is complete, but users report that jobs are running 30% slower than on-premise. The data is stored in Cloud Storage as Parquet files partitioned by date. The Dataproc cluster uses preemptible VMs for worker nodes, and the master node uses a standard VM. The jobs heavily rely on shuffling data between stages. The cluster's autoscaling is enabled with a minimum of 10 and a maximum of 50 workers. During job execution, CPU utilization on workers is low, but disk I/O is high, especially on local SSDs. The network utilization is moderate. The team suspects that the shuffle operation is causing the slowdown. Which action should the team take to improve job performance?

Clue words in this question

Noticing these words before you look at the options changes how you read each choice.

Clue: "minimum / minimize"
Why it matters: Asks for the least resource use — fewest addresses, smallest subnet, lowest overhead. Eliminate over-provisioned options even if they would technically work.

Question 1hardmultiple choice

Full question →

A
Attach additional local SSDs to each worker to increase local disk capacity and I/O throughput.
Why wrong: Adding more disks may help but increases cost and complexity; the problem is likely that shuffle data is spilling to disk excessively; using Cloud Storage shuffle can be more scalable.
B
Enable Cloud Storage as a shuffle destination by setting the property `dataproc:dataproc.shuffle.direct` to `true` and ensure the cluster has appropriate IAM permissions.
Cloud Storage shuffle can offload intermediate shuffle data to Cloud Storage, reducing local disk I/O and potentially improving overall shuffle performance, especially when local disks are saturated.
C
Change all worker VMs from preemptible to standard VMs to avoid preemption and improve reliability.
Why wrong: Standard VMs are more reliable but do not address the shuffle disk I/O bottleneck.
D
Increase the maximum number of preemptible workers to 100 to provide more parallelism.
Why wrong: More workers may increase parallelism but do not directly improve shuffle performance; also preemptible VMs can be reclaimed, causing recomputation.

Full breakdown with real-world context →

Answer choices

Why each option matters

Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.

Correct answer & explanation

✓

Enable Cloud Storage as a shuffle destination by setting the property `dataproc:dataproc.shuffle.direct` to `true` and ensure the cluster has appropriate IAM permissions.

B is correct because the high disk I/O on local SSDs during shuffling indicates that the shuffle data is being written to local disk, which is a bottleneck. By enabling Cloud Storage as a shuffle destination via `dataproc:dataproc.shuffle.direct`, shuffle data is written directly to Cloud Storage, bypassing local disks and leveraging Google Cloud's high-throughput object storage. This reduces disk I/O contention and improves shuffle performance, especially when preemptible VMs are used, as shuffle data is not lost on VM preemption.

Key principle: Answer the scenario, not the keyword: identify the specific constraint before choosing the most familiar-sounding option.

Answer analysis

Option-by-option breakdown

For each option: why learners choose it and why it is or isn't the right answer here.

✗
Attach additional local SSDs to each worker to increase local disk capacity and I/O throughput.
Why it's wrong here
Adding more disks may help but increases cost and complexity; the problem is likely that shuffle data is spilling to disk excessively; using Cloud Storage shuffle can be more scalable.
✓
Enable Cloud Storage as a shuffle destination by setting the property `dataproc:dataproc.shuffle.direct` to `true` and ensure the cluster has appropriate IAM permissions.
Why this is correct
Cloud Storage shuffle can offload intermediate shuffle data to Cloud Storage, reducing local disk I/O and potentially improving overall shuffle performance, especially when local disks are saturated.
Clue confirmation
The clue word "minimum / minimize" in the question point toward this answer.
Related concept
Read the scenario before looking for a memorised answer.
✗
Change all worker VMs from preemptible to standard VMs to avoid preemption and improve reliability.
Why it's wrong here
Standard VMs are more reliable but do not address the shuffle disk I/O bottleneck.
✗
Increase the maximum number of preemptible workers to 100 to provide more parallelism.
Why it's wrong here
More workers may increase parallelism but do not directly improve shuffle performance; also preemptible VMs can be reclaimed, causing recomputation.

Common exam traps

Common exam trap: answer the scenario, not the keyword

The trap here is that candidates often assume adding more local SSDs or increasing worker count will solve shuffle bottlenecks, but the real issue is the I/O bottleneck of local disks, and Cloud Storage shuffle is the specific Dataproc feature designed to offload shuffle data to a scalable, high-throughput object store.

Detailed technical explanation

How to think about this question

Dataproc's Cloud Storage shuffle uses the `dataproc:dataproc.shuffle.direct` property to write shuffle intermediate data to Cloud Storage instead of local disks, leveraging the `gs://` connector with high parallelism and throughput. This is particularly beneficial for jobs with heavy shuffling on preemptible VMs because shuffle data is persisted in Cloud Storage, surviving VM preemption and avoiding recomputation. Under the hood, Spark's shuffle manager is replaced with a custom implementation that writes shuffle blocks as objects in Cloud Storage, using the Hadoop Cloud Storage connector's optimized write path.

KKey Concepts to Remember

Read the scenario before looking for a memorised answer.
Find the constraint that changes the correct option.
Eliminate answers that are true in general but not in this case.

TExam Day Tips

Watch for words such as best, first, most likely and least administrative effort.
Review why wrong options are wrong, not only why the correct option is correct.

Key takeaway

Answer the scenario, not the keyword: identify the specific constraint before choosing the most familiar-sounding option.

Real-world example

How this comes up in practice

A media company stores terabytes of video archives that are accessed once a year for audit purposes. Moving these objects to a cold storage tier (Azure Archive, S3 Glacier, or Google Nearline) costs a fraction of hot storage. Questions like this test whether you understand storage tiers, access frequency tradeoffs, and retrieval latency requirements.

What to study next

Got this wrong? Here's your next step.

Identify which exam domain this question belongs to, review the core concept, then practise similar questions from the same domain.

Related PDE practice-question pages

Use these pages to review the topic behind this question. This is how one missed question becomes focused revision.

Designing data processing systems practice questions

Practise PDE questions linked to Designing data processing systems.

Building and operationalizing data processing systems practice questions

Practise PDE questions linked to Building and operationalizing data processing systems.

Operationalizing machine learning models practice questions

Practise PDE questions linked to Operationalizing machine learning models.

Ensuring solution quality practice questions

Practise PDE questions linked to Ensuring solution quality.

PDE fundamentals practice questions

Practise PDE questions linked to PDE fundamentals.

PDE scenario practice questions

Practise PDE questions linked to PDE scenario.

PDE troubleshooting practice questions

Practise PDE questions linked to PDE troubleshooting.

Practice this exam

Start a free PDE practice session

Short sessions build daily habit. Longer sessions build exam-day stamina. Try a timed session to simulate real conditions.

10 questions 20 questions 30 questions 50 questions Timed 30

PDE practice-test guide →Study guide →Browse all practice tests

FAQ

Questions learners often ask

What does this PDE question test?

Designing data processing systems — This question tests Designing data processing systems — Read the scenario before looking for a memorised answer..

What is the correct answer to this question?

The correct answer is: Enable Cloud Storage as a shuffle destination by setting the property `dataproc:dataproc.shuffle.direct` to `true` and ensure the cluster has appropriate IAM permissions. — B is correct because the high disk I/O on local SSDs during shuffling indicates that the shuffle data is being written to local disk, which is a bottleneck. By enabling Cloud Storage as a shuffle destination via `dataproc:dataproc.shuffle.direct`, shuffle data is written directly to Cloud Storage, bypassing local disks and leveraging Google Cloud's high-throughput object storage. This reduces disk I/O contention and improves shuffle performance, especially when preemptible VMs are used, as shuffle data is not lost on VM preemption.

What should I do if I get this PDE question wrong?

Identify which exam domain this question belongs to, review the core concept, then practise similar questions from the same domain.

Are there clue words in this question I should notice?

Yes — watch for: "minimum / minimize". Asks for the least resource use — fewest addresses, smallest subnet, lowest overhead. Eliminate over-provisioned options even if they would technically work.

What is the key concept behind this question?

Read the scenario before looking for a memorised answer.

About these practice questions

Courseiva creates original exam-style practice questions with explanations and wrong-answer analysis. It does not publish real exam questions, exam dumps, or protected exam content. Learn why practice questions differ from exam dumps →

How Courseiva writes practice questions · Editorial policy

Keep practising

Question Discussion

Share a tip, memory trick, or ask about the reasoning behind this question. Do not post real exam questions, leaked content, braindumps, or copyrighted exam material. Comments are moderated and may be removed without notice.

Loading comments…

This PDE practice question is part of Courseiva's free Google Cloud certification practice question bank. Courseiva provides original exam-style practice questions with explanations, topic-based practice, mock exams, readiness tracking, and study analytics to help learners prepare for the PDE exam.