# Processing Batch Data with Cloud Dataproc

> Chapter 6 of the Courseiva GOOGLE-PDE curriculum — https://courseiva.com/learn/google-pde/processing-batch-data-with-cloud-dataproc

**Official objective:** 2.2 — Design and implement batch data processing using Cloud Dataproc and Apache Spark/Hadoop ecosystem.

## Introduction

Cloud Dataproc is a managed service that lets you run big data jobs on Google Cloud without managing the underlying servers. For the PDE exam, you need to understand how to use Dataproc and the Apache Spark/Hadoop ecosystem to process large volumes of data that arrive in batches, not streams. This matters because most real-world data pipelines (like daily sales reports, monthly financial summaries, or yearly customer analytics) rely on batch processing to transform and analyse terabytes of data cost-effectively.

## The Community Bakery's Monthly Order Analogy

First, the bakery receives thousands of individual cake, bread, and pastry orders throughout the month, and then a baker must process them all at once on a Saturday night. This is like batch data processing — a single, scheduled run that crunches a huge pile of data all at once.

The baker doesn't bake each order separately as it arrives; that would be like streaming data processing (doing things instantly, one by one). Instead, the baker waits until the end of the month, grabs the entire stack of order slips, and organises them by type: all chocolate cakes together, all white loaves together, all croissants together. This sorting step is like the 'map' phase in Apache Spark — breaking the big job into smaller, similar tasks.

Then the baker uses multiple ovens at the same time, each oven handling one batch of identical items. One oven bakes all the cakes, another all the loaves. This parallel work is Cloud Dataproc's strength: it spins up many virtual machines (like ovens) to process data simultaneously. Finally, after baking, the baker boxes the finished goods and preps them for delivery. This assembly-and-pack step mirrors the 'reduce' phase, where results are combined and stored. Cloud Dataproc automates this whole process on Google Cloud — it sets up the ovens (virtual machines), manages the heat (resources), and shuts them down when the baking is done, so you only pay for the time you used. It replaces the old way of buying and maintaining your own bakery (a physical server cluster) with a service that lets you 'rent' the bakery for your Saturday-night batch, then walk away.

## Core explanation

Cloud Dataproc is Google's managed service for running Apache Spark and Apache Hadoop clusters. 'Managed' means Google takes care of spinning up servers, installing software, applying security patches, and shutting everything down — you just define what you need and your code.

Before Dataproc, companies had to buy physical servers, install Hadoop or Spark themselves, configure the network, and employ a team to maintain it all. This was expensive, inflexible, and wasted resources because the cluster sat idle between batch jobs. Dataproc replaces this with a 'pay-as-you-go, spin-up-and-down' model.

**Core concepts to know:**

**Batch data** is collected over a period of time (say, all user clicks from the past 24 hours) and then processed together in one job. This contrasts with **streaming data**, where each event is processed as it arrives. Batch processing is ideal for large-scale transformations, aggregations, and reporting where real-time isn't required.

**Apache Spark** is the most popular processing engine on Dataproc. It breaks a big data job into smaller tasks, distributes them across multiple machines (the cluster), and keeps intermediate results in memory for speed. The key programming abstraction is the **RDD (Resilient Distributed Dataset)** — a fault-tolerant collection of data that can be processed in parallel. You'll also encounter **DataFrames** (like a table in a database) and **SQL queries** on Spark.

**Apache Hadoop** is the older ecosystem that includes HDFS (Hadoop Distributed File System) for storage and MapReduce for processing. MapReduce has two steps: Map (sorting/filtering) and Reduce (aggregating). Spark is generally faster because it uses memory (RAM) instead of reading/writing to disk at every step like MapReduce does.

**How Dataproc works:**
- You define a **cluster**: a group of virtual machines (VMs) that run Spark or Hadoop. A cluster has one **master node** (which coordinates the work) and several **worker nodes** (which do the actual processing).
- You submit a **job** (your Spark or Hadoop code) to the cluster. Dataproc stages the code and data across the nodes.
- The job processes the **data**. Data can come from Google Cloud Storage (GCS, like a cloud hard drive), BigQuery (a data warehouse), or other sources.
- When the job finishes, Dataproc can **auto-delete** the cluster (a 'single-job' cluster) to save money.
- You can also use **workflow templates** to chain multiple jobs together, and **preemptible instances** (cheaper, temporary VMs) for worker nodes to reduce cost.

**Why it exists:** Dataproc solves the problem of 'elasticity'. Your batch job may need 100 machines for 1 hour, then 0 machines for the rest of the day. Instead of owning 100 machines and letting them sit idle, Dataproc lets you start exactly the number you need, only for the time you need. It also integrates with **Cloud Storage** as a replacement for HDFS, so you don't need separate storage servers.

**What it replaces:** On-premises Hadoop clusters, manual server provisioning, and the need to hire specialists to install and tune Spark. It replaces the traditional 'lift-and-shift' model with a cloud-native, managed approach. For the PDE exam, expect questions about choosing between Dataproc and other services (like Dataflow for streaming, or BigQuery for SQL queries), optimising cluster size, and understanding the roles of master and worker nodes.

## Real-world context

Imagine you are a data engineer at a large e-commerce company called 'ShopFast'. Every night, you need to process a full day's worth of user activity: 50 million click events, 2 million purchases, and 1 million product-page views. The marketing team wants a report by 8 AM tomorrow showing the top 100 products by revenue, the average time users spend on a page, and the number of new user sign-ups.

**Step-by-step, what you actually do:**

1. **Define the cluster:** You create a Dataproc cluster with 10 worker nodes (standard VMs) and 1 master node. You know the job will take about 30 minutes, so you set the cluster to auto-delete after the job finishes to avoid extra cost. You tick the box for 'preemptible workers' for 5 of the 10 nodes to save 60% on compute costs (preemptible VMs are cheaper because Google can reclaim them, but you accept the risk of occasional failure).

2. **Stage the code and data:** Your Spark code (written in Python using PySpark) is stored in a Cloud Storage bucket. The raw data (click logs, purchase events) from the day is already sitting in another Cloud Storage bucket. You submit your job to Dataproc through the Console or the gcloud command-line tool. You specify the location of the code and the input/output bucket paths.

3. **Processing:** Dataproc assigns the master node to read input data from Cloud Storage and distribute it across worker nodes. The Spark job runs: first it filters out spam clicks (those happening faster than 100ms), then it groups purchases by product ID, sums revenue, and sorts the results. This all happens in RAM, so it's fast.

4. **Handle failures:** One preemptible worker gets reclaimed by Google in the middle of the job. Spark's fault-tolerance kicks in: it re-computes the lost data from the original copies stored elsewhere. The job completes, but 2 minutes slower.

5. **Output:** The final aggregated report is written back to Cloud Storage as a single file (or partitioned by hour). The marketing team's dashboard reads from that bucket at 7:30 AM. You also set up a **Cloud Logging** alert to notify you if the job takes longer than 45 minutes.

6. **Cleanup:** The cluster auto-deletes itself. You check the billing report: the whole operation cost $4.50. If you had an always-on cluster, it would have cost $120 for the same 24-hour period.

**What an IT professional does daily:**
- Writes PySpark or Scala code that defines transformations (filter, group, join, count).
- Chooses the right cluster size to balance speed vs. cost.
- Monitors job status in the Dataproc UI and looks at logs.
- Uses workflow templates to run multiple jobs in sequence (e.g., first clean the data, then aggregate, then export to BigQuery).
- Migrates old on-premises Hadoop jobs to Dataproc by changing file paths from HDFS to `gs://` (Cloud Storage URIs).
- Thinks about data partitioning — e.g., storing daily logs in folders named by date to make processing faster.

## Exam focus

The PDE exam absolutely tests Dataproc, and it loves specifics. Here is exactly what you need to nail.

**Core exam topics (listed by frequency of appearance):**
- The difference between **Dataproc** and **Dataflow**: Dataproc is for batch (and occasional streaming) using Spark/Hadoop. Dataflow is for streaming-first using Apache Beam. Questions often present a scenario where you must process data once a day — the answer is Dataproc. If they need sub-second latency, it's Dataflow.
- **Managing clusters efficiently**: Single-job clusters vs. long-running clusters. Know that single-job clusters are cheaper because they auto-delete. Long-running clusters should use preemptible instances for workers.
- **Spark vs. MapReduce**: Which engine is in use? Spark uses memory. MapReduce writes to disk at every step. Almost always Spark is the faster choice.
- **Storage integration**: Dataproc can read from Cloud Storage, BigQuery, or HDFS. The PDE will test that Cloud Storage is the recommended default because it's separate from compute (cluster can die, data persists).
- **Preemptible instances**: They cost 60-80% less but can be terminated at any time. Use them for worker nodes, never for the master node (a master failure kills the job).
- **Initialisation actions**: Scripts you run on every cluster node when it starts (e.g., installing libraries). This is how you customise the cluster.
- **Workflow templates**: A way to orchestrate multiple Dataproc jobs (like Step A, then Step B, then Step C) without writing a separate scheduler.

**Traps they set:**
- They will give you a scenario with a job that takes 2 hours and ask 'what cluster should you use?' The trap: thinking a larger cluster always helps. Diminishing returns apply — doubling the nodes doesn't always halve the time because of overhead. Know to look for words like 'data skew' or 'shuffle-heavy job' (these may need more memory per node, not more nodes).
- They will ask: 'Your cluster is idle 50% of the day. What should you do?' The answer: use single-job clusters or auto-scaling (available for Dataproc 2.0+). Never keep a large cluster running.
- They will test that **Cloud Storage** is NOT HDFS. HDFS requires the cluster to be running. Cloud Storage persists jobs. A common wrong answer is to suggest using HDFS for long-term storage.
- They will ask about **optional components**: Dataproc can start with Spark alone, or add Hadoop, Hive, Pig, etc. Know that you enable these at cluster creation time.

**Key definitions to memorise:**
- **Cluster scaling**: Horizontal scaling (adding more workers) vs. vertical scaling (bigger VMs). Vertical scaling helps with memory-intensive jobs.
- **YARN**: The resource manager that allocates CPU and RAM to Spark tasks on the cluster.
- **Spark Executor**: A process running on a worker node that executes Spark tasks. The number of executors affects parallelism.

## Step by step

1. **Define the Cluster Template** — Before any data touches a machine, you define what your cluster looks like: number of worker nodes, type of VMs (e.g., n1-standard-4), whether to use preemptible workers, which Spark version to install, and which optional components (like Hadoop or Hive) to enable. This step is crucial because cluster configuration directly determines job speed and cost.
2. **Stage Data and Code in Cloud Storage** — Upload your input data (e.g., CSV, Parquet, or Avro files) and your Spark script (Python or Scala) to a Cloud Storage bucket. This decouples data from compute — if your job fails or the cluster is deleted, the data remains safe. This is also where you practise naming conventions: use separate folders for raw data, code, and output.
3. **Create the Cluster and Submit the Job** — You submit a job through the Google Cloud Console, gcloud CLI, or API. Dataproc automatically creates the cluster, installs Spark (and any optional components), distributes your code to the master node, and begins executing the job. The master node's YARN resource manager allocates workers to run Spark executors. This is when the actual processing starts.
4. **Monitor the Job** — While the job runs, you check its progress in the Dataproc 'Jobs' tab. You can see Spark stages, number of completed tasks, and worker resource usage. If a preemptible worker dies, you'll see a spike in 'speculative execution' (re-running lost tasks). You can also stream logs to Cloud Logging to debug errors in your code.
5. **Handle Output and Delete the Cluster** — Once the job finishes, the output (e.g., an aggregated table) is written to Cloud Storage or directly to BigQuery. If you configured auto-deletion, the cluster shuts down automatically, stopping billing. If not, you manually delete it. The final step is to verify the output data and clear any temporary files (like Spark shuffle directories) that Dataproc may have left in your storage bucket.

## Comparisons

### Dataproc vs Dataflow

**Dataproc:**
- Designed for batch processing using Spark/Hadoop.
- Uses pre-installed Spark and YARN for resource management.
- Best for migrating existing Spark or Hadoop workloads to the cloud.

**Dataflow:**
- Designed for streaming processing using Apache Beam.
- Manages resources itself, no YARN or Spark required.
- Best for real-time, exactly-once, low-latency pipelines.

### Single-job cluster vs Long-running cluster

**Single-job cluster:**
- Created per job, auto-deletes after the job finishes.
- Cost-effective for infrequent or scheduled batch jobs.
- All clusters are identical and ephemeral.

**Long-running cluster:**
- Stays up across multiple jobs (days/weeks).
- Useful for interactive queries or recurring small jobs.
- More costly due to idle time, but faster job startup.

### Cloud Storage vs HDFS (Hadoop Distributed File System)

**Cloud Storage:**
- Persists independently of the cluster; survives deletion.
- Cheaper, with no compute cost for storage.
- Slower for intermediate shuffle data compared to HDFS.

**HDFS (Hadoop Distributed File System):**
- Only exists while the cluster is running; data lost on deletion.
- Compute and storage are coupled, costing money for idle VMs.
- Faster for temporary data within a job because of local disk I/O.

### Preemptible worker nodes vs Standard worker nodes

**Preemptible worker nodes:**
- Up to 80% cheaper per hour.
- Can be terminated at any time by Google (max 24h).
- Suitable for worker nodes in fault-tolerant jobs (Spark).

**Standard worker nodes:**
- Full price, guaranteed to run until job finishes or you stop them.
- Never terminated unexpectedly by Google.
- Required for master node to avoid job failure.

### Apache Spark vs Apache MapReduce (Hadoop)

**Apache Spark:**
- Processes data in memory (RAM) for much faster job execution.
- Supports DataFrames, SQL, and machine learning libraries.
- The primary processing engine used with Dataproc.

**Apache MapReduce (Hadoop):**
- Writes intermediate results to disk at every map/reduce step.
- Only supports map and reduce operations, more verbose code.
- Legacy engine, rarely the best choice for new workloads.

## Diagram

_Flowchart showing how a Dataproc batch job reads data from Cloud Storage, distributes work across master and worker nodes, processes in memory, writes output, and then deletes the cluster._

```mermaid
flowchart TD
    A[Raw Data in Cloud Storage] --> B[Create Dataproc Cluster]
    B --> C[Submit Spark Job]
    C --> D[Master Node: YARN Resource Manager]
    D --> E[Worker Node 1: Spark Executor]
    D --> F[Worker Node 2: Spark Executor]
    D --> G[Worker Node 3: Spark Executor (Preemptible)]
    E --> H[In-memory Processing: Filter, Group, Join]
    F --> H
    G --> H
    H --> I[Final Aggregation]
    I --> J[Output to Cloud Storage / BigQuery]
    J --> K[Delete Cluster (Auto or Manual)]
```

## Common misconceptions

- **Misconception:** Dataproc is just Google's version of Hadoop, and you need to know Hadoop thoroughly for the exam. **Reality:** Dataproc manages both Spark and Hadoop, but the exam focuses almost entirely on Spark because it's the modern standard. You need to understand Spark's architecture (RDDs, DataFrames, actions vs. transformations) more than MapReduce. (Many beginners hear 'Hadoop' and think they need to memorise every Hadoop configuration. The PDE is a cloud data engineering exam, not a Hadoop admin exam.)
- **Misconception:** Using Dataproc means you have to pay for the cluster even when no jobs are running. **Reality:** You only pay for the VMs in the cluster while they are running. You can use single-job clusters that auto-delete after the job ends, or use the 'idle deletion' feature to shut down the cluster after a period of inactivity. (People confuse 'managed service' with 'always on'. They assume the cluster persists like a traditional on-premises server, but cloud is pay-as-you-go.)
- **Misconception:** Dataproc can only process data stored in HDFS, just like on-premises Hadoop. **Reality:** Dataproc natively reads from and writes to Google Cloud Storage via the `gs://` scheme. Cloud Storage is the recommended default because it's cheaper and decouples storage from compute. (This confusion comes from the Hadoop background, where HDFS is the default. But in the cloud, you want storage that survives cluster shutdown.)
- **Misconception:** If a preemptible worker fails, the entire Spark job fails and you need to restart it from scratch. **Reality:** Spark has built-in fault tolerance. If a preemptible worker is reclaimed, Spark re-computes the lost data partitions from the original input or from cached data on other nodes. The job continues, though it may be slower. (Users assume cloud resources are fully reliable. Preemptible instances are designed to be forfeited, and Spark handles this gracefully.)
- **Misconception:** The master node in a Dataproc cluster does the actual data processing. **Reality:** The master node coordinates the job and schedules work. The actual data processing is done by the worker nodes (each running Spark executors). The master only runs the driver program for light orchestration. (The word 'master' sounds like it does all the work, but its role is management, not heavy lifting.)

## Key takeaways

- Dataproc is a managed service that automates the creation, configuration, and deletion of Spark and Hadoop clusters on Google Cloud.
- Batch processing processes a large set of data all at once, typically on a schedule, as opposed to streaming which processes data as it arrives.
- Single-job clusters with auto-deletion are the most cost-effective way to use Dataproc because you only pay for compute while the job runs.
- Preemptible instances can reduce worker node costs by up to 80% but should never be used for the master node because the master must remain available to coordinate the job.
- Cloud Storage is the recommended default storage for input and output data; HDFS should only be used for temporary intermediate data within a job.
- Spark's in-memory processing makes it significantly faster than Hadoop MapReduce for most batch workloads, and the PDE exam focuses on Spark concepts.
- Workflow templates let you chain multiple Dataproc jobs together (e.g., step A, then B, then C) without needing an external scheduler.
- Initialisation actions are scripts that run on every cluster node at startup, allowing you to install custom dependencies or configure the environment.

## FAQ

**Can I use Dataproc for real-time streaming data?**

Yes, Dataproc supports Spark Streaming, but it is not the best choice for true sub-second latency. For streaming workloads, Google recommends Cloud Dataflow (Apache Beam) which handles exactly-once processing better.

**Do I need to install Spark myself on Dataproc?**

No, Dataproc comes with Spark pre-installed. You just specify which version you want when creating the cluster. You can also enable additional optional components like Hadoop, Hive, or Pig.

**What happens to my data if I delete the Dataproc cluster?**

If your data is in Cloud Storage (recommended), it stays safely in the bucket. If you stored data in HDFS on the cluster, it is lost when the cluster is deleted. Always use Cloud Storage for persistent data.

**How does Dataproc fault tolerance work?**

Spark automatically re-computes lost data partitions if a worker fails. The input data is typically stored in Cloud Storage (which is durable), and Spark can re-read missing partitions from there.

**Can I run jobs on Dataproc without writing any code?**

You can run pre-built tools like Apache Hive (SQL-on-Hadoop) or Apache Pig, but you still need to write SQL queries or Pig Latin scripts. For custom logic, you write Spark code (Python, Scala, Java).

**Is Dataproc more expensive than running my own Hadoop cluster on-premises?**

It depends. For intermittent batch jobs, Dataproc is almost always cheaper because you don't pay for idle hardware. For 24/7 usage, on-premises may be cheaper, but you must factor in maintenance and staff costs.

## Check your understanding

1. **What is the main advantage of using a single-job cluster in Dataproc?**

   Answer: It auto-deletes after the job finishes, which minimises cost by ensuring you only pay for the compute time used by the job.

2. **Can you use preemptible instances for the master node in a Dataproc cluster?**

   Answer: No, the master node must be a standard instance because it coordinates the entire job; a preemptible master could be terminated and cause the job to fail.

3. **Which storage option is recommended for long-term data persistence when using Dataproc?**

   Answer: Cloud Storage, because it is durable, cheap, and survives cluster deletion, whereas HDFS data is lost when the cluster is deleted.

4. **A dataproc job processes data once per day. The team wants the cluster to shut down automatically after the job completes. What cluster type should they choose?**

   Answer: A single-job cluster with the auto-deletion flag enabled, or a workflow job that creates and deletes the cluster automatically.

5. **You need to run a Spark job that joins two large datasets. The job is memory-intensive. Should you scale the cluster horizontally (more nodes) or vertically (larger nodes)?**

   Answer: Vertical scaling (larger nodes with more RAM) is more effective for memory-intensive joins because each node needs enough memory to hold its portion of the data.

---

Interactive version with quiz and diagrams: https://courseiva.com/learn/google-pde/processing-batch-data-with-cloud-dataproc