Cloud Dataproc for Spark and Hadoop

Cloud Dataproc is a fully managed, cloud-native service for running Apache Spark, Apache Hadoop, and associated open-source data processing frameworks. It automates the provisioning, scaling, and management of clusters so you can focus on running data pipelines without worrying about infrastructure. Dataproc is designed for speed: clusters can be created in under 90 seconds, and you only pay for the resources you use (per-second billing after a 1-minute minimum). It integrates deeply with Google Cloud Storage (GCS), BigQuery, Cloud Monitoring, and Cloud Logging.

On-premises or self-managed Hadoop/Spark clusters require significant operational overhead: hardware procurement, software installation, configuration, tuning, patching, and capacity planning. Dataproc eliminates this by providing a turnkey environment. It is ideal for: - ETL pipelines – transforming large datasets using Spark or Hive. - Machine learning – training models with Spark MLlib. - Data migration – moving on-premises Hadoop workloads to GCP. - Ad-hoc analysis – running interactive queries on large datasets.

Dataproc uses a master-worker architecture. The master node runs YARN ResourceManager, HDFS NameNode (if HDFS is enabled), and Spark driver. Worker nodes run YARN NodeManager, HDFS DataNode, and Spark executors. When you create a cluster, Dataproc automatically: 1. Provisions Compute Engine VMs (master and workers). 2. Installs and configures Hadoop, Spark, and optional components (Hive, Pig, Tez, etc.). 3. Configures network firewalls and IAM permissions. 4. Sets up monitoring with Cloud Monitoring and Logging.

Create a cluster using gcloud:

gcloud dataproc clusters create my-cluster \
    --region=us-central1 \
    --zone=us-central1-a \
    --master-machine-type=n1-standard-4 \
    --worker-machine-type=n1-standard-4 \
    --num-workers=3 \
    --image-version=2.0-debian10 \
    --optional-components=ANACONDA,JUPYTER \
    --enable-component-gateway

Submit a Spark job:

gcloud dataproc jobs submit spark \
    --cluster=my-cluster \
    --region=us-central1 \
    --class=org.apache.spark.examples.SparkPi \
    --jars=file:///usr/lib/spark/examples/jars/spark-examples.jar \
    -- 1000

List clusters:

gcloud dataproc clusters list --region=us-central1

Update cluster (add workers):

gcloud dataproc clusters update my-cluster \
    --region=us-central1 \
    --num-workers=5

Dataproc autoscaling uses a primary worker group and an optional secondary worker group (preemptible). Autoscaling policies define: - Scale-in/out thresholds – e.g., scale out when YARN memory utilization > 80% for 2 minutes. - Cooldown period – default 2 minutes between scaling actions. - Max instances – limit on total workers.

Dataproc clusters can be deleted automatically via: - Idle timeout: Auto-delete after N minutes of inactivity (no jobs running). - Cluster scheduling: Use Cloud Scheduler to delete clusters at specific times. - Workflow templates: Define a sequence of jobs; the cluster can be deleted after the workflow completes.

Common issues: - Out of memory: Increase worker machine type or add more workers. - Job failure: Check logs in Cloud Logging or use gcloud dataproc jobs wait. - Preemptible worker loss: Ensure job is fault-tolerant (e.g., use checkpointing).

Enterprise Scenario 1: ETL Pipeline for Retail Analytics A large retailer processes daily sales data from thousands of stores. They use Dataproc to run a 10-node Spark cluster (n1-standard-8 workers) that reads raw CSV files from Cloud Storage, cleans and transforms the data using Spark SQL, and writes the results to BigQuery for dashboarding. The cluster is created via a Cloud Composer DAG that runs daily at 2 AM. After the job finishes, the cluster is deleted using an idle timeout of 5 minutes. The retailer saves costs by using preemptible instances for 50% of workers, as the job is fault-tolerant with checkpointing to GCS. A common misconfiguration is not setting the idle timeout, leading to the cluster running all day and incurring unnecessary costs.

Enterprise Scenario 2: Ad-Hoc Data Science Exploration A data science team needs to explore a 5 TB dataset stored in GCS. They create a Dataproc cluster with Jupyter and Anaconda components enabled via --optional-components. They use the Component Gateway to access JupyterLab. The cluster has 5 preemptible workers for cost savings. However, during a long-running hyperparameter tuning job, preemptible VMs are frequently terminated, causing job failures. The solution: switch to standard workers or implement checkpointing in Spark. The team learns that preemptible instances are best for batch jobs that can tolerate interruptions.

Enterprise Scenario 3: Migrating On-Premises Hadoop to GCP A financial services company migrates its on-premises Hadoop cluster to GCP. They use Dataproc with HDFS enabled (temporarily) to replicate data. They run MapReduce jobs on Dataproc to validate data integrity. They then transition to using GCS as the primary data store, removing HDFS. The migration involves using gcloud dataproc clusters create with the same Hadoop version as on-premises. A pitfall is forgetting to set the --bucket flag for staging, causing job failures due to missing dependencies. They also use initialization actions to install custom security certificates.