A data analytics team runs Apache Spark jobs to process large datasets. They need a managed cluster that provisions quickly, scales dynamically, and integrates with Cloud Storage and BigQuery. Which service should they use?
Cloud Dataproc is the fully managed Apache Spark and Hadoop service on Google Cloud, purpose-built to run Spark jobs at scale. It offers fast cluster provisioning via ephemeral clusters, direct connectors to Cloud Storage and BigQuery for reading and writing data without ETL, and automatically manages the HDFS/YARN infrastructure. Because it is native to Spark and integrates with the Google Cloud ecosystem, it is the correct choice for executing Spark workloads.
Why this answer
Cloud Dataproc is the correct choice because it is a managed Spark and Hadoop service that provisions clusters in under 90 seconds, supports autoscaling, and natively integrates with Cloud Storage (via the gs:// connector) and BigQuery (via the BigQuery Storage API and Spark BigQuery connector). This makes it ideal for teams needing fast, dynamic, and integrated Spark job execution.
Exam trap
The trap here is that candidates confuse Cloud Dataflow (a Beam-based service) with a managed Spark service, or assume Cloud Run can handle dynamic Spark cluster scaling, when in fact only Cloud Dataproc provides the native Spark runtime and auto-scaling cluster management required for this use case.
How to eliminate wrong answers
Option A is wrong because Cloud Dataflow is a unified stream and batch data processing service based on Apache Beam, not Apache Spark, and it does not provide a managed Spark cluster. Option C is wrong because Cloud Composer is a managed Apache Airflow workflow orchestration service, not a compute engine for running Spark jobs; it can trigger Dataproc jobs but does not run Spark itself. Option D is wrong because Cloud Run is a serverless container platform that does not support dynamic cluster scaling for Spark workloads and lacks native integration with Cloud Storage and BigQuery for Spark; running a custom Spark container on Cloud Run would require manual cluster management and does not provide the managed, auto-scaling Spark environment that Dataproc offers.