A data engineer needs to design a stream processing pipeline that reads events from Pub/Sub, enriches them with data from a Cloud Storage file, and writes aggregated results to BigQuery. The pipeline must handle late-arriving events up to 1 hour. Which Dataflow feature should be used to manage late data?
A financial services company stream trades into Pub/Sub and processes with Dataflow. The pipeline must ensure exactly-once processing of each trade for regulatory compliance. However, Pub/Sub guarantees at-least-once delivery. Which combination of features should the Dataflow pipeline use to achieve exactly-once semantics?
A company wants to use Cloud Data Fusion to build ETL pipelines. They need to connect to a legacy on-premises database using JDBC and also want to use prebuilt transforms from the Hub. Which two features should they use?
A data engineer is designing a real-time fraud detection system using Dataflow. The system must detect patterns across events from multiple users within a sliding window of 10 minutes. Events arrive on Pub/Sub topics per user. Which approach should they use to join the streams?
A company wants to use Dataprep to clean and transform raw CSV files stored in Cloud Storage before loading into BigQuery. The data quality checks show missing values and inconsistent date formats. Which Dataprep feature should they use to handle these issues?
A retail company uses Dataflow to process real-time clickstream data. They need to enrich each event with customer profile data from Cloud Bigtable and session metadata from Cloud Spanner. Which two Dataflow features should they use?
A company is migrating on-premises Hadoop Hive workloads to Google Cloud. They want to use Dataproc for Spark processing and require a managed Hive metastore that can be shared across multiple Dataproc clusters. Which TWO components should they use?
A data engineer needs to design a BigQuery dataset for a multi-team environment. Each team should have read access only to specific tables, and the data must be protected from accidental deletion. Which THREE steps should they take?
A company wants to design a data pipeline for real-time fraud detection. The system must process streaming financial transactions, enrich them with user profiles from a lookup table, and flag suspicious activities within seconds. Which architecture pattern would be MOST suitable?
You are designing a BigQuery data warehouse for a multi-tenant SaaS application. Each tenant's data must be isolated and queried only by that tenant. You need to minimise management overhead and allow tenants to be added dynamically. Which approach should you use?
You need to process large-scale log files (hundreds of terabytes) using Apache Spark on Google Cloud. The job runs nightly and you want to minimise costs. Which Dataproc cluster configuration is MOST cost-effective?
A data pipeline ingests streaming events into Pub/Sub. You need to guarantee that each event is processed exactly once downstream in Dataflow. Which combination of Pub/Sub and Dataflow configurations should you use?
You are moving an on-premises Hadoop workload to Google Cloud. The workload uses Hive for metadata and HDFS for storage. Which services should you use to minimise reconfiguration?
A company uses BigQuery with partitioned tables by ingestion time. They notice that queries scanning recent partitions are fast but queries scanning older partitions are slow. What is the most likely cause?
You need to run a one-time data transformation job on a small CSV file (100 MB) using a visual, code-free interface. Which Google Cloud service is designed for this?
A company wants to build a real-time dashboard for monitoring application logs. The logs are ingested via Pub/Sub and must be processed with low latency (sub-second). You need to enrich the logs with user metadata from Cloud SQL and store the results in BigQuery for analysis. Which TWO services should be used for the stream processing? (Choose two.)
A data pipeline processes sensitive customer data. You need to ensure that only authorised users can query the data in BigQuery, and that the data is encrypted at rest and in transit. Which THREE steps should you take? (Choose three.)
You are designing a batch data pipeline that runs daily to ingest data from an on-premises database into BigQuery. The ingestion volume is approximately 50 GB per day. The data must be available in BigQuery by 6 AM each day. The on-premises database supports change data capture (CDC) via logs. Which approach minimizes operational cost and complexity?
Your company uses Cloud Data Fusion to build ETL pipelines. You have a pipeline that reads from Cloud Storage, transforms data using a custom Wrangler recipe, and writes to BigQuery. The pipeline is failing with an error indicating that the Wrangler directive is invalid. You have verified the recipe works in the Cloud Data Fusion Studio. What is the most likely cause of the failure?
Your team is migrating a legacy batch processing system that uses Apache Spark on-premises. The migration must be completed with minimal code changes and support both batch and streaming in the future. You want to use a fully managed service. Which Google Cloud service is most appropriate?
You are designing a Dataflow pipeline that reads from Pub/Sub, aggregates events into 10-minute windows, and writes the results to BigQuery. The pipeline must reliably handle late-arriving data (up to 1 hour) and prevent duplicate aggregations. Which combination of pipeline options should you use?
Your organization is designing a data lake on Google Cloud using Cloud Storage. You need to choose a file format for storing raw data that supports schema evolution, is splittable for parallel processing, and is optimized for query performance in BigQuery. Which TWO formats meet these requirements? (Choose 2.)
A data engineer wants to create a BigQuery table that is partitioned by day and clustered by user_id and product_id. Which SQL statement should they use?
A data pipeline uses Dataflow to read from Pub/Sub, window messages into 1-minute fixed windows, and write to BigQuery. The pipeline occasionally has late-arriving data. How should they configure the pipeline to allow late data up to 5 minutes and then trigger a final pane?
A Dataflow pipeline processes a high-volume stream of JSON events. The pipeline has a bottleneck where a ParDo transformation performs an external API call for each element, causing high latency. Which strategy would BEST improve throughput without sacrificing correctness?
A company needs to process data from a legacy system that outputs CSV files daily. They want to visually build transformations without writing code. Which Google Cloud service should they use?
A company is designing a data pipeline using the lambda architecture. They need to process both real-time streams and batch historical data. Which THREE components are essential for a lambda architecture on Google Cloud?
A company needs to process streaming sensor data and run both real-time analytics and batch reanalysis on historical data. They want to minimize infrastructure management. Which architecture and service combination is MOST suitable?
A Dataflow streaming pipeline reads from Pub/Sub, processes events with a fixed window of 1 minute, and writes to BigQuery. Some events arrive late due to network issues. You need to ensure late events are still included in the correct window but the pipeline must not wait indefinitely. What configuration should you use?
A Dataflow pipeline with multiple steps uses a side input from a slowly changing reference table stored in BigQuery. The side input is updated every hour. To avoid reprocessing the entire pipeline on each update, which approach should you use?
You need to transform and clean messy CSV data using a visual interface without writing code. The transformation should be scheduled to run weekly. Which Google Cloud service should you use?
Your team wants to share a BigQuery dataset with another project while ensuring that users from that project can only query specific tables. Which BigQuery feature should you use?
You are designing a data pipeline for a financial services company that requires exactly-once processing semantics. Which TWO services or configurations provide exactly-once guarantees?
A data pipeline is built with Cloud Dataflow that reads from Pub/Sub, applies transformations, and writes to BigQuery. The pipeline is experiencing high latency and occasional data loss during worker failures. The engineer wants to improve reliability and performance. Which two actions should they take?
A company has a BigQuery table that is partitioned by ingestion time and clustered by the 'customer_id' column. They notice that queries filtering on 'customer_id' are not benefiting from clustering as expected. What is the most likely cause?
A data pipeline using Cloud Dataflow reads from a Pub/Sub subscription that has a dead letter topic configured. Some messages are being sent to the dead letter topic. Upon investigation, the engineer finds that the messages contain valid data but are malformed according to the schema. What is the most likely reason for the messages being dead-lettered?
A company is migrating their on-premises Hadoop workloads to Google Cloud. They want to use Dataproc for data processing and need to minimize costs for non-critical batch jobs that can tolerate interruptions. Which TWO configurations should they use?
A data engineering team is designing a streaming pipeline using Cloud Dataflow. They need to join two unbounded PCollections based on a common key. The join must handle late data up to 10 minutes. Which THREE components should they use?
A Dataflow pipeline using Apache Beam processes unbounded data from Pub/Sub. The pipeline uses fixed windows of 1 minute and a trigger that fires early every 30 seconds and at watermark. The team observes that the output pane for window [10:00:00, 10:01:00) contains events with timestamps from 10:00:15 and 10:00:45, but also an event with timestamp 10:02:00. What is the most likely cause?
A company wants to use Pub/Sub Lite to reduce costs for a high-throughput, low-latency streaming pipeline. However, they have a requirement to retain messages for up to 7 days for reprocessing. Which Pub/Sub Lite configuration supports this retention?
A data engineer needs to create a BigQuery table that is optimized for queries that filter on a 'customer_id' column and sort by 'transaction_date'. The table will be used for interactive analysis. Which combination of table features should be used?
A data team is migrating an on-premises Hadoop cluster to Dataproc. The cluster runs a mix of long-running services (Hive, HBase) and transient Spark jobs. They want to minimize cost while maintaining performance. Which TWO strategies should they implement?
A data engineer is designing a streaming pipeline using Dataflow with Apache Beam. The pipeline reads from Pub/Sub, performs a stateful transformation (e.g., session windowing), and writes to BigQuery. The pipeline must handle late data and ensure exactly-once semantics. Which THREE configurations are required?
A company uses Cloud Data Fusion for ETL pipelines. They need to transform sensitive data (PII) by masking certain columns before writing to BigQuery. They also need to ensure the pipeline can be monitored and restarted from failure points. Which THREE features should they use?
A company is designing a data pipeline that ingests real-time events from IoT devices and must handle late-arriving data (up to 1 hour late) while minimizing duplicate processing. They plan to use Dataflow with Pub/Sub. Which combination of windowing and trigger settings should they use?
A financial services company has a BigQuery dataset containing sensitive customer data. They need to share a subset of this data (excluding PII columns) with an external analytics partner. The partner should be able to query the data using their own BigQuery account, but the company must maintain full control over the underlying table and ensure the partner cannot see or access the original table. Which approach should they use?
A data engineering team is designing a streaming pipeline using Dataflow to process real-time clickstream data from a website. They need to aggregate user session metrics (e.g., number of sessions, average duration) every 5 minutes. The pipeline must handle late-arriving events (up to 2 minutes late) and ensure exactly-once processing semantics. Which TWO of the following should they configure? (Choose two.)
A company is migrating their on-premises Hadoop/Spark workloads to Google Cloud. They need a fully managed service that supports existing Spark jobs with minimal code changes, allows autoscaling, and provides integration with Cloud Storage and BigQuery. The team also wants to avoid managing cluster infrastructure and pay only for what they use. Which TWO services meet these requirements? (Choose two.)
A media company ingests 20 TB of raw video metadata files daily into Cloud Storage. Analysts need to run SQL queries on this data with minimal latency, but the files are not partitioned by date and the team wants to avoid managing a cluster. They also want to minimize query costs. Which Google Cloud solution should they use?
A retail company streams point-of-sale transactions into Cloud Pub/Sub. The data engineering team must build a pipeline that enriches each transaction with customer loyalty data stored in Cloud BigQuery, then writes the enriched records to Cloud Bigtable for low-latency lookup by a mobile app. The enrichment must not block the streaming pipeline and must handle BigQuery API quota limits gracefully. Which design should they use?
An analytics team maintains a BigQuery dataset where a daily ELT pipeline truncates and reloads a 2 TB fact table at 02:00 UTC. Analysts report that ad-hoc queries run against the table in the early morning sometimes fail with a 'table not found' error, while the same query succeeds later in the day. The pipeline uses a single CREATE OR REPLACE TABLE statement. Which change should you make to eliminate the query failures while keeping the data fresh?
A media company ingests 4 TB of newline-delimited JSON log files into Cloud Storage every day and needs to query them with standard SQL immediately, without loading them into a native table and without paying for storage twice. Analysts will query only a few fields per record and should not pay for columns they never read. Which BigQuery design should you choose?
A retail analytics team stores point-of-sale transactions in Cloud Storage as newline-delimited JSON, arriving continuously throughout the day, and loads them into BigQuery with a Dataflow streaming pipeline. Analysts complain that ad-hoc queries on the raw landing table are slow and that duplicate transaction IDs appear after pipeline restarts. The team wants the target table to automatically deduplicate on transaction_id and to serve fast aggregate queries without analysts writing MERGE logic. Which BigQuery design should the data engineer implement?
A retail company wants to ingest point-of-sale events into BigQuery with minimal operational overhead. The events arrive continuously and must be queryable within seconds. The team has no existing message broker and wants to avoid managing servers. Which ingestion approach should they choose?
A financial services firm is designing a data processing system on Google Cloud. They need to ingest sensitive customer data from an on-premises Oracle database into BigQuery for analytics. The data must be encrypted at rest with customer-managed encryption keys (CMEK) and must comply with strict audit requirements. The ingestion must occur daily and minimize data movement costs. Which approach should they use?
A retail company ingests point-of-sale events into Pub/Sub and lands them in Cloud Storage using a Dataflow streaming pipeline. Analysts query the raw files with BigQuery external tables, but queries are slow and expensive because the files are small JSON objects. You need to redesign the storage layer so that BigQuery can query the data efficiently while preserving the ability to reprocess raw events. What should you do?
A retail analytics team must run a nightly batch pipeline that reads from BigQuery, applies Python transformations with pandas, and writes results back to BigQuery. The team has no Kubernetes expertise, wants minimal operational overhead, and needs the pipeline to run on a schedule with retries and email alerts on failure. Which Google Cloud service should they use?
You are designing a data pipeline that ingests data from multiple sources (streaming and batch) into BigQuery. You need to ensure that the data is partitioned and clustered for optimal query performance and cost. Which two strategies should you use? (Choose two.)
A financial services firm is designing a Dataflow pipeline that reads from Pub/Sub and writes to BigQuery. The pipeline must handle late-arriving events and produce accurate aggregations per account. The team wants to minimize data loss and duplication. Which two design choices should they make? (Choose two.)
A media company streams video playback events into Pub/Sub. They need to design a Dataflow pipeline that computes session windows per user and writes aggregated results to BigQuery. The pipeline must handle late-arriving events up to 24 hours and must not drop any data. Which two design choices should they make? (Choose two.)
A media company ingests video playback telemetry from a mobile app into Pub/Sub. The analytics team wants the raw events retained in Cloud Storage for compliance for at least three years at the lowest storage cost, with retrieval taking up to a day acceptable. Only a rare audit needs to read the data. Which Cloud Storage configuration should the data engineer use?
A media company needs to process video files uploaded to Cloud Storage. The processing involves transcoding and generating thumbnails, which takes several minutes per file. The pipeline must be triggered automatically when a new file is uploaded, and the processing must be resilient to failures with automatic retries. They want to minimize operational overhead. Which Google Cloud service should they use to orchestrate this workflow?
A retail company ingests point-of-sale transactions into Cloud Storage as newline-delimited JSON files. Analysts need to query the data with standard SQL in BigQuery, but the schema changes frequently as new product attributes are added, and the team wants to avoid managing table schemas manually. They also want to minimize storage cost for data older than 90 days. Which design should they implement?
You are designing a data processing pipeline that ingests application logs from a Compute Engine instance group. The logs must be stored in Cloud Storage and then loaded into BigQuery for analysis. The pipeline should handle failures gracefully and ensure that no data is lost during a temporary outage of the downstream systems. You want to minimize operational overhead. Which Google Cloud service should you use to orchestrate this batch pipeline?
A media company stores raw JSON logs in Cloud Storage and needs to make them queryable in BigQuery with minimal cost. The logs are written once and never modified. Analysts run SQL queries that filter on a few fields and aggregate large time ranges. Which table type should you use?
A media company is designing a data processing system on Google Cloud to analyze video streaming logs. The logs are written as newline-delimited JSON files to a Cloud Storage bucket every hour, and the analytics team needs to run SQL queries on the data with minimal latency. The company wants a serverless solution that requires no cluster management and can automatically scale. Which two Google Cloud services should they use together? (Choose two.)
A logistics company processes shipment events in a Dataflow pipeline that writes enriched records to both BigQuery and Cloud Storage. During a deployment, the pipeline was updated while a streaming job was running, and the team now sees duplicated shipment records in BigQuery. They want future updates to be safe to apply to a running streaming job without reprocessing already-emitted data. What should they do?
A logistics company ingests 4 TB of shipment manifests per day as newline-delimited JSON into Cloud Storage. Analysts run ad-hoc SQL over the entire history (currently 900 TB) but rarely touch records older than 90 days. The team wants to minimize both storage cost and bytes scanned by BigQuery without rewriting the upstream export job. What should they do?
A retail analytics team runs an Apache Hadoop workload on Dataproc that reads and writes shared metadata in a Hive metastore. The team wants the metastore to survive cluster deletion, be shared by multiple ephemeral clusters, and require no HDFS-backed local storage. They also want to keep the metastore reachable by clusters in a different VPC within the same project. Which design should they choose?
A media company streams video playback events into Pub/Sub and processes them with a Dataflow streaming pipeline that writes enriched records to BigQuery. The pipeline must handle sudden traffic spikes without data loss and must produce exactly-once results in the BigQuery output table. Which two design choices should the engineer make? (Choose two.)
A logistics company runs a Dataproc cluster to process daily shipment manifests with Spark. Jobs must complete within a fixed two-hour window, and the team wants to reduce cost without risking the deadline. They observe that the cluster is idle for most of the day and that shuffle-heavy stages spill to disk. Which two changes should the data engineer make to meet the deadline while lowering cost? (Choose two.)
A financial services company needs to process credit card transactions in real-time to detect fraudulent patterns. The transactions are streamed into Pub/Sub. The detection logic requires maintaining state across a sliding window of 5 minutes per credit card number. The results must be written to BigQuery with exactly-once semantics. Which approach should you use with Cloud Dataflow?
A healthcare company needs to process sensitive patient records in a Dataflow pipeline. They must ensure that data is encrypted at rest and in transit, and that only authorized services can access the data. They also need to audit all access to the data. Which combination of Google Cloud features should they implement?
A logistics company runs a Dataflow pipeline that processes shipment events and writes to BigQuery. The pipeline currently uses a single global window and writes each event as a separate row. During peak hours, BigQuery rejects requests with 'rateLimitExceeded'. The team wants to reduce API calls without losing data. What should they do?
A healthcare company is designing a system to process sensitive patient records in BigQuery. They need to ensure that data analysts can only view aggregated, de-identified data, while data engineers can access raw data for pipeline development. They also need to maintain an audit trail of all access. Which two Google Cloud features should they implement? (Choose two.)
A healthcare provider must consolidate patient encounter data from three on-premises systems into BigQuery. Two sources are Oracle databases that change continuously, and one is a set of nightly CSV extracts. Regulators require that no protected health information leave the on-premises network except as encrypted, de-identified records, and the team wants minimal operational overhead for the continuous sources. Which design should the data engineer choose?
A startup needs to run a nightly batch job that reads a 2 TB Avro dataset from Cloud Storage, joins it with a slowly changing dimension in BigQuery, and writes the result back to BigQuery. The team has no Spark or Hadoop expertise and wants to minimize operational effort. Which Google Cloud service should they use?
A logistics company needs to join a high-volume stream of GPS pings from 20,000 vehicles with a slowly changing vehicle registry table of about 50,000 rows that is updated several times a day. The pipeline runs on Dataflow and must reflect registry changes within minutes without restarting the job. Which design should the data engineer implement?
A financial services firm must process payment events with strict ordering per customer and exactly-once results, while allowing the pipeline to be replayed from a known point after a bug fix. Events arrive in Pub/Sub and are written to BigQuery. Which design should the data engineer implement in Dataflow?
A media company processes viewer telemetry that must be retained for seven years for regulatory reasons, but only the most recent 90 days are queried frequently. Queries older than 90 days are rare and can tolerate higher latency. The data volume is several petabytes and growing. The team wants to minimize storage cost while keeping the recent data fast to query. (Choose two.)
A financial services company runs a batch ETL pipeline on Dataproc. The pipeline reads from Cloud Storage, transforms data with Spark, and writes to BigQuery. The nightly job currently takes 6 hours, and the team wants to reduce wall-clock time without changing the Spark code. The cluster is configured with a fixed number of on-demand worker nodes. Which change should the data engineer make to improve performance?
A healthcare analytics company ingests HL7 messages into Cloud Pub/Sub and processes them with Dataflow before writing to BigQuery. The pipeline must ensure that each patient's records are processed in the order they were received and that duplicates are eliminated. The messages contain a patient ID and a sequence number. Which Dataflow feature should the engineer use to meet these requirements?
A company uses Dataproc to run daily Spark ML jobs. The jobs run for 2 hours each day. The team wants to reduce costs without changing job characteristics. Which strategy is MOST cost-effective?
A data engineer needs to create a BigQuery table that is partitioned by ingestion time and clustered by customer_id and transaction_date. They also want to limit access so that only users from a specific domain can query the table. Which approach should they use?
A startup needs a fully managed, serverless Spark service to run occasional data processing jobs without managing clusters. They want to pay only for the resources used during job execution. Which Google Cloud service should they use?
A company uses Pub/Sub with push subscriptions to deliver events to a Cloud Run service. Recently, the service has been returning HTTP 429 (Too Many Requests), causing messages to be retried and eventually sent to the dead letter topic. What is the MOST likely cause?
A data engineer needs to process data in a Dataflow pipeline that reads from a Pub/Sub topic. The pipeline must group events into 5-minute windows and compute the average value per key. Which Beam transform should they use after windowing?
A company uses BigQuery for analytics. They have a table that is queried frequently by date range. To reduce costs, they want to ensure queries only scan the relevant partitions. They also want to improve performance for queries filtering on a specific customer_id. Which table design should they use?
A company needs a messaging service for event-driven applications that require low cost for high-throughput, but can tolerate occasional message loss. Which Pub/Sub product should they choose?
You are designing a Dataflow pipeline that reads from Pub/Sub and writes to BigQuery. The pipeline must handle late-arriving data (up to 1 hour) and group events into 10-minute windows. Which configuration is correct?
Which Google Cloud service provides a visual interface for building ETL pipelines using a drag-and-drop design and includes pre-built transforms from a marketplace?
You are designing a streaming pipeline that needs to handle sudden spikes in traffic without losing data. The pipeline uses Pub/Sub and Dataflow. Which configuration ensures data is not lost if Dataflow falls behind?
You need to analyse streaming data from thousands of IoT devices, each sending temperature readings every second. You want to calculate the average temperature per device over the last 5 minutes, updating every minute. Which windowing strategy should you use in Dataflow?
You are designing a Dataflow pipeline for processing real-time clickstream data. The pipeline must group events into 30-second windows and handle late data up to 5 minutes. You want to output partial results every 10 seconds for low-latency monitoring. Which THREE configurations should you use? (Choose three.)
Your data engineering team needs to process a continuous stream of clickstream events from a website and update a real-time dashboard showing user activity over the last hour. The pipeline should have minimal operational overhead and support exactly-once processing semantics. Which Google Cloud service should you use?
Your company ingests millions of events per second into a Pub/Sub topic. The downstream consumer must process events with minimal latency and high throughput. However, the consumer occasionally falls behind during traffic spikes, and you need to ensure no data loss while minimizing costs. Which subscription type and configuration should you choose?
You are designing a data pipeline that processes streaming events with late-arriving data (up to 2 hours late). The pipeline must compute hourly aggregations and emit results as soon as possible, but must also accurately update results when late data arrives. You want to minimize overall processing cost. Which Dataflow windowing and trigger configuration should you use?
You are migrating on-premises Hadoop jobs to Google Cloud. The existing jobs use Spark for ETL and Hive for querying. You want to minimize changes to the existing code and maintain the ability to use Hive queries with the same metastore across multiple clusters. Which service combination should you use?
You have a BigQuery table that is partitioned by ingestion time and clustered on user_id. The table stores event logs and is queried frequently by user_id to analyze user behavior over the last 30 days. Queries are still scanning too many partitions. Which optimization should you apply first?
You need to allow a data analyst to run queries on a BigQuery dataset but prevent them from modifying the data or deleting the dataset. Which IAM role should you grant?
You need to create a BigQuery table that stores customer transaction data. The table will be queried frequently by a customer_id column to retrieve recent transactions (last 30 days). Which table design optimizes query performance and cost?
Your company is building a real-time anomaly detection system for financial transactions. The system must process streams of transactions and flag anomalies within seconds. The volume is moderate (5000 transactions per second). You want a fully managed solution that integrates with BigQuery for historical analysis. Which service should you use for stream processing?
Your company runs a Dataflow streaming pipeline that processes user activity from Pub/Sub and writes aggregated results to BigQuery. Lately, the pipeline is experiencing high latency and backlog growth during peak hours. You need to troubleshoot and improve performance. Which THREE actions should you take? (Choose 3.)
Your team is using Cloud Dataprep to clean and transform a dataset. Which TWO features of Cloud Dataprep help you understand data quality issues before running the pipeline? (Choose 2.)
A company needs to process streaming sensor data from millions of devices with sub-second latency, apply transformations, and write results to BigQuery for real-time dashboards. The data volume varies, and they want to avoid managing servers. Which service should they use?
A company uses Cloud Pub/Sub to ingest events from multiple sources. They need to guarantee that each event is processed exactly once by downstream consumers. However, Pub/Sub guarantees at-least-once delivery. Which additional steps should they implement to achieve exactly-once processing?
A team wants to use Cloud Pub/Sub Lite for a high-throughput, low-cost messaging system. They need exactly-once delivery to subscribers. What should they know about Pub/Sub Lite's delivery guarantees?
A company wants to use BigQuery materialized views to accelerate queries on a table that is updated every hour. Which statement about materialized views is true?
A company wants to use Cloud Data Fusion for ETL pipelines. They need to integrate with custom transformations not available in the marketplace. What should they do?
A Dataproc cluster uses preemptible worker nodes to reduce costs. The cluster runs a long-running Spark job that occasionally experiences worker failures. How should the job be configured to handle preemptible worker failures gracefully?
A company uses Cloud Pub/Sub for event ingestion. They want to ensure that if a subscriber fails to process a message after 5 attempts, the message is sent to a dead letter topic for analysis. Which TWO configurations are needed?
You are designing a BigQuery data warehouse for a retail company. Queries frequently filter on order_date and customer_id. To optimize query performance and cost, which table design should you use?
You need to process a large Spark ML training job on a Dataproc cluster. The job is fault-tolerant and can handle occasional node failures. To reduce costs, which type of worker nodes should you use?
Your company uses Pub/Sub to ingest clickstream data. Messages must be processed in order for the same user_id. How should you configure the Pub/Sub subscription to guarantee ordering?
A company uses Dataproc Serverless for Spark batch jobs. They notice that some jobs are failing due to out-of-memory (OOM) errors. Which configuration parameter should they adjust to allocate more memory per executor?
You are building a real-time fraud detection system using Dataflow. Events from Pub/Sub need to be grouped by user_id within a 5-minute window to detect suspicious patterns. Some events may be delayed by up to 2 minutes. How should you configure the window and trigger to balance accuracy and latency?
You need to choose a messaging service for a real-time streaming application that requires low cost and can tolerate occasional message loss. Which service is MOST suitable?
A data engineer needs to run an existing Spark job on Google Cloud with minimal code changes. The job requires Hive metastore access. Which Dataproc feature should they use to provide a managed Hive metastore?
A media company processes video metadata using a Dataflow pipeline. They need to join two streaming sources: user activity (Pub/Sub) and video catalog updates (Pub/Sub). Which THREE transforms should be used in the pipeline?
You are designing a BigQuery data lake for a healthcare organization. The data includes patient records that must be access-controlled at the row level. Which TWO features should you use to meet this requirement?
A data engineer needs to process streaming data from thousands of IoT devices and generate real-time dashboards. The data volume is low but requires exactly-once processing semantics. Which Google Cloud service combination should they use?
A company has a BigQuery dataset containing sensitive customer data. They want to share a subset of this data with external partners, ensuring that partners can only see specific columns and rows. Which BigQuery feature should they use?
An organization runs periodic Apache Spark jobs on Dataproc to process data from Cloud Storage. They want to reduce costs by using preemptible instances for worker nodes. What is a key consideration when using preemptible instances in Dataproc?
A company needs to process high-throughput streaming data with low latency. They are considering Cloud Pub/Sub for ingestion and Cloud Dataflow for processing. However, they are concerned about cost. Which alternative to Cloud Pub/Sub would reduce costs while still meeting the throughput requirements?
A data engineer is designing a pipeline that reads from Cloud Pub/Sub, aggregates events into 5-minute windows, and writes the results to BigQuery. The engineer wants to ensure that late-arriving data (up to 2 minutes late) is included in the correct window. Which Dataflow feature should they configure?
A company is using Cloud Storage to store raw logs. They want to use Cloud Data Fusion to transform and load the data into BigQuery on a daily schedule. The transformations are complex and involve joining multiple datasets. What is the most efficient way to run these pipelines?
An organization is implementing a data lake on Google Cloud using Cloud Storage. They need to process both batch and streaming data with a unified pipeline. The team has experience with Apache Beam. Which architecture should they use to minimize operational overhead?
A company uses Cloud Dataproc to run Spark ML training jobs. They want to persist the trained models and metadata in a Hive-compatible metastore. Which Dataproc feature should they use?
An engineer needs to create a Pub/Sub subscription that sends messages to an HTTPS endpoint. The endpoint must be able to acknowledge messages individually. Which type of subscription should they use?
An organization is using BigQuery for analytics. They have a table that is 500 GB and is frequently queried by 'date' and 'region'. They want to optimize query performance and reduce costs. Which TWO actions should they take?
A data pipeline ingests streaming events into Pub/Sub and needs to join them with a slowly updating reference table (few thousand rows) from a Cloud Storage CSV file. The pipeline runs on Dataflow with Apache Beam. Which approach is most cost-effective and operationally simple?
A developer wants to create a BigQuery table that automatically expires data older than 30 days to reduce storage costs. Which table design feature should be used?
A company runs Apache Spark jobs on Dataproc. They want to reduce costs by using preemptible instances for worker nodes. The jobs are fault-tolerant and can handle occasional node loss. However, the cluster must remain available for interactive querying during business hours. Which Dataproc cluster configuration meets these requirements?
A data engineer needs to design a streaming pipeline that ingests events from multiple sources, enriches them with a lookup table stored in BigQuery (updated every hour), and writes the results to a BigQuery table for real-time dashboards. The pipeline must handle late-arriving data up to 1 hour. Which Dataflow feature should be configured to manage late data?
A company is using Pub/Sub to ingest clickstream events. They need to ensure that events are delivered to a subscriber at least once, but duplicates can be tolerated. They also need to filter events by type before processing. Which subscription configuration should be used?
A data pipeline uses Cloud Data Fusion to perform ETL jobs. The pipeline reads from BigQuery, transforms data using Wrangler, and writes to Cloud Storage. The team notices that the pipeline runs slower than expected. They suspect the Data Fusion instance is under-provisioned. Which action should be taken to improve performance?
A company uses Pub/Sub to ingest events from multiple sources. They need to ensure that messages from a specific source are processed in order (per source partition). They also need to deduplicate messages. Which TWO features should they use?
A company is evaluating BigQuery for a data warehouse migration. They have a mix of reporting queries and ad-hoc analytical queries. They want to control query costs and prevent runaway queries. Which THREE strategies should they implement?
A data team is building a near-real-time dashboard that displays aggregated metrics from Kafka topics. They want to use Pub/Sub as a managed messaging service and Dataflow for stream processing. They need to ingest data from Kafka into Pub/Sub with minimal custom code. Which THREE Google Cloud services should they use together? (Choose three.)
A retail company ingests point-of-sale clickstream events into Cloud Pub/Sub at roughly 200,000 messages per second during flash sales. Analysts need near-real-time dashboards that aggregate revenue by product category over sliding 5-minute windows, with results visible in BigQuery within 30 seconds of the event. The pipeline must handle occasional bursts up to 3x the normal rate without dropping messages, and the team wants to minimize operational overhead. Which design should the data engineer use?
A startup wants to analyze user clickstream data stored in Cloud Storage in Parquet format. They need to run ad-hoc SQL queries without managing any servers and want to pay only for the queries they run. Which Google Cloud service should they use?
A retail company ingests point-of-sale events from thousands of stores into Cloud Pub/Sub. They need to process these events in a streaming Dataflow pipeline that enriches each event with store metadata from a slowly changing BigQuery table. The enrichment table is updated only once per day. The pipeline must minimize latency and avoid querying BigQuery for every event. Which approach should they use?
Your team is designing a data processing system that ingests JSON messages from millions of IoT devices. The ingestion rate is highly variable, with spikes up to 500,000 messages per second. You need a fully managed, serverless messaging service that can buffer messages and decouple producers from consumers. Which Google Cloud service should you choose?
A financial services firm needs to design a batch processing system on Google Cloud to analyze large volumes of historical transaction data stored in Cloud Storage. The data is in Parquet format and must be processed using Apache Spark. The firm wants to minimize operational overhead and only pay for the resources used during job execution. Which Google Cloud service should they use?
A financial services company needs to process credit card transactions in real time to detect fraudulent patterns. The pipeline must handle late-arriving data (up to 2 hours) and produce accurate results. They want to use a unified programming model that works for both batch and streaming. Which Google Cloud service should they use?
A healthcare company is designing a system to ingest HL7 messages from multiple hospitals into Google Cloud. The messages must be processed in near real-time to extract patient vitals and trigger alerts if thresholds are exceeded. The system must guarantee that no messages are lost and that processing is exactly-once. Which combination of Google Cloud services should they use?
You need to store petabytes of data in a data warehouse that supports ANSI SQL, automatic scaling, and real-time analytics. The data is primarily used for ad-hoc queries and business intelligence. Which Google Cloud service should you use?
A media company is designing a data processing system on Google Cloud to analyze video streaming logs. The logs are generated continuously and stored in Cloud Storage. The company wants to use Cloud Dataflow to process these logs, but they need to ensure the pipeline can handle late-arriving data and provide accurate results for both real-time dashboards and historical analysis. Which two features of Cloud Dataflow should they use? (Choose two.)
A retail company runs a nightly batch pipeline that loads point-of-sale transactions into BigQuery. The pipeline uses a Cloud Composer (Apache Airflow) DAG with a BigQueryInsertJobOperator task that runs a SQL MERGE statement to upsert yesterday's sales into a large fact_sales table partitioned by transaction_date. The data engineering team notices that the MERGE task occasionally fails with a 'Resources exceeded during query execution' error when the source staging table contains more than 50 million rows. They need to redesign the DAG to reliably load large daily volumes without changing the final table schema or downstream dashboards. What should they do?
Your company is migrating an on-premises Apache Hadoop cluster to Google Cloud. The cluster runs Hive for SQL-like queries and stores data in HDFS. You want a managed service that minimizes operational overhead while supporting existing Hive scripts. Which Google Cloud service should you choose?
A logistics company wants to optimize their delivery routes using historical GPS data. The data is stored in BigQuery and is updated daily. They need to run a complex machine learning model that requires iterative processing over the entire dataset using Apache Spark. The model training takes several hours and must be run weekly. They want to minimize cost and operational overhead. Which approach should they take?
A financial services firm is designing a Dataflow pipeline that reads from Pub/Sub and writes enriched transactions to BigQuery. The pipeline must guarantee exactly-once processing semantics for the BigQuery sink, even during pipeline updates and worker restarts. The team plans to use the Apache Beam Java SDK with the BigQueryIO connector. Which combination of configurations should they use?
A financial services firm runs a batch risk calculation on Dataproc. The job reads from Cloud Storage, processes data in memory, and writes results to BigQuery. The job must complete within a 2-hour window each night, and the cluster must be shut down automatically after completion to minimize cost. You want to orchestrate this with minimal operational overhead. What should you do?
A financial services firm runs a Dataflow batch pipeline that joins a 2 TB transaction dataset with a 40 GB customer reference dataset. The reference data changes only once per day and is currently read from a BigQuery table with a side-input transform on every element. Job cost is dominated by repeated BigQuery reads, and the pipeline occasionally hits quota errors. The team wants to minimize cost and quota pressure while keeping the daily refresh. What should the data engineer change?
A healthcare analytics team needs to run a series of SQL transformations on data stored in BigQuery. The transformations must run on a schedule, and the team wants to minimize operational overhead by using a fully managed service that integrates with BigQuery and Cloud Logging. They also need to parameterize the SQL queries with runtime values such as the current date. Which Google Cloud service should they use?
A startup is building a data lake on Google Cloud. They need to store raw JSON logs in a cost-effective manner and later query them using SQL with minimal transformation. The logs are infrequently accessed but must be retained for 7 years for compliance. Which storage solution should they use?
A financial services company must analyze transaction data that includes customers' full names, account numbers, and home addresses. Regulations require that this personally identifiable information (PII) never be stored in raw form in their BigQuery analytics warehouse. The data engineering team plans to use Cloud Dataflow to read from a Pub/Sub topic and write to BigQuery. Which approach best satisfies the regulatory requirement while keeping the pipeline simple?
A financial services firm is designing a data processing system on Google Cloud that must ingest change data capture (CDC) streams from an on-premises PostgreSQL database into BigQuery with sub-minute latency, preserve the ordering of changes per primary key, and apply updates and deletes so that BigQuery reflects the current state of each row. The source database cannot be modified to add triggers. Which two design elements should you include? (Choose two.)
A logistics company ingests GPS telemetry from delivery vehicles into Pub/Sub. They need to process the stream in Dataflow to calculate real-time estimated arrival times (ETAs). The pipeline must handle late-arriving data up to 2 hours and must emit results every 5 minutes. The team wants to use Apache Beam's windowing and triggering. Which windowing strategy and trigger should they use to meet these requirements?
A retail company runs a batch pipeline in Cloud Dataflow that reads from Cloud Storage and writes to BigQuery. The pipeline uses a GroupByKey operation on a key that is heavily skewed: one customer ID accounts for 40% of all transactions. This causes a single worker to process a massive amount of data, and the job takes hours longer than expected. The team wants to reduce the skew without changing the pipeline's logic or output. What should they do?
An online gaming company runs a Dataflow streaming pipeline that aggregates player actions into per-session metrics. Sessions are defined by a gap duration of 30 minutes of inactivity, and the pipeline must emit final session results even when a session's events arrive out of order by up to 10 minutes. Late data beyond that window can be dropped. Which combination of Beam concepts should the pipeline use?
A startup wants to build a data lake on Google Cloud to store raw JSON, CSV, and Parquet files from various sources. They need a storage solution that is highly durable, globally accessible, and integrates natively with BigQuery and Dataproc. They want to minimize management overhead. Which Google Cloud service should they use?
A startup is building a data lake on Google Cloud. They need to store raw JSON, CSV, and Parquet files from various sources. The files will be accessed by multiple analytics tools, including BigQuery and Dataproc. The startup wants a cost-effective, durable, and highly available storage solution that integrates natively with these services. Which Google Cloud service should they use?
A financial services firm ingests trade events into Cloud Storage and must load them into BigQuery. Compliance requires that each event be processed exactly once and that the load be idempotent across retries, even if a Dataflow job restarts mid-batch. The destination table must also be queryable immediately after each successful load. Which loading approach best satisfies these requirements?
A retail analytics team must move 30 TB of Parquet files from an on-premises Hadoop cluster into BigQuery once, then run standard SQL dashboards. The transfer window is 48 hours and the source cluster has limited outbound bandwidth. Which approach should the data engineer choose?
A logistics company uses Cloud Dataflow to process a continuous stream of GPS events from delivery trucks. The pipeline must compute the distance traveled per truck per hour and write the results to BigQuery. The events are keyed by truck ID, and the pipeline uses windowing with a one-hour fixed window. The team notices that some trucks report events with timestamps that are several minutes late due to network delays. They want to ensure that late events are still included in the correct window and that the results are emitted only after a reasonable wait. What should they configure?
Your team is migrating an on-premises Apache Hadoop cluster to Google Cloud. The cluster runs MapReduce jobs that read and write data to HDFS. You want to minimize code changes and operational overhead. Which Google Cloud service should you use to run these jobs?
A media company streams playback telemetry through Pub/Sub into a Dataflow pipeline that writes to BigQuery. During prime-time peaks, the pipeline's BigQuery write step shows growing latency and the job repeatedly reports that it is backing off on insert retries. The team wants to reduce write pressure without changing the downstream table schema or losing exactly-once semantics. What should they do?
You need to design a data processing system that ingests streaming data from thousands of IoT devices. The data must be processed in real-time to calculate average temperature per device over 1-minute intervals, and the results should be stored in BigQuery for analysis. You want a serverless solution with minimal management. Which combination of Google Cloud services should you use?
You are designing a data processing architecture on Google Cloud. You need to ingest data from multiple sources, including streaming events and batch files, and process them to produce a unified dataset for analytics. The solution must support both real-time and historical processing with the same codebase, and be able to handle late-arriving data. Which two Google Cloud services should you use together to achieve this? (Choose two.)
A financial services firm must process payment events in strict order per account and cannot tolerate duplicates. The events arrive in Pub/Sub and must be written to BigQuery. The engineering team is designing the pipeline and wants to guarantee that each account's events are applied in the order they were published. Which approach should they take?
A healthcare analytics group must build a pipeline that ingests HL7 messages from an on-premises interface engine, must retain raw messages for seven years for compliance, and must expose de-identified aggregates to analysts. The security team requires that protected health information never be written to a dataset analysts can query, and that all data be encrypted with keys the organization manages and can revoke. Which two design choices satisfy these requirements? (Choose two.)
A logistics company collects GPS pings from delivery vehicles into Pub/Sub and needs to compute the distance traveled per vehicle per hour. The data volume is high and bursty, and the company wants a managed service that automatically scales the number of workers based on load while allowing custom windowing and stateful processing. Which service should they use?
A small startup wants to run a nightly batch job that transforms a 2 GB CSV file in Cloud Storage and writes the result back as Parquet. The team has no cluster administration experience, wants per-job pricing rather than an always-on cluster, and needs the transformation to run in under thirty minutes. Which Google Cloud approach should they choose?
A retail company is designing a Dataflow pipeline to process point-of-sale transactions from Cloud Pub/Sub and write to BigQuery. The pipeline must handle late-arriving data up to 24 hours and ensure that all data is written to BigQuery exactly once, even in the event of worker failures. Which two features should the engineer implement to meet these requirements? (Choose two.)
A media company needs to process a large number of small JSON files stored in Cloud Storage. They want to use a serverless, SQL-based approach to transform and aggregate the data without managing infrastructure. Which Google Cloud service should they use?
A logistics company uses Cloud Pub/Sub to ingest shipment tracking events. They want to archive all events to Cloud Storage for long-term retention and also process them in real time with Dataflow. The events are published to a single topic. Which design should the data engineer use to ensure both archiving and real-time processing without data loss?