CCNA Designing Data Processing Systems Questions

Dataproc Serverless for Spark can read data directly from BigQuery using the BigQuery connector, eliminating the need to export data. It is a serverless solution, so it automatically provisions and scales resources, and charges only for the duration of the job. This minimizes both cost and operational overhead, as there is no cluster to manage. It supports complex Spark MLlib models and iterative processing, making it ideal for this scenario.

Cloud Run scales based on requests; if max instances reached, it returns 429. Increasing instances or adjusting concurrency resolves this.

Session windows merge based on inactivity gaps, so a 30-minute gap duration models the session definition exactly. Setting allowed lateness to 10 minutes keeps window state alive so out-of-order events within that bound are still incorporated, and firing on the watermark produces final results once the window closes, which matches the emit-final-results requirement.

Clustering by region sorts data within partitions by that column, so filters on region read only relevant blocks rather than scanning everything. This satisfies the stem's region-filtering access pattern, reducing bytes billed and improving performance.

Secondary workers (preemptible workers) are ideal for fault-tolerant batch jobs. They do not store HDFS data, so losing them does not affect data durability. The cluster remains available because primary workers and master nodes are regular instances.

Exactly-once delivery on the subscription removes duplicate messages by tracking acknowledgement state, satisfying the deduplication requirement. However, it does not guarantee ordering; that needs the ordering key or message ordering property set on the publisher side, so this feature alone covers only one of the two stated constraints.

Transfer Appliance is purpose-built for bulk offline migration when network bandwidth makes online transfer impractical. Shipping encrypted appliances avoids the limited outbound link entirely and comfortably moves 30 TB within the window. After Google uploads the appliance contents to Cloud Storage, a BigQuery load job reads the Parquet objects into native tables, after which dashboards query managed storage with full columnar performance.

Performing de-identification in Dataflow before the data reaches BigQuery ensures PHI never enters an analyst-queryable dataset, satisfying the hard boundary. Writing aggregates to a distinct dataset protected by CMEK gives the organization revocable key control, and policy tags add fine-grained access restrictions. Analysts are granted access only to this curated dataset, keeping the raw zone entirely separate.

ParDo is needed to process each element and extract the common key (e.g., video_id) before joining.

Row access policies enforce row-level filtering directly in BigQuery by attaching a filter expression to a table, so each user sees only permitted patient records. This satisfies the stem's row-level access-control constraint natively, without duplicating data or maintaining separate views per role.

Ingestion-time partitioning groups rows by load time, so a 30-day event query scans every partition loaded in that period regardless of event dates. Partitioning on a DATE derived from event_timestamp enables partition pruning to only the relevant event dates, while clustering on user_id still accelerates per-user filtering.

Pub/Sub delivers at-least-once, so the same message can arrive repeatedly. Persisting each message's unique ID and discarding already-seen IDs gives idempotent deduplication at the consumer, converting at-least-once delivery into effectively exactly-once processing as the stem requires.

The Storage Write API supports an exactly-once stream type that uses offsets to deduplicate on retry, preserving exactly-once semantics while allowing high-throughput batched appends. Batching amortizes request overhead and dramatically lowers the per-row cost and quota pressure that caused the backoff. Because it writes to the same table schema, no downstream change is required, and it is the recommended high-volume ingestion path for Dataflow into BigQuery.

Allowed lateness keeps each 5-minute window's state alive for 2 minutes past the watermark, and a late-data trigger emits updated results when those events arrive. This ensures events up to 2 minutes late land in the correct window rather than being dropped.

Fixed one-hour windows with two hours of allowed lateness retain late events, while early periodic triggers plus a watermark trigger emit results promptly and accumulating panes revise prior output. This satisfies both low-latency emission and accurate late-data correction without over-provisioning resources.

Cloud Dataflow provides fully managed, autoscaling stream processing with exactly-once semantics and low-latency windowing, satisfying the sub-second anomaly flagging requirement. Its native BigQuery sink connector streams results directly into BigQuery for historical analysis, meeting the integration constraint without custom code or cluster management.

STREAMING_INSERTS with a deterministic insertId enables BigQuery to deduplicate retried inserts, providing exactly-once semantics. RETRY_ALWAYS ensures that transient failures are retried, and because the insertId is stable, duplicates are suppressed. This combination is the recommended way to achieve exactly-once writes to BigQuery from a streaming Dataflow pipeline, even across worker restarts.

Triggers in Dataflow control when to emit aggregated results for a window. Using triggers, such as early triggers or late triggers, allows the pipeline to produce speculative results before the window closes and then update them as late data arrives. This enables real-time dashboards to show near-instantaneous insights while still refining results for historical accuracy. Triggers are a key mechanism for balancing latency and completeness in streaming pipelines.

BigQuery scheduled queries are a fully managed feature that runs SQL on a schedule, supports parameterization with @run_date and other system variables, and integrates with Cloud Logging for monitoring. It requires no infrastructure management, making it ideal for scheduled SQL transformations. This directly meets the team's requirements with minimal operational overhead.

Pub/Sub Lite guarantees at-least-once delivery, so duplicate messages can reach subscribers; exactly-once is unavailable. This satisfies the stem's constraint by clarifying that the team cannot rely on deduplication, and must implement idempotent processing or their own deduplication logic to achieve exactly-once semantics.

Preemptible instances cost significantly less than standard Dataproc worker nodes, and Spark can tolerate their loss through retries and task rescheduling. Since the daily jobs are short and their characteristics stay unchanged, using preemptible workers cuts compute spend without altering the job design.

Ordering in Pub/Sub is enforced per ordering key, so setting enable_message_ordering=true on a pull subscription makes the service deliver messages sharing the same user_id sequentially. Without this flag, Pub/Sub provides at-least-once delivery with no ordering guarantee, so clickstream events for one user could arrive out of sequence.

The Cloud DLP API is purpose-built for discovering and de-identifying sensitive data such as names, account numbers, and addresses. Calling it from a Dataflow ParDo transform lets you tokenize, mask, or encrypt PII in-flight, so only de-identified records reach BigQuery. This directly satisfies the regulation without adding storage layers or external systems.

Dataflow provides serverless, autoscaling stream processing with exactly-once semantics, satisfying the sub-second latency and variable-volume constraints. Its Apache Beam pipeline can transform sensor data and write directly into BigQuery via the built-in BigQueryIO connector, with no servers to manage.

A pull subscription supports attribute filters, so Pub/Sub discards non-matching messages server-side before delivery, satisfying the filtering requirement. Pull delivery retains at-least-once semantics, meaning occasional duplicates may reach the subscriber, which the stem explicitly tolerates.

Authorized views grant external partners query access to a view while restricting them to the columns and rows the view exposes, without granting underlying table access. This directly satisfies the requirement to share only a subset of sensitive customer data.

Cloud Dataflow runs the same Apache Beam pipeline for both bounded and unbounded sources, so one codebase serves batch and streaming. This satisfies the unified-pipeline requirement while remaining fully managed, minimising operational overhead for the Beam-experienced team.

Partitioning by ingestion time and clustering by customer_id and transaction_date optimises the table's physical layout. An authorised view then runs with the owner's permissions, so granting domain users access to the view alone satisfies the domain-restriction requirement without exposing the base table.

Pub/Sub Lite provides the same high-throughput, low-latency ingestion as Pub/Sub but at a substantially lower cost, because it trades away automatic multi-zone replication and requires you to provision capacity in a specific zone. That trade-off satisfies the throughput requirement while directly addressing the stated cost concern.

Exporting the slowly changing 40 GB reference data once per day to Cloud Storage removes per-element BigQuery reads entirely, and Avro gives compact, schema-carrying storage that Dataflow reads efficiently. The single daily export keeps the reference fresh while eliminating the quota pressure and repeated scan cost that the BigQuery side input caused on every pipeline run.

Data Fusion instance type determines the available compute and memory for pipeline execution. Basic instances are limited to a small, fixed profile, so upgrading to Enterprise provides greater resources and the ability to scale executors, directly addressing the suspected under-provisioning causing slow ETL runs.

BigQuery is a fully managed, petabyte-scale data warehouse that supports ANSI SQL and provides automatic scaling and real-time analytics. It is designed for ad-hoc queries and business intelligence workloads, with separation of storage and compute, and integrates with BI tools. It is the ideal choice for this scenario.

Partitioning by order_date prunes scanned data when queries filter on that column, while clustering by customer_id co-locates rows sharing that value within each partition. This combination satisfies both frequent filter predicates, reducing bytes scanned and query cost compared with clustering or partitioning alone.

Dataproc Serverless runs Spark workloads without provisioning or managing a cluster, and billing is based on the resources the workload consumes during execution. For a nightly 2 GB transformation, the job completes quickly and the team pays only for that run. This matches the requirements for no administration and per-job cost.

Fixed windows align to epoch boundaries, so a 5-minute window groups events by user_id without overlap. Allowed lateness of 2 minutes retains late-arriving events, satisfying the stated delay constraint. Early triggers every minute emit speculative results, balancing latency against the accuracy gained from late data.

A side input reads the CSV once and broadcasts the small reference table to every worker, so each streaming element joins locally without per-element Cloud Storage reads or external lookups. This satisfies the cost-effectiveness and operational-simplicity constraints for a few-thousand-row table, avoiding the latency and expense of querying an external store per event.

Cloud Storage is the foundational object store for data lakes on Google Cloud. It provides durable, highly available, and cost-effective storage with native integration into BigQuery, Dataproc, and other analytics services. The Standard class is suitable for frequently accessed data, and lifecycle policies can transition older data to colder classes to optimize cost.

Configuring allowed lateness to a duration greater than the maximum expected delay ensures that late events are still accepted and processed into the correct window. The trigger firing on the watermark passing the window end provides timely results, and the allowed lateness extends the window's lifetime to accommodate late data. This balances completeness and latency.

Hot key fanout splits a single hot key into multiple keys by appending a random suffix, so the GroupByKey distributes the load across many workers. After the first grouping, a second grouping removes the suffix and combines the partial results. This reduces skew without altering the pipeline's logic or output, and it is a standard Dataflow pattern for hot keys.

The Storage Write API with exactly-once semantics assigns stream offsets and only commits data when checkpoints succeed, so retries do not duplicate rows. Combined with deterministic record identifiers and Dataflow checkpointing, a restarted job resumes without re-inserting committed records. Data written through committed streams becomes immediately queryable, satisfying the requirement that the table be readable right after each successful load.

Dataflow provides exactly-once streaming semantics, autoscaling that absorbs the 3x burst, and native Pub/Sub and BigQuery connectors. Sliding windows with allowed lateness handle out-of-order events, and the Storage Write API gives low-latency, high-throughput BigQuery inserts, satisfying the 30-second visibility requirement without managing clusters.

Dataproc Serverless runs Spark workloads without provisioning or managing a cluster, allocating resources only while the job executes and charging for that consumption. This matches the stem's requirements for occasional jobs, no cluster management and pay-per-use billing, unlike a standard Dataproc cluster.

Datastream performs log-based CDC by reading the PostgreSQL write-ahead log, so no triggers or schema changes are needed on the source, satisfying the constraint that the database cannot be modified. It delivers low-latency change records that can land in Cloud Storage or be written directly into BigQuery, providing the ingestion mechanism the design requires.

Setting allowed lateness to 24 hours allows the pipeline to accept and process data that arrives up to 24 hours after the window ends. This directly addresses the late-arriving data requirement. Combined with a proper trigger, it ensures that late data is included in the results and written to BigQuery.

Cloud Dataproc is a managed Spark and Hadoop service that supports Hive, HDFS, and other Hadoop ecosystem tools. It allows you to run existing Hive scripts with minimal changes and provides cluster management, autoscaling, and integration with Cloud Storage. It is the ideal choice for migrating on-premises Hadoop workloads.

Raising `spark.task.maxFailures` lets individual tasks retry after a preemptible worker is reclaimed, so the job survives transient node loss without failing the stage. However, this only addresses task-level retries; Dataproc's enhanced flexibility mode, which pairs a standard primary pool with preemptible secondary workers, is what actually enables graceful recovery.

Fixed 30-second windows partition the clickstream into non-overlapping intervals matching the grouping requirement. Combined with allowed lateness and early triggering, they satisfy the windowing constraint while the other settings handle late data and partial results.

Pub/Sub Lite provisions capacity in zonal or regional reservations, giving a much lower throughput price than standard Pub/Sub. Its at-least-once delivery offers no replication guarantee across zones, matching the stated tolerance for occasional message loss at high volume.

Dataproc Metastore is a fully managed, highly available service that replicates metadata across zones and performs automatic failover without administrative intervention. This satisfies the stem's requirement for a managed metadata service for Spark jobs, eliminating the single point of failure inherent in a self-managed Hive Metastore deployment.

Setting allowed lateness on the window lets Dataflow retain window state and emit updated results for up to one hour after the watermark passes, directly satisfying the stem's late-arriving data constraint. Unlike discarding mode, it triggers late firings so enriched events still reach the BigQuery dashboard table.

Push subscriptions deliver messages to an HTTPS endpoint and let the endpoint return an HTTP success code per message, which acknowledges that individual message. This satisfies the requirement for individual acknowledgement without the subscriber pulling.

Cloud Pub/Sub is a scalable, serverless messaging service for ingesting streaming data. It can handle high-throughput event streams and integrate seamlessly with Dataflow. Pub/Sub ensures reliable delivery and can buffer messages, which is essential for ingesting streaming events into a unified pipeline that also processes batch data.

Pub/Sub allows multiple subscriptions on a single topic, each receiving a copy of every message. By creating one subscription for Dataflow and another for a Cloud Function that archives to Cloud Storage, both consumers independently receive all events. This ensures no data loss and allows independent processing. This is the standard fan-out pattern.

Dataproc Metastore provides a fully managed, Hive-compatible metastore service that persists table metadata and model artefacts independently of cluster lifecycle. This satisfies the requirement to retain trained models and metadata in a Hive-compatible store across ephemeral Dataproc clusters.

Cloud Data Fusion pipelines execute on ephemeral Dataproc clusters, which provide the distributed compute needed for complex joins across multiple datasets. Scheduling daily runs on Dataproc satisfies the stem's transformation complexity while remaining cost-efficient between executions.

Combine.perKey performs per-key aggregation after windowing, computing the average value for each key within each 5-minute window. It satisfies the requirement to group events and average per key, unlike global combines that would merge across keys.

Archive storage is the lowest-cost option for long-term retention and is ideal for data accessed less than once a year. BigQuery external tables allow querying JSON data directly from Cloud Storage without loading, satisfying the SQL query requirement with minimal transformation. This combination is both cost-effective and functional for infrequent access over 7 years.

Cloud Data Fusion provides a graphical drag-and-drop interface for building ETL pipelines, with a reusable plugin marketplace of pre-built transforms and connectors. This directly satisfies the stem's requirement for visual pipeline design plus marketplace transforms, unlike code-first services such as Dataflow or Dataproc.

This approach avoids the memory-intensive shuffle of a single large MERGE by splitting work into a bounded DELETE on a partition and a streaming-friendly INSERT. BigQuery can execute each statement with less resource contention, and WRITE_APPEND avoids rewriting the entire target table. It preserves the schema and downstream behavior while scaling to large daily volumes.

Cloud Dataflow, based on Apache Beam, provides a unified programming model for batch and streaming. It supports event-time processing, watermarks, and triggers to handle late-arriving data accurately. With windowing and allowed lateness, it can produce correct results even when data arrives up to 2 hours late, making it ideal for real-time fraud detection with late data.

roles/bigquery.dataViewer grants read access to datasets, tables and views, permitting queries while denying write operations such as inserts, updates, deletes or dataset removal. It therefore satisfies both constraints: the analyst can run queries but cannot modify data or delete the dataset.

Fixed 10-minute windows satisfy the grouping requirement, while one hour of allowed lateness lets late events still update their window's results. The trigger firing on watermark plus early firings emits speculative results promptly, then corrects them once the watermark passes, so no late data within the hour is dropped.

The CDAP SDK lets developers author custom transformations as plugins, package them, and deploy into Cloud Data Fusion, extending the pipeline beyond marketplace offerings. This satisfies the requirement for transformations unavailable in the marketplace, which built-in operators and existing plugins cannot supply.

Cloud Pub/Sub ingests the device streams, while Dataflow provides exactly-once processing through its streaming engine's state and deduplication. This satisfies the stem's low-volume, real-time dashboard requirement, since Dataflow's windowing and triggers emit results continuously rather than only on batch completion.

Scaling worker machine type raises per-worker CPU and memory, directly relieving the compute saturation causing backlog growth at peak. Dataflow autoscaling adds workers, but each remains bound by its machine's vCPU and memory limits, so upgrading the type increases throughput per worker for CPU-bound aggregation stages.

Partitioning on transaction_date lets queries prune to the last 30 days, and clustering on customer_id sorts data within each partition so lookups scan far fewer bytes. Together they cut both query latency and bytes-billed cost for the stated access pattern.

A pull subscription with seven-day retention keeps unacknowledged messages durably stored while Dataflow's backlog grows during spikes, so no data is dropped. This satisfies the stem's no-loss constraint when the pipeline falls behind, since push subscriptions cannot replay expired messages.

Cloud Dataproc Serverless for Spark is a fully managed, serverless Spark environment that eliminates the need to provision or manage clusters. It automatically scales resources based on workload and charges only for the resources consumed during job execution. It natively supports Spark jobs and can read Parquet data from Cloud Storage, making it the ideal choice for minimizing operational overhead and paying only for what is used.

Cloud Dataproc is a fully managed service for running Apache Hadoop and Spark clusters. It supports running existing MapReduce jobs with minimal changes, as it provides HDFS-compatible storage via the cluster's local HDFS or Cloud Storage connector. Dataproc handles cluster provisioning, scaling, and management, reducing operational overhead while allowing you to lift and shift your Hadoop workloads.

Cloud Pub/Sub provides reliable, scalable message ingestion with at-least-once delivery. When combined with Cloud Dataflow, which supports exactly-once processing semantics for streaming pipelines, this ensures that each message is processed exactly once. Dataflow can handle near real-time processing, extract vitals, and trigger alerts. This combination meets the requirements for no message loss and exactly-once processing in a near real-time system.

Cloud Dataflow with Apache Beam provides serverless, autoscaling stream processing with exactly-once semantics via its streaming engine, satisfying the minimal operational overhead and exactly-once constraints. Beam's windowing and triggers handle the rolling one-hour dashboard aggregation over continuous clickstream events.

Dataproc Serverless runs Spark workloads on managed, ephemeral infrastructure, so no cluster provisioning, sizing or teardown is required. This satisfies the serverless constraint directly, unlike Dataproc on Compute Engine, which requires you to create and manage clusters yourself.

A custom quota capping concurrent queries prevents runaway ad-hoc queries from monopolising slots and exhausting capacity. It satisfies the cost-control requirement by throttling query concurrency, so a single user or workload cannot overwhelm the project's shared BigQuery resources.

BigQuery materialized views refresh automatically, but not instantly on every base-table change. They use a configurable refresh interval, with a default of 30 minutes, so hourly updates are covered. This satisfies the stem's hourly-update constraint without manual intervention, unlike logical views, which recompute on every query.

Pub/Sub Lite provides zonal, low-cost messaging with no replication across zones, so it tolerates occasional message loss while meeting the low-cost constraint. Standard Pub/Sub replicates messages, incurring higher cost for durability the scenario does not require.