Data Management practice questions

Dataflow template

Why it fails: While Dataflow is for pipelines, Dataplex quality tasks are configured via their own task interface.

BigQuery Data Transfer Service

Why it fails: This service is for ingestion, not data quality validation.

Cloud Scheduler

Why it fails: Cloud Scheduler is an external trigger, not the execution engine for the task.

roles/bigquery.dataViewer

Why it fails: roles/bigquery.dataViewer grants access to BigQuery datasets and tables, not to Dataplex lake or zone metadata, so it cannot expose the zone's assets. It is tempting because it does allow querying data, but Dataplex metadata visibility requires roles/dataplex.viewer or roles/dataplex.dataViewer granted at the lake level.

roles/dataplex.viewer

Why it fails: roles/dataplex.viewer grants read access to metadata only; it does not include the dataplex.zones.queryData permission needed to query data within the zone. It is tempting because viewer roles are the standard least-privilege choice for read-only metadata inspection, which would be correct if querying were not required.

roles/dataplex.admin

Why it fails: roles/dataplex.admin grants full owner-equivalent control over the lake, including creating and deleting zones and assets, which exceeds the stated requirement of viewing metadata and querying data. It is tempting because admin roles guarantee access, and would be correct for a platform team managing the lake's configuration.

At the Zone level

Why it fails: Zones group assets and govern discovery and security; they hold no table partitioning configuration. Partitioning is defined on the underlying BigQuery table or when registering the asset. Zones suit organising data by domain or sensitivity.

At the Lake level

Why it fails: The lake is a logical container and does not hold storage configuration.

In the IAM policy

Why it fails: IAM policies are for authorization.

Asset

Why it fails: An Asset maps an individual BigQuery dataset, table or Cloud Storage bucket into Dataplex; it cannot itself group many resources across projects into one governance boundary. It is tempting because assets are the building blocks of Dataplex organisation, and creating one would be correct when registering a single resource for discovery and metadata.

Taxonomy

Why it fails: A taxonomy classifies data with policy tags for access control and discovery; it does not group BigQuery datasets and Cloud Storage buckets into one governed entity. It is tempting because taxonomies organise metadata hierarchically, and they would be the right choice when you need consistent business glossaries or column-level policy tags across projects.

Data Zone

Why it fails: A data zone groups assets within a single lake, not datasets and buckets spanning multiple projects; cross-project organisation requires a lake as the top-level boundary. Data zones are tempting because they do logically organise related assets, and they would be correct for segmenting data within one lake by team, domain, or region.

Location-based IAM policies

Why it fails: Dataplex has no location-based IAM policy construct; IAM conditions filter on attributes such as resource name or request time, not geography. Regional restriction is enforced by VPC Service Controls perimeter boundaries around the specified location. Location-based IAM would be tempting if a native geo-condition existed, but it does not.

Data residency tags

Why it fails: Data residency tags are not a Dataplex feature; no such tagging mechanism enforces geographic boundaries. Restricting access by region is achieved with VPC Service Controls perimeters, which block requests originating outside the defined location. Residency tags would be tempting if labelling alone could constrain access, but labels do not enforce network-level boundaries.

VPC Service Controls

Why it fails: While it enforces boundaries, it is not a Dataplex feature for geographic data governance.

Data tags on the assets

Why it fails: Data tags classify and describe assets for discovery and policy conditions, but they do not grant or restrict access by project. Per-department visibility needs IAM policies bound at the zone or asset level. Tags would be the right choice for organising data by sensitivity or business domain, not for enforcing project-scoped access.

IAM roles on the bucket only

Why it fails: Bucket-level IAM governs access to the underlying Cloud Storage objects, not to Dataplex lake, zone, or asset resources, so departments could still enumerate and query each other's assets through Dataplex. Bucket IAM is the correct control when the requirement is to restrict access to the storage objects themselves.

Separate Lakes for each department

Why it fails: Separate lakes per department isolates assets but abandons the shared-lake design the scenario specifies, and does not by itself restrict visibility within a lake. Distinct lakes are the right choice when departments need fully independent governance, discovery, and metadata boundaries rather than shared curation.

Remove the user from the Dataplex Lake Viewer role

Why it fails: They might still have direct access to the GCS bucket.

Create a Dataplex deny-policy

Why it fails: Dataplex does not have a native 'deny' policy for underlying data access.

Delete the asset from Dataplex

Why it fails: This removes the logical link but does not revoke existing bucket access.

It restricts access to users in a single region

Why it fails: A multi-region lake spans a broad geographic area such as the United States or Europe, so it serves users across that footprint rather than confining them to one region. It is tempting because single-region lakes do restrict resources geographically, but that is the opposite of what multi-region selects.

It prevents the use of BigQuery datasets

Why it fails: Multi-region lakes still attach BigQuery datasets as zone assets; the location type governs geographic placement, not asset compatibility. It is tempting because BigQuery datasets themselves carry their own regional or multi-regional settings, but choosing a multi-region lake does not block their use.

It reduces the cost of storing data

Why it fails: Location type controls where Dataplex resources reside, not billing rates; multi-region placement typically carries a higher storage price than single-region. It is tempting because region choice does affect cost in many GCP services, but Dataplex location selection is about data residency and availability, not a discount mechanism.

Grant public access to the dataset

Why it fails: Public access exposes the dataset to every Google account, bypassing Dataplex lake IAM and row-level policies, so cross-organisation governance is lost. It is tempting because public datasets are genuinely shareable without per-user grants, which suits open reference data rather than controlled multi-org sharing.

Copy all data to a single project

Why it fails: Consolidating copies into one project duplicates storage and breaks the source organisations' ownership and update paths, so Dataplex cannot enforce their policies. It is tempting because a single project simplifies billing and IAM, which works when one team owns all data and no external organisation retains control.

Export the data to CSV files

Why it fails: CSV export strips BigQuery's fine-grained access controls and creates unmanaged copies outside Dataplex, so cross-organisation governance cannot be applied. It is tempting because exports are a familiar interchange format, correct for one-off offline analysis or migration where no ongoing governed sharing is required.

Dataproc

Why it fails: Dataproc runs Spark and Hadoop clusters for batch processing, not a single SQL interface spanning storage systems. It is tempting because it queries data, but it requires per-cluster engines rather than Dataplex's unified SQL layer over lakes, warehouses and object stores.

Data Catalog

Why it fails: Data Catalog is a metadata inventory for discovery and tagging, not a query engine; it cannot execute SQL across storage systems. It is tempting because it spans many sources, but it indexes descriptions rather than returning rows, which Dataplex's query component does.

Cloud Storage

Why it fails: Cloud Storage is object storage, not a Dataplex query component; it cannot run SQL across BigQuery, lakes or external systems. It is tempting because Dataplex zones can point at buckets, but the service stores bytes rather than federating queries.

The user is not an administrator of the lake

Why it fails: Lineage capture depends on the service account or user holding the required Data Catalog and Dataplex permissions on the resources involved, not on lake administrator status. Lake administrator is tempting because it grants broad control, but lineage is recorded from job and process metadata, so any principal with the appropriate IAM roles can generate it.

The data files are in a multi-region bucket

Why it fails: Multi-region Cloud Storage buckets are fully supported for lineage; location affects latency and redundancy, not whether Dataplex records the asset. The bucket's region is tempting to blame because lineage is regional, but capture follows the resource's registered entry, so a multi-region bucket still produces lineage normally.

You must pay for them separately

Why it fails: Dataplex and Data Catalog share one billing model; Dataplex metadata management is included, not charged separately. Separate billing suits standalone products with independent SKUs, but Data Catalog's capabilities are folded into Dataplex, so no duplicate charge applies.

They have separate IAM systems

Why it fails: Dataplex and Data Catalog use unified IAM; Data Catalog roles are managed through the same Dataplex permissions model. Separate IAM systems suit genuinely independent platforms, but here one policy governs both, so distinct identity systems do not exist.

They are completely separate products

Why it fails: Dataplex absorbed Data Catalog's metadata and discovery capabilities, so they are not separate products; Data Catalog now surfaces within Dataplex's unified governance layer. Treating them as independent is tempting because Data Catalog predates Dataplex as a standalone service, and older documentation still describes it that way.

Create a Dataplex Data Quality task with a null rule on customer_id and configure it to write results to a Cloud Storage bucket as JSON files, then load those files into BigQuery with a scheduled query.

Why it fails: Writing Data Quality results to Cloud Storage and then loading them into BigQuery adds an unnecessary intermediate step and latency. Dataplex Data Quality tasks can write results directly to a BigQuery table, so this design is more complex than required. It also does not address scoping the scan to the most recent partition as cleanly as a native partition scope.

Create a Dataplex Data Quality task with a null rule on customer_id, then use a Cloud Scheduler job to invoke the Dataplex API nightly and configure the task to publish results to a Pub/Sub topic.

Why it fails: Dataplex Data Quality tasks already support native scheduling, so adding Cloud Scheduler is unnecessary complexity. More importantly, Pub/Sub is a messaging service, not a queryable result store, so the operations team could not easily query historical failures. The scenario explicitly requires results in a BigQuery table, which this approach does not deliver.

Create a Dataplex Data Quality task with a null rule on customer_id and set the task to run on a schedule, but leave the scope at the table level so that all partitions are scanned each night to ensure completeness.

Why it fails: Scanning the entire table every night contradicts the requirement to scan only the most recently loaded partition. This would increase cost and runtime as the table grows, and it does not match the incremental load pattern. The scenario asks for partition-limited validation, so a full-table scope is the wrong choice here.

Transfer Appliance

Why it fails: Transfer Appliance is a physical device shipped to your site for offline data transfer, typically used when network bandwidth is insufficient or data volume is very large. For 500 GB over a reliable 1 Gbps link, shipping hardware would add days of logistics and is unnecessary. Online transfer is faster and simpler in this scenario.

Cloud Interconnect with a Partner Interconnect attachment

Why it fails: Dedicated or Partner Interconnect provides private, high-bandwidth connectivity between on-premises and Google Cloud, but it is a network link, not a transfer service. Provisioning Interconnect can take weeks and is overkill for a 500 GB transfer over an existing reliable internet connection. It does not by itself move data.

Cloud VPN with gsutil parallel composite uploads

Why it fails: A Cloud VPN provides encrypted connectivity but does not orchestrate bulk transfer jobs or provide resumability and scheduling. gsutil can upload files, but managing 500 GB manually with parallel composite uploads is operationally heavy. This combination is not the recommended managed service for large-scale structured transfers.

Create a clustered table on the transaction_date column.

Why it fails: Clustering on transaction_date would sort data by that column, which can improve performance for filtered queries, but it does not physically partition the data. BigQuery still scans all partitions unless a partition filter is applied. For a 5-year table where most queries target the last 30 days, partitioning is more effective at reducing bytes scanned and cost. Clustering alone would not provide the same cost reduction.

Create a view that filters the last 30 days and have analysts query the view.

Why it fails: A view does not store data separately; it is just a saved query. When analysts query the view, BigQuery still scans the underlying table and processes the full 5 years of data unless the base table is partitioned. Therefore, this approach does not reduce cost or improve performance. It also requires analysts to change their queries to use the view instead of the table.

Export the last 30 days of data to a new table and have analysts query that table.

Why it fails: Exporting data to a new table would require ongoing ETL processes to keep the table up to date, adding complexity and potential data staleness. It also forces analysts to change their queries to point to the new table. While it reduces data scanned, it is not a minimal-change solution and does not leverage BigQuery's native partitioning capabilities, which are designed for exactly this use case.

Set a partition expiration on the table to remove old partitions

Why it fails: Partition expiration deletes entire partitions after a set period, regardless of customer. It cannot target individual customer records and would remove unrelated transactions. This violates the requirement to delete only specific rows and would cause data loss for other customers. It is a retention control, not a row-level deletion mechanism.

Recreate the table with a CREATE TABLE AS SELECT that excludes the customer rows

Why it fails: Recreating the table with CTAS rewrites all data and requires dropping the original, which is expensive for large tables and risks losing streaming buffer data. It also briefly doubles storage cost. This approach is heavier than necessary when a targeted DELETE can remove specific rows while respecting partitioning.

Use a column-level security policy to hide the customer records

Why it fails: Column-level security restricts who can view a column but does not delete data. The rows would still exist and be subject to retention, failing the compliance requirement for actual deletion. Policy tags control access, not data lifecycle. This option confuses access control with data deletion.

Export the EMEA rows to a separate BigQuery dataset and grant the compliance officer access to that dataset only.

Why it fails: Exporting creates and maintains a second copy of the data, which the scenario explicitly wants to avoid. It also introduces staleness and additional storage and governance overhead, so it does not meet the requirement to enforce restrictions without duplicating data.

Use column-level security to grant access only to account_id and transaction_date, and use row-level security to filter region to EMEA.

Why it fails: Column-level and row-level security policies can restrict access, but they apply to the base table for all users, so they would also restrict the analysts who must see all columns and rows. Applying them here would break the analysts' access, making this the wrong tool for a single-user restriction.

Grant the compliance officer the BigQuery Data Viewer role on the dataset and rely on their query habits to select only the allowed columns.

Why it fails: Dataset-level viewer access exposes every column and row in the table, so nothing prevents the officer from reading data outside their mandate. Relying on query habits is not enforcement, and it fails the requirement to restrict access at query time.

Use a materialized view that precomputes the extracted JSON fields and refreshes automatically.

Why it fails: Materialized views can precompute results, but they do not support JSON_EXTRACT_SCALAR directly in the definition because the function is not deterministic in the required way for materialization. BigQuery materialized views have limitations on the functions and expressions allowed, so this approach would fail to create or would not refresh correctly.

Create a separate table that stores the parsed JSON fields and use a scheduled query to keep it in sync.

Why it fails: A separate table with a scheduled query adds latency and complexity, and it duplicates data. It does not provide a seamless way for analysts to query the original table and requires maintenance of the sync process. This is not the most efficient or native solution compared to generated columns, which are part of the table schema and automatically updated.

Create a view that uses the JSON_EXTRACT_SCALAR function to parse the JSON column and expose the fields as separate columns.

Why it fails: A view with JSON_EXTRACT_SCALAR can simplify queries, but it still parses the JSON at query time for each access. It does not improve efficiency because the parsing occurs on every query execution. The requirement is to avoid repeated parsing, so a view alone does not satisfy that; it merely hides the complexity.

BigQuery row-level security with a filter on transaction amount

Why it fails: Row-level security filters which rows a user can see, not which columns. It would not prevent analysts from viewing credit card numbers in the rows they are allowed to access, so it does not meet the column-level masking requirement.

Cloud DLP inspection of the table with a scheduled scan

Why it fails: Cloud DLP can discover and classify sensitive data, but inspection alone does not enforce access control. It can tag findings, yet analysts could still query the raw column unless policy tags or other access mechanisms are configured.

Authorized views that exclude the credit card column

Why it fails: Authorized views grant access to a view without granting access to the underlying table, but they do not automatically mask individual columns for arbitrary queries. Analysts would need to use the view, and direct table access would still expose data unless additional controls are applied.

Set a custom quota on the dataset that limits the number of bytes each user can query per day.

Why it fails: Custom quotas restrict how much data users are allowed to query, which can cap spend, but they do not reduce the bytes scanned by any individual query. The underlying full-table scan behavior remains unchanged, and users hitting the quota simply get failures instead of faster, cheaper results. This addresses budget symptoms rather than the scanning inefficiency.

Enable the BigQuery BI Engine reservation on the dataset and let the team continue querying the original table.

Why it fails: BI Engine accelerates certain interactive queries by caching data in memory, but it is designed for sub-second dashboard-style workloads and does not change how many bytes a query is billed for against the base table. It also requires the queries and data to fit its caching model. It is not a general-purpose fix for reducing scanned bytes on a large append-only table.

Create a materialized view that selects all columns from the table, and have the team query the materialized view instead.

Why it fails: A materialized view that selects all columns still stores and scans the same volume of data; BigQuery can only use materialized views to avoid scanning base-table data when it can answer the query from the view's precomputed results. With no aggregation or filtering in the view definition, date-filtered queries still touch the same amount of data, so cost is not meaningfully reduced.