Google Cloud Data Catalog

Google Cloud Data Catalog is a fully managed, scalable metadata management service that helps organizations discover, manage, and govern their data assets. It provides a unified view of metadata across Google Cloud services (BigQuery, Cloud Storage, Pub/Sub, Dataflow, Dataproc, etc.) and external systems via integrations. Data Catalog is not a data storage system; it stores only metadata — information about your data — not the data itself. The service automatically crawls and catalogs technical metadata (schemas, data types, partitioning, clustering) and allows users to add business metadata (descriptions, tags, classifications, ownership).

In large organizations, data is scattered across hundreds of datasets, tables, files, and streams. Without a central metadata repository, data analysts and scientists spend up to 80% of their time just finding and understanding data. Data Catalog solves this by providing: - Data Discovery: Search and browse all data assets across the organization. - Data Understanding: View schema, lineage, and business context for each asset. - Data Governance: Apply policies, classifications, and access controls based on metadata. - Data Lineage: Track how data flows from source to destination (in preview during the exam timeframe).

Data Catalog operates as a metadata service that stores and indexes metadata in a scalable, searchable store. The key components are:

1. Metadata Sources: Data Catalog connects to various Google Cloud services and external systems via built-in connectors. For BigQuery, it automatically crawls all datasets, tables, views, and routines. For Cloud Storage, it crawls buckets and objects. For Pub/Sub, it crawls topics and subscriptions. Each crawl extracts technical metadata such as schema, partitioning, clustering, and timestamps.

2. Metadata Store: All crawled metadata is stored in a centralized, highly available metadata store. This store is partitioned and indexed for fast search. The metadata includes both technical metadata (automatically extracted) and business metadata (manually added via tags and descriptions).

3. Search and Discovery: Data Catalog provides a search interface (console, API, and SDK) that uses full-text search and faceted filtering. Users can search by asset name, description, tags, columns, and data types. The search index is built from the metadata store and updated in near real-time as metadata changes.

4. Tag Templates and Tags: Tags are the primary mechanism for adding business metadata. A tag template defines a set of fields (e.g., PII, sensitivity, data steward). Users create tags based on these templates and attach them to data assets (datasets, tables, columns, files). Tags are inheritable: a tag on a dataset can be inherited by its tables unless overridden.

5. Policy Tags (with Data Catalog): Data Catalog integrates with BigQuery column-level security via policy tags. You can create policy tags in Data Catalog, attach them to columns, and then use BigQuery's IAM-based access control to restrict access to sensitive columns. Policy tags are essentially Data Catalog tags with special semantics for access control.

To use Data Catalog, you must enable the Data Catalog API in your project:

gcloud services enable datacatalog.googleapis.com

To list all tag templates in a project:

gcloud data-catalog tag-templates list --project=PROJECT_ID

To create a tag template:

gcloud data-catalog tag-templates create TEMPLATE_ID \
    --location=LOCATION \
    --display-name="Display Name" \
    --field=id=field_id,type=string,display-name="Field Display",required=TRUE \
    --field=id=field_id2,type=enum(Value1|Value2),display-name="Field Display 2"

To attach a tag to a BigQuery table:

gcloud data-catalog tags create \
    --entry=ENTRY_NAME \
    --tag-template=TEMPLATE_ID \
    --tag-file=tag.json

To search for assets:

gcloud data-catalog entries search --query="sensitive"

Data Catalog uses an entry-based metadata model: - Entry Group: A container for entries. For example, BigQuery datasets are entry groups. - Entry: A single data asset, such as a BigQuery table, a Cloud Storage bucket, or a Pub/Sub topic. Each entry has a unique name (fully qualified resource name) and contains technical metadata. - Tag: A piece of business metadata attached to an entry or a column. Tags are instances of tag templates. - Tag Template: A schema that defines the fields of a tag.

Data Catalog is designed to handle millions of entries. There are quotas and limits:

Data Catalog uses IAM for access control. Key roles: - roles/datacatalog.admin: Full control over all Data Catalog resources. - roles/datacatalog.editor: Can edit entries and tags but cannot manage IAM. - roles/datacatalog.viewer: Read-only access to entries and tags. - roles/datacatalog.tagTemplateCreator: Can create and manage tag templates. - roles/datacatalog.tagTemplateViewer: Read-only access to tag templates.

Permissions are evaluated at the resource level. For example, to attach a tag to a BigQuery table, you need datacatalog.tags.create on the entry and datacatalog.tagTemplates.use on the tag template.

For the GCDL exam, focus on:

Common exam scenarios include: choosing the right service for metadata management, understanding how to tag sensitive data, and identifying the benefits of Data Catalog for data discovery.

A large bank uses Data Catalog to manage thousands of BigQuery tables across multiple projects. They have strict regulatory requirements for data privacy (GDPR, CCPA). They create tag templates for 'PII Classification' (fields: PII Type, Sensitivity, Data Steward) and 'Retention Policy' (fields: Retention Period, Legal Hold). Using Cloud DLP, they automatically scan tables for PII and write findings as Data Catalog tags. This allows them to enforce column-level security in BigQuery using policy tags. For example, columns tagged as 'Confidential' are restricted to users with the appropriate IAM role. The bank also uses Data Catalog's search to help analysts quickly find approved datasets, reducing data discovery time from hours to minutes. A common pitfall is not granting the Data Catalog service account the necessary BigQuery metadata viewer role, causing crawling to fail silently. Another is over-tagging: creating too many tags on a single table can exceed the 1000-tag limit, causing errors.

A retail company ingests data from multiple sources into Cloud Storage and BigQuery. They use Data Catalog to catalog all files in Cloud Storage buckets (parquet, CSV, JSON) and BigQuery tables. They create a 'Data Source' tag template to track the origin of each dataset (e.g., 'POS System', 'Web Analytics'). Data engineers use Data Catalog to find the correct source tables for building data pipelines, avoiding duplicate or stale data. They also use Data Catalog's lineage (preview) to trace data from raw files to transformed tables, helping with debugging and impact analysis. One challenge is that Cloud Storage crawling requires manual enabling and appropriate permissions; if not configured, files remain invisible in Data Catalog. Additionally, if buckets have millions of objects, crawling can take time and may hit API rate limits. The company mitigates this by using bucket-level filters to exclude temporary or staging directories.

A healthcare organization uses Data Catalog to manage PHI (Protected Health Information) across BigQuery and Cloud Storage. They create a 'PHI' tag template with fields like 'PHI Category' (enum: Direct Identifier, Quasi-Identifier) and 'De-identification Status'. They use Cloud DLP to automatically classify data and write tags. They then use BigQuery column-level security with policy tags to restrict access to PHI columns to only authorized researchers. Data Catalog's search allows auditors to quickly find all assets containing PHI and verify that appropriate controls are in place. A common misconfiguration is not setting up tag inheritance correctly: if a dataset is tagged as 'PHI', but a table in that dataset is accidentally tagged as 'Non-PHI', the table-level tag overrides the dataset tag, potentially exposing data. The organization trains data stewards to always use inheritance unless there is a specific reason to override.