Data Types and Analytics Workloads

Data types classify the format and structure of data. Analytics workloads describe the processing patterns used to derive insights from data. On Google Cloud, choosing the right service for a given data type and workload is critical for cost, performance, and scalability.

Structured data adheres to a predefined schema with rows and columns. Each column has a data type (e.g., integer, string, date). Examples include relational database tables, CSV files, and spreadsheet exports. Structured data is highly organized, making it easy to query using SQL. In Google Cloud, structured data is typically stored in: - Cloud SQL: For OLTP workloads (e.g., transaction processing). - Cloud Spanner: For globally distributed, strongly consistent OLTP. - BigQuery: For OLAP and data warehousing (columnar storage). - Datastore/Firestore: For NoSQL document/entity storage (semi-structured but often used with structured properties).

Structured data is the most efficient for exact-match lookups and aggregations. However, it requires schema flexibility to be low; schema changes can be costly.

Semi-structured data has some organizational properties but does not require a rigid schema. It often uses tags or markers to separate data elements. Common formats include JSON, Avro, Parquet, and XML. Semi-structured data is self-describing: each record can have different fields. Google Cloud services that handle semi-structured data: - Cloud Storage: Stores files in any format. - BigQuery: Natively supports JSON and Avro with schema autodetection. - Dataproc: Processes semi-structured data using Hadoop/Spark. - Pub/Sub: Messages are often JSON or Avro.

Semi-structured data offers flexibility but may require more processing to parse and query efficiently. BigQuery’s JSON functions (e.g., JSON_EXTRACT) allow querying nested fields.

Unstructured data has no predefined structure. Examples include text documents, images, audio, video, and binary files. It is the most abundant type of data (over 80% of enterprise data). Processing unstructured data often requires machine learning or specialized tools. Google Cloud services: - Cloud Storage: Primary store for unstructured data. - Cloud Vision API: Extracts text from images. - Cloud Speech-to-Text: Converts audio to text. - Document AI: Parses documents (e.g., invoices). - Vertex AI: For custom ML models.

Unstructured data is challenging to search and analyze directly; transformations (e.g., OCR, transcription) are usually needed.

Analytics workloads are categorized by processing style, latency, and data volume.

#### Batch Processing Batch processing handles large volumes of data at scheduled intervals. It is ideal for non-time-sensitive tasks like daily reports, ETL jobs, and historical analysis. Google Cloud services: - Cloud Dataproc: Managed Hadoop/Spark for batch jobs. - Cloud Dataflow: Supports both batch and stream (unified model). - BigQuery: Can be used for batch queries. - Cloud Composer: Orchestrates batch workflows.

Batch jobs typically run for minutes to hours. They are cost-effective because resources can be scaled down after completion.

#### Stream Processing Stream processing ingests and processes data in real-time as it arrives. Latency is measured in seconds or milliseconds. Use cases include fraud detection, IoT telemetry, and live dashboards. Google Cloud services: - Cloud Dataflow: Exactly-once semantics, auto-scaling. - Pub/Sub: Ingestion and delivery of streams. - BigQuery: Streaming inserts for near-real-time analytics. - Dataproc: Spark Streaming.

Stream processing must handle out-of-order data, late arrivals, and exactly-once or at-least-once guarantees.

#### Interactive Analytics Interactive analytics allows users to query data with sub-second to few-second latency. It supports iterative exploration and ad-hoc analysis. Google Cloud services: - BigQuery: On-demand petabyte-scale SQL analytics. - Looker: Business intelligence and visualization. - Dataplex: Data mesh and lakehouse.

Interactive analytics relies on columnar storage, caching, and distributed query engines.

#### Machine Learning Workloads ML workloads involve training and deploying models. Training is compute-intensive; inference can be real-time or batch. Google Cloud services: - Vertex AI: End-to-end ML platform. - BigQuery ML: Create models using SQL. - Cloud TPUs: Specialized hardware for training. - AI Platform: For model deployment.

ML workloads blur the line between analytics and AI.

A typical analytics pipeline ingests data (stream or batch), stores it (structured/unstructured), processes it (transformation, enrichment), and serves it (queries, dashboards, ML). For example, IoT sensor data (unstructured streams) is ingested via Pub/Sub, processed with Dataflow (stream), stored in BigQuery (structured), and analyzed with Looker (interactive).

To check data types in BigQuery:

SELECT column_name, data_type FROM `project.dataset.INFORMATION_SCHEMA.COLUMNS` WHERE table_name = 'mytable';

To verify stream processing latency in Dataflow:

gcloud dataflow jobs list --region=us-central1 --status=active
gcloud dataflow metrics list --job-id=JOB_ID --metric=SystemLatency

Data types and workloads are tightly coupled with storage, compute, and networking. For example, Cloud Storage classes (Standard, Nearline, Coldline, Archive) affect cost and retrieval latency for unstructured data. VPC Service Controls can restrict data access. Cloud NAT allows outbound connections from private instances.

The GCDL exam tests your ability to recommend the right service for a given scenario. Focus on:

A large retailer ingests point-of-sale transactions from thousands of stores. Each transaction is a JSON message with store ID, product SKU, quantity, price, and timestamp. The data is semi-structured. The retailer uses Pub/Sub to ingest these messages in real-time. A Dataflow pipeline cleans the data (removing malformed records), enriches it with product catalog data from Cloud SQL, and writes to BigQuery. Every night, a batch job runs to compute daily sales summaries. The interactive dashboard in Looker allows regional managers to see sales by hour. When misconfigured, Dataflow workers may not scale fast enough during Black Friday, causing backlog and delayed insights. Properly configuring max workers and using streaming engine avoids this.

A health insurance company receives claims as PDF documents (unstructured). They use Document AI to extract structured fields (member ID, date, amount). The extracted data (structured) is stored in BigQuery. A batch Dataflow job runs nightly to validate claims against member eligibility (stored in Cloud Spanner). Fraud detection models in Vertex AI score claims in real-time using a stream processing pipeline. The company also archives raw PDFs in Cloud Storage with lifecycle policies to move to Nearline after 90 days. Common issues include Document AI misclassification due to poor OCR, leading to incorrect data. Tuning Document AI processors and validating output with a secondary check (e.g., regex) mitigates this.

A manufacturing plant has thousands of sensors emitting temperature, pressure, and vibration readings every second. This is a high-velocity stream of structured data. They use Pub/Sub for ingestion, Dataflow for stream processing (filtering out-of-range values, computing rolling averages), and BigQuery for storage. A real-time dashboard in Looker alerts operators when thresholds are exceeded. For historical analysis, they run batch queries on BigQuery. Scaling issues arise when sensor data spikes; using Pub/Sub flow control and Dataflow autoscaling helps. Misconfigured windows in Dataflow can cause incorrect aggregates; using event-time processing with allowed lateness (default 0) ensures accuracy.