# ML Lifecycle: Data, Training, Deployment

> Chapter 54 of the Courseiva GOOGLE-CLOUD-DIGITAL-LEADER curriculum — https://courseiva.com/learn/google-cloud-digital-leader/ml-lifecycle-gcdl

**Official objective:** 3.2 — Data Analytics AI

## Introduction

The complete machine learning lifecycle on Google Cloud, encompassing data preparation, model training, and deployment, is the foundation for understanding end-to-end ML workflows. You will learn the key stages from data ingestion to serving predictions, including the tools and best practices for each phase. For the GCDL exam, approximately 15-20% of questions touch on ML lifecycle concepts, making this a critical area for understanding how Google Cloud enables end-to-end ML workflows. Expect scenario-based questions that test your ability to recommend the right services and sequence of steps.

## Building a Custom Car from Scratch

A dusty garage workshop, steel sheets and rubber tires stacked against the walls. First, you gather raw materials: steel, rubber, glass, and electronics. This is like collecting raw data. Next, you clean and shape these materials—cut steel, mold rubber, wire electronics—which is data preprocessing and feature engineering. Then, you assemble components into a prototype: engine, chassis, and body. This is model training, where algorithms learn from processed data. You test the prototype on a track, measure performance (speed, handling), and tweak the design—this is model evaluation and hyperparameter tuning. Once satisfied, you mass-produce the car, install it in dealerships, and customers drive it. This is model deployment, where the trained model serves predictions in production. But a car needs regular maintenance: oil changes, software updates, and part replacements. Similarly, a model requires monitoring, retraining, and versioning. If you skip cleaning the steel (data quality), the car rusts. If you don't test enough, the car fails on curves. If you deploy without monitoring, the engine may overheat unnoticed. Each step mirrors the ML lifecycle: data → training → deployment → monitoring → iteration. The GCDL exam tests your understanding of this pipeline as a continuous loop, not a one-time project.

## Core explanation

### What is the ML Lifecycle?
The ML lifecycle is the end-to-end process of developing and maintaining machine learning models. It encompasses everything from defining the business problem and collecting data, to training, evaluating, deploying, and monitoring models in production. Google Cloud provides a suite of services to support each stage, including BigQuery for data warehousing, Vertex AI for model training and deployment, and Cloud Monitoring for ongoing model performance. The lifecycle is iterative; models degrade over time and require retraining, monitoring, and versioning.

### Why the ML Lifecycle Matters for GCDL
As a Digital Leader, you need to understand the high-level workflow and the Google Cloud services that enable it. The exam tests your ability to identify the correct service for each stage, recognize common pitfalls, and design a lifecycle that meets business requirements. You will not be asked to write code, but you must know the purpose of tools like Vertex AI, AI Platform, Dataflow, and Cloud ML Engine.

### Stage 1: Data Ingestion and Preparation
Data is the foundation of any ML project. Google Cloud offers several services for data ingestion:
- **Cloud Storage**: For batch data uploads (e.g., CSV, images).
- **Pub/Sub**: For streaming data ingestion (e.g., real-time sensor data).
- **Dataflow**: For batch and stream processing (e.g., Apache Beam pipelines).
- **BigQuery**: For large-scale data warehousing and querying.

Data preparation includes cleaning, transforming, and splitting data into training, validation, and test sets. Vertex AI provides **Vertex AI Datasets** for managing labeled data, supporting tabular, image, text, and video. The exam expects you to know that data should be split (e.g., 80/10/10) and that **feature engineering** is critical for model performance.

### Stage 2: Model Training
Training involves selecting an algorithm and feeding it training data to learn patterns. Google Cloud offers multiple training options:
- **Vertex AI Training**: Managed training with pre-built containers, custom containers, or distributed training.
- **AutoML**: For non-experts; automatically trains high-quality models with minimal effort.
- **Custom Training**: Using frameworks like TensorFlow, PyTorch, or scikit-learn.

Key concepts:
- **Hyperparameter tuning**: Vertex AI Vizier for optimizing hyperparameters.
- **Training jobs**: Submit a training job with specification of machine type, accelerator (GPU/TPU), and region.
- **Checkpoints**: Save model checkpoints during training to resume if interrupted.

### Stage 3: Model Evaluation and Validation
After training, you evaluate the model on a held-out test set. Metrics depend on the problem type:
- Classification: Accuracy, precision, recall, F1-score, AUC-ROC.
- Regression: MAE, MSE, RMSE, R².
- AutoML automatically computes these and provides a confusion matrix and feature importance.

**Vertex AI Model Evaluation** gives you these metrics. You must ensure the model meets business thresholds before deployment. The exam may ask about **overfitting** (model performs well on training data but poorly on test data) and **underfitting** (model fails to capture patterns).

### Stage 4: Model Deployment
Deployment makes the model available for predictions. Vertex AI supports:
- **Vertex AI Endpoints**: For online (real-time) prediction with autoscaling.
- **Batch Prediction**: For offline predictions on large datasets.
- **Model Registry**: For versioning and managing model lifecycle.

When deploying, you specify:
- **Machine type**: e.g., n1-standard-2, with or without GPU.
- **Min/max replicas**: For autoscaling.
- **Traffic split**: Gradually roll out new model versions (e.g., 10% new, 90% old).

### Stage 5: Monitoring and Management
Once deployed, you must monitor for:
- **Data drift**: Changes in input data distribution.
- **Model drift**: Degradation in prediction accuracy over time.
- **Prediction anomalies**: Outliers or unexpected values.

**Vertex AI Model Monitoring** alerts on skew and drift. You can also use **Cloud Monitoring** for infrastructure metrics (latency, error rates). The exam emphasizes that models need **continuous retraining** to stay relevant, often using **Vertex AI Pipelines** to automate the retraining workflow.

### Stage 6: Iteration and Automation
The ML lifecycle is cyclical. **Vertex AI Pipelines** (based on Kubeflow Pipelines) let you define and run the entire workflow as code, enabling repeatable and auditable processes. **CI/CD for ML** (MLOps) integrates with Cloud Build and Cloud Source Repositories.

### Key Google Cloud Services Summary
- **Vertex AI**: Unified platform for data, training, deployment, monitoring.
- **BigQuery**: Data warehouse and ML (BigQuery ML for SQL-based models).
- **Dataflow**: Data processing pipelines.
- **Cloud Storage**: Object storage for datasets and models.
- **Cloud Monitoring**: Infrastructure and model monitoring.
- **Cloud Pub/Sub**: Event ingestion.

### Exam-Relevant Details
- **Vertex AI** is the primary service for ML lifecycle on Google Cloud. It replaced AI Platform.
- **AutoML** is for users with limited ML expertise.
- **BigQuery ML** allows creating and executing ML models using SQL.
- **Dataflow** is for both batch and streaming data processing.
- **Model deployment** supports both online (endpoints) and batch prediction.
- **Model monitoring** checks for skew and drift.
- **Training** can use CPUs, GPUs, or TPUs.
- **Hyperparameter tuning** can be done with Vertex AI Vizier.
- **Pipelines** automate the ML workflow.

### Common Pitfalls
- Not splitting data correctly (e.g., using test data for training).
- Deploying without monitoring, leading to silent degradation.
- Using the wrong service for the task (e.g., using Cloud Functions for batch processing).
- Ignoring data quality issues.

### Conclusion
The ML lifecycle on Google Cloud is a structured process from data to deployment and beyond. Understanding each stage and the corresponding Google Cloud service is essential for the GCDL exam. Remember that the lifecycle is iterative; monitoring and retraining are continuous.

## Real-world context

### Enterprise Scenario 1: Retail Customer Churn Prediction
A large retailer wants to predict which customers are likely to churn in the next month. They have historical purchase data, customer service interactions, and demographic data stored in BigQuery. They use Vertex AI Datasets to label churners (churned within 30 days) and non-churners. Data preparation includes feature engineering: recency, frequency, monetary value (RFM), and average support call duration. They train an AutoML model (tabular) with a 80/10/10 split. After evaluation (AUC > 0.85), they deploy as an endpoint with 2 n1-standard-4 machines and autoscaling (min=1, max=5). They set up Vertex AI Model Monitoring to check for drift in input distributions. When a new promotion changes customer behavior, drift is detected, triggering a retraining pipeline. Common misconfiguration: not splitting data temporally (using future data to predict past) leads to overoptimistic metrics.

### Enterprise Scenario 2: Medical Image Classification
A healthcare provider uses chest X-rays to classify pneumonia. They have thousands of DICOM images stored in Cloud Storage. They use Vertex AI Datasets for image data with bounding boxes. They train a custom TensorFlow model with TPU acceleration (v3-8) for faster training. Hyperparameter tuning with Vertex AI Vizier optimizes learning rate and batch size. After evaluation (F1 > 0.92), they deploy to a private endpoint with HIPAA compliance. They use batch prediction for nightly runs on new images. Monitoring includes prediction confidence thresholds; low confidence cases are flagged for human review. Pitfall: class imbalance (few pneumonia cases) requires careful sampling; otherwise model is biased.

### Enterprise Scenario 3: Financial Fraud Detection
A bank detects fraudulent transactions in real-time. Data streams from Kafka to Pub/Sub, then Dataflow processes and enriches features (e.g., transaction velocity, geolocation). Features are stored in BigQuery for training. They use Vertex AI with a gradient boosting model (XGBoost). Deployment is a high-throughput endpoint with min=3, max=10 replicas, each with 8 vCPUs. They monitor prediction latency (target < 50ms) and model accuracy daily. If fraud patterns change, Vertex AI Model Monitoring triggers retraining. Common issue: concept drift due to new fraud techniques; retraining must be frequent. They use Vertex AI Pipelines to automate the entire workflow weekly.

## Exam focus

The GCDL exam (Domain: Data Analytics AI, Objective 3.2) tests your understanding of the ML lifecycle stages and the appropriate Google Cloud services. Expect 3-5 questions on this topic. Key areas:

1. **Service matching**: Know which service is used for each stage (e.g., BigQuery for data warehousing, Vertex AI for training/deployment, Dataflow for data processing).
2. **AutoML vs Custom Training**: AutoML is for non-experts; custom training for more control. The exam may ask when to use each.
3. **Online vs Batch Prediction**: Online for real-time (low latency), batch for offline (high throughput).
4. **Model Monitoring**: Focus on drift and skew detection, and the need for retraining.

**Common wrong answers**:
- Choosing Cloud Functions for data processing (should be Dataflow for complex pipelines).
- Thinking AutoML always outperforms custom models (not true; it depends on data and problem).
- Believing once deployed, a model never needs retraining (models degrade over time).
- Mixing up training and prediction services (e.g., using Cloud ML Engine for prediction only).

**Exam trap**: Questions may describe a scenario with streaming data and ask for the best service. Answer: Pub/Sub + Dataflow. Wrong answers: Cloud Storage (for batch) or BigQuery (for analytics).

**Numbers to remember**:
- Default data split: 80/10/10 (training/validation/test).
- Autoscaling: specify min and max replicas.
- Traffic split: e.g., 10% new model, 90% old model for canary deployment.

**Edge cases**:
- When data is highly imbalanced, use resampling or class weights.
- For very large datasets, use distributed training with TPUs.
- For regulatory compliance, use Vertex AI with VPC-SC and CMEK.

**How to eliminate wrong answers**: Identify the stage (data, training, deployment) and pick the service that specializes in that stage. For example, if the question is about building a ML model without coding, AutoML is correct. If it's about processing streaming data, Dataflow is correct.

## Step by step

1. **Define Business Problem and Objectives** — Clearly articulate the problem to solve (e.g., predict customer churn, classify images). Establish success metrics (e.g., accuracy > 90%, latency < 100ms). Determine if ML is the right approach; sometimes a rule-based system suffices. This step aligns with business goals and sets the stage for data requirements.
2. **Collect and Ingest Data** — Gather raw data from sources: databases, APIs, files, streams. Use Cloud Storage for batch uploads, Pub/Sub for streaming, or Dataflow for complex pipelines. Ensure data is stored in a central location like BigQuery for tabular data or Cloud Storage for unstructured data. Consider data volume, velocity, and variety.
3. **Prepare and Preprocess Data** — Clean data: handle missing values, remove duplicates, correct errors. Perform feature engineering: create new features, transform variables (e.g., scaling, encoding categoricals). Split data into training (80%), validation (10%), and test (10%) sets. Use Vertex AI Datasets to manage labeled data. This is often the most time-consuming step.
4. **Train and Tune Model** — Select algorithm (e.g., linear regression, neural network) and framework (TensorFlow, PyTorch). Submit training job to Vertex AI Training with specified machine type (e.g., n1-standard-4) and accelerator (GPU/TPU). Optionally, use Vertex AI Vizier for hyperparameter tuning. Monitor training through logs and metrics. Save model artifacts to Cloud Storage.
5. **Evaluate and Validate Model** — Evaluate trained model on test set using appropriate metrics (accuracy, precision, etc.). Use Vertex AI Model Evaluation to get detailed reports. Check for overfitting/underfitting. If performance is unsatisfactory, iterate: adjust data, features, or hyperparameters. Validate against business thresholds.
6. **Deploy Model to Production** — Register model in Vertex AI Model Registry. Deploy to an endpoint for real-time predictions or use batch prediction for offline. Configure machine type, scaling (min/max replicas), and traffic split for A/B testing. Test endpoint with sample requests. Monitor initial latency and error rates.
7. **Monitor and Retrain Continuously** — Set up Vertex AI Model Monitoring to detect data drift and model skew. Use Cloud Monitoring for infrastructure metrics. If drift exceeds thresholds, trigger retraining pipeline (Vertex AI Pipelines). Retrain with new data, evaluate, and deploy updated model. Log all versions for audit.

## Comparisons

### AutoML vs Custom Training

**AutoML:**
- Requires no ML expertise
- Limited to supported problem types (tabular, image, text, video)
- Automatically searches for best architecture and hyperparameters
- Less control over model internals
- Faster time to market for standard problems

**Custom Training:**
- Requires ML expertise and coding
- Supports any framework (TensorFlow, PyTorch, scikit-learn, etc.)
- Full control over architecture, training loop, and hyperparameters
- Can implement custom loss functions and layers
- Better for novel or complex problems

## Common misconceptions

- **Misconception:** Once a model is deployed, it works forever. **Reality:** Models degrade over time due to data drift and concept drift. Continuous monitoring and retraining are essential.
- **Misconception:** AutoML is always better than custom models. **Reality:** AutoML is great for non-experts and common tasks, but custom models can outperform when domain expertise or specific architectures are needed.
- **Misconception:** BigQuery ML can replace Vertex AI for all ML tasks. **Reality:** BigQuery ML is limited to SQL-based models and tabular data. Vertex AI supports complex models (images, text, custom frameworks) and full lifecycle management.
- **Misconception:** Data preparation is a one-time step. **Reality:** Data preparation is iterative. As new data arrives, preprocessing steps may need adjustment, especially when data distributions change.
- **Misconception:** Online prediction and batch prediction are interchangeable. **Reality:** Online prediction is for real-time, low-latency requests; batch prediction is for large volumes with no real-time requirement. They have different infrastructure configurations.

## Key takeaways

- The ML lifecycle includes data ingestion, preparation, training, evaluation, deployment, monitoring, and retraining.
- Vertex AI is the unified platform for the entire ML lifecycle on Google Cloud.
- AutoML is for users with limited ML expertise; custom training offers more control.
- Data should be split into training (80%), validation (10%), and test (10%) sets.
- Online prediction uses endpoints for real-time inference; batch prediction is for offline large-scale predictions.
- Model monitoring detects data drift and model skew, triggering retraining.
- Vertex AI Pipelines automates the ML workflow for repeatability and CI/CD.
- BigQuery ML enables ML using SQL for tabular data within BigQuery.
- Dataflow processes both batch and streaming data for feature engineering.
- Hyperparameter tuning can be performed using Vertex AI Vizier.
- Model deployment supports traffic splitting for canary testing.
- Continuous retraining is necessary to maintain model accuracy over time.

## FAQ

**What is the difference between Vertex AI and AI Platform?**

Vertex AI is the successor to AI Platform, offering a unified UI and API for the ML lifecycle. AI Platform is being phased out. Vertex AI integrates capabilities like Dataset management, Model Registry, and Pipeline orchestration that were separate in AI Platform.

**When should I use AutoML vs custom training?**

Use AutoML when you have limited ML expertise, standard problem types (tabular, image, text), and want quick results. Use custom training when you need full control, custom architectures, or non-standard frameworks.

**How do I monitor model performance after deployment?**

Use Vertex AI Model Monitoring to detect data drift and model skew. Also use Cloud Monitoring for infrastructure metrics (latency, error rate). Set up alerts and automated retraining pipelines.

**What is the difference between online and batch prediction?**

Online prediction serves individual requests with low latency (milliseconds) via endpoints. Batch prediction processes large datasets asynchronously, outputting results to Cloud Storage. Use online for real-time apps, batch for offline analysis.

**Can I use BigQuery ML for real-time predictions?**

BigQuery ML is primarily for batch predictions using SQL queries. For real-time, deploy the model to Vertex AI Endpoints. BigQuery ML models can be exported and deployed to Vertex AI for online serving.

**What is data drift and how do I detect it?**

Data drift is a change in the distribution of input data over time. Vertex AI Model Monitoring compares training data distribution with serving data using statistical tests (e.g., Kolmogorov-Smirnov). Alerts are triggered when drift exceeds a threshold.

**How do I automate retraining?**

Use Vertex AI Pipelines to define a workflow that includes data processing, training, evaluation, and deployment. Trigger the pipeline on a schedule or via Cloud Functions when drift is detected.

---

Interactive version with quiz and diagrams: https://courseiva.com/learn/google-cloud-digital-leader/ml-lifecycle-gcdl