PMLE Automating and orchestrating ML pipelines — 20 Questions

Question 1

An MLOps team is implementing a CI/CD pipeline for a TensorFlow model on Vertex AI. The model training job takes 2 hours and produces a SavedModel. The team wants to automatically trigger a new pipeline run whenever a change is pushed to the 'main' branch of their source repository. The pipeline should include training, evaluation, and if metrics exceed a threshold, deploy the model to a Vertex AI endpoint. Which trigger configuration should they use?

Accepted Answer

Configure a Cloud Build trigger that watches the 'main' branch of Cloud Source Repositories; in the build config, use steps to run the pipeline via the Vertex AI API.. Option D is correct because Cloud Build triggers can be configured to watch a specific branch (e.g., 'main') in Cloud Source Repositories and automatically execute a build configuration. Within that build config, you can use the `gcloud` or `curl` steps to invoke the Vertex AI Pipeline API, which starts the training, evaluation, and conditional deployment workflow. This directly matches the requirement for a branch-based push trigger that orchestrates the full ML pipeline.

Answer

Use Eventarc to listen for Cloud Source Repository push events and invoke a Cloud Run service that starts the pipeline.

Answer

Use an Artifact Registry trigger to detect new model images and then start the pipeline.

Answer

Set up a Cloud Scheduler job that runs every 2 hours and triggers a Vertex AI Pipeline run.

Question 2

A data science team is deploying a PyTorch model for real-time inference using Vertex AI Endpoints. The model requires a custom container with specific CUDA drivers and Python packages. They have created a Docker image and pushed it to Artifact Registry. The pipeline should automatically retrain the model every week and deploy the new version if it passes validation. However, the deployment step fails intermittently with the error 'The container image is not compatible with the machine type.' What is the most likely cause?

Accepted Answer

The container image requires GPU support but the machine type specified in the endpoint is a CPU-only machine.. The error 'The container image is not compatible with the machine type' indicates a mismatch between the container's hardware requirements and the machine type selected for the Vertex AI Endpoint. Since the custom container requires specific CUDA drivers, it is built for GPU acceleration. If the endpoint is configured with a CPU-only machine type (e.g., n1-standard-4), the container will fail to run because the GPU drivers cannot initialize, triggering this incompatibility error.

Answer

The service account does not have permission to pull the container from Artifact Registry.

Answer

The container's health check endpoint is not responding correctly.

Answer

The model artifact size exceeds the maximum allowed for the machine type.

Question 3

An ML engineer is using Vertex AI Pipelines with Kubeflow Pipelines SDK (KFP) to orchestrate a training and deployment workflow. They want to reuse a custom component across multiple pipelines. The component is defined in a Python file 'preprocess.py' that includes a function decorated with @kfp.components.create_component_from_func. How should they package this component for reuse?

Accepted Answer

Import the preprocess module and call create_component_from_func on the function, then use the resulting component in pipeline definitions.. Option A is correct because the recommended way to reuse a custom component defined via `@kfp.components.create_component_from_func` is to import the Python module containing the decorated function and call `create_component_from_func` on that function in each pipeline definition. This creates a reusable component object that can be used directly in the pipeline's `@dsl.pipeline` definition without additional packaging steps. The KFP SDK treats the function as the source of truth, and re-importing ensures the component logic is always current.

Answer

Save the component as a YAML file using kfp.components.ComponentStore and load it in other pipelines.

Answer

Compile the pipeline that uses the component into a JSON file and upload it to Vertex AI.

Answer

Build a custom container image with the function and use it as a base image in other pipelines.

Question 4

A company has a Vertex AI pipeline that trains a model on streaming data from Pub/Sub. The pipeline is triggered by a Cloud Function when new data arrives. Recently, jobs have been failing with 'ResourceExhausted: Quota limit exceeded for regional CPUs in us-central1.' The team needs to ensure successful job execution while minimizing changes. Which approach should they take?

Accepted Answer

Configure the pipeline's training job to use preemptible VMs (which count toward a separate, usually higher quota).. Option D is correct because preemptible VMs count toward a separate, often higher quota for 'Preemptible CPUs' rather than the standard regional CPU quota. By configuring the training job to use preemptible VMs, the team can bypass the exhausted quota without requesting a limit increase or changing the pipeline architecture. This minimizes changes while leveraging the fact that Vertex AI training jobs can be configured to use preemptible VMs via the `worker_pool_specs` with `accelerator_type` and `machine_type` settings.

Answer

Request a quota increase from Google Cloud Support.

Answer

Change the pipeline to run in a different region with available quota.

Answer

Reduce the number of parallel pipeline runs by using a Cloud Tasks queue with rate limiting.

Question 5

An ML team is designing an automated pipeline to retrain a recommendation model every day using new user interaction data stored in BigQuery. The pipeline must be cost-efficient, scalable, and require minimal manual intervention. Which two approaches should they consider?

Accepted Answer

Use Cloud Scheduler to publish a Pub/Sub message daily, which triggers a Cloud Function that starts the Vertex AI Pipeline.. Option C is correct because Cloud Scheduler triggers a Pub/Sub message that invokes a Cloud Function, which starts a Vertex AI Pipeline. This serverless approach is cost-efficient (no idle compute), scales automatically, and requires minimal manual intervention. Option E is correct because Vertex AI Pipelines natively orchestrates ML workflows, and using preemptible VMs reduces training costs by up to 80% while maintaining scalability.

Answer

Deploy a custom Kubernetes cron job on GKE to run the training script directly.

Answer

Use Cloud Composer (Airflow) to schedule the pipeline with a DAG.

Answer

Use Dataflow to continuously read from BigQuery and trigger training when new data arrives.

Question 6

You are an ML engineer at a large e-commerce company. Your team has developed a product recommendation model using TensorFlow and deployed it on Vertex AI Endpoints for real-time inference. The model is retrained weekly using a Vertex AI Pipeline that reads new user interaction data from BigQuery, trains the model, evaluates it, and deploys the new version to the endpoint with a traffic split: 10% to the new model and 90% to the previous champion model. Recently, the team noticed that the new model's online prediction latency has increased significantly (from 50ms to 200ms) after deployment, causing timeouts for some requests. The training code has not changed, and the model size is similar. The pipeline uses a custom container with the same TensorFlow Serving image as before. The deployment step uses the same machine type (n1-standard-4) for the endpoint. What is the most likely cause of the latency increase?

Accepted Answer

The pipeline now includes a data validation step that modifies the SavedModel's serving signature, adding an extra preprocessing operation.. Option C is correct because the pipeline now includes a data validation step that modifies the SavedModel's serving signature, adding an extra preprocessing operation. This additional operation runs during inference on Vertex AI Endpoints, increasing the per-request latency from 50ms to 200ms, even though the model architecture and size remain unchanged. The custom container and machine type are identical, so the latency increase must stem from a change in the serving graph itself.

Answer

The endpoint is using a machine type that is not optimized for the new model's computation.

Answer

The new model has a significantly different architecture that requires more computation.

Answer

The new model is experiencing data skew because the training data distribution has changed.

Question 7

You are designing an ML pipeline for a large-scale recommendation system that runs weekly retraining on historical user interaction data. The pipeline uses TensorFlow and is deployed on Google Cloud. The pipeline must be orchestrated and automated with minimal manual intervention. Which THREE options should you include in your design? (Choose three.)

Accepted Answer

Use Vertex AI Pipelines to define the ML pipeline as a Directed Acyclic Graph (DAG) of components.. Vertex AI Pipelines (option B) is correct because it provides a managed, serverless orchestration service for building, testing, and deploying ML pipelines as Directed Acyclic Graphs (DAGs). This directly supports the requirement for automated, minimal-intervention weekly retraining by allowing you to define reusable components and schedule pipeline runs via Cloud Scheduler or event triggers, integrating natively with TensorFlow and Google Cloud services.

Answer

Use BigQuery scheduled queries to run the training script on a schedule.

Answer

Use AI Platform Notebooks to schedule the training job on a recurring basis.

Question 8

A developer creates a Cloud Build trigger that runs a training pipeline whenever code is pushed to the main branch of the repository. The trigger is configured to use a source archive stored in Cloud Storage. After pushing code to main, the build fails with the error shown. What is the most likely cause of this failure?

Accepted Answer

The source archive is not being updated when code is pushed, so the trigger tries to fetch an old or nonexistent object.. Option C is correct because the trigger is configured to use a source archive stored in Cloud Storage. When code is pushed to the main branch, the trigger attempts to fetch the archive from the specified Cloud Storage location. If the archive is not updated (e.g., via a separate upload or a Cloud Function that rebuilds the archive on push), the trigger will either fetch an old version or fail if the object does not exist. The error indicates that the build cannot proceed because the source archive is stale or missing, not because of a missing config file or permission issue.

Answer

The build configuration file is missing from the source archive.

Answer

The included files filter 'train/**' excludes all files outside the train directory, causing the build to have no source.

Answer

The service account does not have storage.objectViewer permission on the bucket.

Question 9

Your team manages a production ML pipeline on Google Cloud that trains a fraud detection model every 6 hours using new transaction data. The pipeline steps are: (1) Cloud Function triggered by new files in Cloud Storage to validate data, (2) Dataflow job for feature engineering, (3) Vertex AI CustomJob for training, (4) Cloud Function to deploy the model to a Vertex AI endpoint after evaluation. You notice that the pipeline sometimes fails during the Dataflow job step with an error: 'Workflow failed. Causes: The job encountered a system error. Please try again later.' The error occurs sporadically, and retrying the pipeline manually usually succeeds. The team needs a reliable automated solution. What should you do?

Accepted Answer

Orchestrate the pipeline using Cloud Composer with retry policies on the Dataflow operator.. Option D is correct because Cloud Composer (Apache Airflow) provides native retry policies on its Dataflow operators, enabling automatic retries of the Dataflow job when it fails due to transient system errors. This addresses the sporadic failure pattern without manual intervention, ensuring the pipeline runs reliably every 6 hours.

Answer

Schedule the pipeline to run less frequently to reduce load on the Dataflow service.

Answer

Use Cloud Tasks to queue the Dataflow job and retry on failure.

Answer

Increase the number of Dataflow workers and use flexRS to handle transient errors.

Question 10

Drag and drop the steps to set up a BigQuery ML linear regression model for forecasting in the correct order.

Question 11

Drag and drop the steps to set up a batch prediction job using Vertex AI in the correct order.

Question 12

Match each Google Cloud AI/ML service to its primary purpose.

Question 13

Match each MLOps practice to its description.

Question 14

An MLOps team wants to automate the retraining of a model each time new data arrives in a BigQuery table. What is the most efficient Google Cloud service to orchestrate this pipeline?

Accepted Answer

Vertex AI Pipelines with a schedule trigger. Vertex AI Pipelines is purpose-built for orchestrating ML workflows, including model retraining. It integrates natively with BigQuery for data ingestion and supports schedule triggers to automate retraining upon new data arrival, making it the most efficient and managed option for this ML-specific task.

Answer

Cloud Composer with an Airflow DAG

Answer

Dataflow pipeline with a periodic trigger

Answer

Cloud Functions triggered by BigQuery events

Question 15

A data scientist has trained a model using Vertex AI Training and wants to deploy it to a Vertex AI Endpoint for online predictions. Which orchestration service should be used to automate the deployment step after training completes?

Accepted Answer

Vertex AI Pipelines. Vertex AI Pipelines is the correct orchestration service because it is purpose-built for automating and managing end-to-end ML workflows on Google Cloud. It allows you to define a pipeline that includes both the training step (using Vertex AI Training) and the subsequent deployment step (creating or updating a Vertex AI Endpoint) as a single, repeatable, and monitored workflow. This ensures that after training completes, the model is automatically deployed without manual intervention, leveraging the pipeline's ability to pass artifacts and trigger conditional logic.

Answer

App Engine

Answer

Cloud Functions

Answer

Cloud Build

Question 16

A company uses Cloud Composer to orchestrate their ML pipelines. They notice that tasks are being queued but not executed, causing delays. What is the most likely cause?

Accepted Answer

The Airflow worker resources are exhausted. When tasks are queued but not executed, it typically indicates that the Airflow workers have no available slots to pick up new tasks. In Cloud Composer, the Celery executor distributes tasks to workers; if all worker concurrency slots are saturated or the worker node pool is under-provisioned, tasks remain in the 'queued' state until a worker becomes free. This is the most likely cause given the symptom of tasks being queued without execution.

Answer

The Airflow web server is down

Answer

The DAG file is corrupted

Answer

The Cloud Storage bucket containing DAGs is not accessible

Question 17

An ML engineer is using Vertex AI Pipelines and wants to reuse a trained model across multiple pipeline runs without retraining each time. Which artifact management strategy should be used?

Accepted Answer

Use Vertex AI ML Metadata to track and retrieve model artifacts. Vertex AI ML Metadata is the correct artifact management strategy because it is purpose-built for tracking and retrieving model artifacts across pipeline runs. It stores metadata about models, datasets, and other artifacts in a lineage graph, enabling you to query and reuse a specific model version without retraining. This integrates natively with Vertex AI Pipelines, allowing you to pass model artifacts between components and retrieve them by ID or custom properties.

Answer

Store the model in BigQuery as a ML model

Answer

Use Cloud Functions to cache the model

Answer

Save the model to a Cloud Storage bucket and reference by path

Question 18

A team wants to implement CI/CD for their ML models using Cloud Build. They have a pipeline that trains a model and deploys it. What is the best practice for triggering the pipeline when a new commit is pushed to the source repository?

Accepted Answer

Configure a Cloud Build trigger on the source repository (e.g., Cloud Source Repositories, GitHub). Option D is correct because Cloud Build natively supports triggers that automatically start a pipeline when a new commit is pushed to a connected source repository (e.g., Cloud Source Repositories, GitHub, Bitbucket). This is the simplest, most event-driven approach, requiring no polling, custom services, or additional messaging infrastructure. It directly maps the git push event to a build invocation, ensuring near-instantaneous pipeline execution.

Answer

Set up a Cloud Scheduler job to poll the repository periodically

Answer

Deploy a custom web service on App Engine to call Cloud Build API

Answer

Use Pub/Sub to notify Cloud Build of new commits

Question 19

A data-processing pipeline using Dataflow needs to incorporate a custom ML prediction step. The team wants to maintain fast processing and minimize latency. What is the optimal approach?

Accepted Answer

Use a custom ParDo transform in Dataflow that calls Vertex AI Prediction API directly. Option B is correct because using a custom ParDo transform in Dataflow allows the pipeline to call the Vertex AI Prediction API synchronously within each worker, avoiding the overhead of external triggers, intermediate storage, or asynchronous messaging. This keeps the data in-memory and minimizes latency by processing predictions inline with the Dataflow streaming or batch pipeline.

Answer

Write the data to Cloud Storage, trigger a Cloud Function to call the model, and write results back

Answer

Send data to a Pub/Sub topic and have a separate subscriber that runs predictions

Answer

Stream data through Cloud Functions that serve predictions and write to BigQuery

Question 20

A company is using Vertex AI Pipelines with reusable components. They observe that a component that performs hyperparameter tuning is failing intermittently with a 'ResourceExhausted' error. The component is configured with a small custom service account. What is the most likely cause?

Accepted Answer

The service account does not have sufficient quotas or permissions to create the required number of trials or workers. The 'ResourceExhausted' error in Vertex AI Pipelines typically indicates that the component is trying to create more resources (e.g., trials or workers for hyperparameter tuning) than allowed by the assigned service account's quotas or permissions. A small custom service account often has restricted quotas for AI Platform services, such as the number of concurrent trials or training workers, leading to this failure.

Answer

The component code has a bug causing infinite recursion

Answer

The KFP executor is not properly configured

Answer

The pipeline system memory is insufficient for the component