Automating and Orchestrating ML Pipelines practice questions

dsl.PipelineParam

Why wrong: PipelineParam is a base class for parameters, not for parallel constructs.

dsl.Condition

Why wrong: dsl.Condition does not exist; conditional branching uses dsl.If.

dsl.ExitHandler

Why wrong: ExitHandler runs cleanup code when a pipeline exits, not for parallel processing.

@dsl.component\ndef my_component(input_text: str) -> Metric:\n metrics = Metric()\n metrics.log_metric('length', len(input_text))

Why wrong: Incorrect artifact type: should be Metrics (plural) and base_image missing; also Metric class doesn't exist in kfp.dsl.

@dsl.pipeline\ndef my_pipeline(input_text: str):\n metrics = Metrics()\n metrics.log_metric('length', len(input_text))

Why wrong: Incorrect: @dsl.pipeline is for pipeline definitions, not components. Also missing return type.

def my_component(input_text: str) -> Metrics:\n from kfp.dsl import Metrics\n metrics = Metrics()\n metrics.log_metric('length', len(input_text))\n return metrics

Why wrong: Missing @dsl.component decorator; without it, this is a plain Python function, not a KFP component.

HyperparameterTuningJob

Why wrong: HyperparameterTuningJob is for tuning hyperparameters, not a single training job.

BatchPredictionJob

Why wrong: BatchPredictionJob is for batch predictions, not training.

ModelDeploy

Why wrong: ModelDeploy is for deploying models to endpoints, not for training.

Set the 'caching' flag to 'True' in the pipeline definition using 'pipeline.caching = True'.

Why wrong: There is no such pipeline attribute; caching is per-component and controlled via the component decorator options.

Set the environment variable 'ENABLE_CACHE' to 'true' on the pipeline run request.

Why wrong: There is no such environment variable; caching is controlled by the component definition or pipeline run request parameter.

Re-compile the pipeline with the '--enable-cache' flag.

Why wrong: Compilation does not have a caching flag; caching is a runtime feature.

Cloud Scheduler for the schedule, and Cloud Pub/Sub with Push subscription to Vertex AI for event-driven.

Why wrong: Cloud Pub/Sub cannot directly trigger Vertex AI pipelines; it requires a subscriber like Cloud Functions.

Cloud Functions for both schedule and event-driven, using cron trigger.

Why wrong: Cloud Functions cannot use a cron trigger to respond to new data arriving in a Cloud Storage bucket; cron triggers are time-based, not event-based. While Cloud Functions are suitable for both event-driven processing via Cloud Storage events and scheduled execution when paired with Cloud Scheduler, the option incorrectly suggests a single cron trigger mechanism for both requirements. Cloud Functions are often a correct choice for lightweight, event-driven tasks that need to react to various Google Cloud events or execute on a schedule, acting as the compute layer that then orchestrates other services.

Vertex AI Pipelines built-in scheduler for schedule, and Cloud Pub/Sub for event-driven.

Why wrong: Vertex AI Pipelines does not have a built-in scheduler; it relies on Cloud Scheduler. Also Pub/Sub alone cannot trigger pipelines.

Pass data in-memory as Python objects between components.

Why wrong: In-memory passing is not supported in KFP; data must be serialized and stored externally.

Use BigQuery table names as component outputs and inputs.

Why wrong: BigQuery table names are acceptable but less portable; GCS URIs are more standard for artifact passing.

Use Cloud SQL to store intermediate results and pass connection strings.

Why wrong: Cloud SQL is not designed for intermediate data passing in ML pipelines; adds latency and complexity.

Cloud Tasks

Why wrong: Cloud Tasks is for task queuing and distributed execution, not for complex DAG orchestration.

Cloud Scheduler

Why wrong: Cloud Scheduler is for simple cron jobs, not for orchestrating multi-step pipelines with dependencies.

Workflows

Why wrong: Workflows is a serverless orchestration service but has limited integration with Vertex AI and Dataflow compared to Cloud Composer.

dsl.ParallelFor

Why wrong: dsl.ParallelFor is for parallel iteration, not conditional execution.

dsl.ExitHandler

Why wrong: ExitHandler is for cleanup tasks after pipeline completion, not for conditional branching.

dsl.Collected

Why wrong: dsl.Collected is used to collect results from parallel for loops, not for conditionals.

Add a 'ignoreFiles' field with 'training/**' to the trigger.

Why wrong: ignoreFiles excludes matching files; the team wants to include only training/ changes, not ignore them.

Use a 'substitutions' field with a regex pattern to filter commits.

Why wrong: Substitutions are for variable replacement, not for filtering triggers.

Configure a Cloud Function to check the commit diff and call Cloud Build API conditionally.

Why wrong: This is an overcomplicated approach; Cloud Build trigger filtering natively supports file globs.

A Lightweight Python component without base_image.

Why wrong: Lightweight components are Python functions; they cannot use custom containers.

A Python function component with a custom base_image.

Why wrong: Python function components with base_image still run Python code; they don't allow full custom container images.

A pre-built component from GCPC.

Why wrong: Pre-built components are for standard tasks; they don't allow arbitrary containers.

Write the dataset to a temporary BigQuery table and pass the table name.

Why wrong: Writing a large dataset to a temporary BigQuery table and passing the table name introduces unnecessary latency and cost from BigQuery storage and query overhead, whereas Vertex AI pipelines natively support passing large data objects via cloud storage URIs without an intermediate database. This option is tempting because BigQuery is often used for large-scale analytics and temporary tables are a common pattern for sharing data between SQL-based jobs, making it a correct choice when the downstream component requires SQL querying rather than direct file access.

Serialize the dataset to a string and pass it as a pipeline parameter.

Why wrong: Pipeline parameters are for small values; large data will hit size limits and cause performance issues.

Use a Cloud Storage bucket and pass the bucket name as a parameter.

Why wrong: Passing the bucket name is vague; the pipeline needs specific file paths, which artifacts provide.

The precision value is stored as a float but the condition expects a string.

Why wrong: KFP supports float comparisons in conditions; type mismatch is unlikely.

The evaluation component did not output the metric correctly.

Why wrong: Even if the metric is logged correctly, it doesn't become a pipeline parameter automatically.

The dsl.If block is placed incorrectly in the pipeline definition.

Why wrong: Placement affects flow but not the value; the condition still evaluates based on the parameter.

Cloud Monitoring and Cloud Logging

Why wrong: These are for monitoring infrastructure, not for data skew detection.

Cloud DLP and Cloud KMS

Why wrong: These are for data security and encryption, not for data skew.

BigQuery and Dataflow

Why wrong: BigQuery and Dataflow can be used for data processing but not for built-in skew detection.

Use Cloud Functions to periodically check for new model versions and deploy.

Why wrong: Periodic polling is inefficient; event-driven deployment via pipeline is better.

Manually approve each model version before deployment.

Why wrong: Manual approval contradicts continuous delivery; automated gates are preferred.

Deploy every model version directly to production without evaluation.

Why wrong: Skipping evaluation gate increases risk; not a best practice.

Reduce the size of the file being downloaded.

Why wrong: Reducing file size may help but is not a reliability practice; also may not be feasible.

Increase the machine type for the component.

Why wrong: Machine type affects compute resources, not network timeout; not directly related.

Pipeline parameters must be serialized to JSON before use.

Why wrong: Parameters are natively passed as typed values; no manual serialization needed.

Pipeline parameters can only be of type str.

Why wrong: They can be str, int, float, bool, dict, list, etc.

Pipeline parameters can be used to pass large datasets between components.

Why wrong: Large datasets should use artifacts, not parameters.

dsl.Conditional

Why wrong: dsl.Conditional is not a valid construct in KFP SDK v2.

dsl.ExitHandler

Why wrong: dsl.ExitHandler is used for cleanup actions regardless of success/failure.

dsl.Collected

Why wrong: dsl.Collected is used for parallel loops, not conditional branching.

Use the Vertex AI Pipelines console to set a cron schedule directly on the pipeline.

Why wrong: Vertex AI Pipelines does not have a built-in scheduler; you must use Cloud Scheduler or other external triggers.

Create a Cloud Build trigger that runs the pipeline on a schedule.

Why wrong: Cloud Build triggers are for CI/CD, not scheduling recurring pipelines.

Use Cloud Tasks to create a recurring task that invokes the pipeline.

Why wrong: Cloud Tasks is for distributed task queues, not recurring schedules.

Use the dsl.CacheKey annotation to explicitly set cache keys.

Why wrong: dsl.CacheKey is not a valid KFP SDK v2 construct.

Change the component's image tag to :latest.

Why wrong: Using :latest is not recommended and may not invalidate cache if the digest is unchanged; it also breaks reproducibility.

Add a random integer parameter to the component to vary the inputs.

Why wrong: This would force re-execution but is a hack; the proper way is to disable caching.

Cloud Storage → BigQuery → Vertex AI Pipelines

Why wrong: BigQuery is not an event trigger service.

Cloud Storage → Cloud Build → Vertex AI Pipelines

Why wrong: Cloud Build is not designed for event-driven data triggers; it's for source code changes.

Cloud Storage → Cloud Scheduler → Vertex AI Pipelines

Why wrong: Cloud Scheduler is for time-based schedules, not event-driven triggers.