ADF Triggers: Scheduled, Tumbling, Event-Based

Triggers in Azure Data Factory are the mechanism that initiates a pipeline run. They define when and why a pipeline should execute. Without triggers, pipelines must be started manually or via API calls. The DP-900 exam expects you to understand three trigger types: Schedule Trigger, Tumbling Window Trigger, and Event-based Trigger (specifically Storage Event and Custom Event triggers). Each has distinct characteristics, scheduling semantics, and best-fit scenarios.

A Schedule Trigger is the simplest. It runs a pipeline on a recurring schedule, similar to a cron job. You define a start time, end time (optional), and recurrence pattern (e.g., every 5 minutes, hourly, daily, weekly). The trigger does not wait for the previous run to finish before starting the next. If a pipeline takes longer than the interval, multiple runs can overlap. This is the default behavior and is acceptable for idempotent pipelines.

Key properties: - Recurrence: Defined using a cron expression or a visual builder. Minimum interval is 1 minute. - Start time: The first time the trigger fires. - End time: Optional. If omitted, the trigger runs indefinitely. - Time zone: Must be specified. Default is UTC. - Pipeline parameters: Can pass static values or dynamic expressions using @trigger().scheduledTime.

Example cron expression for every 15 minutes: 0 */15 * * * * (seconds, minutes, hours, day of month, month, day of week).

Behavior: - If the pipeline fails, the trigger does not automatically retry. You must configure retry policies on the pipeline activity. - If the trigger is paused, all future runs are skipped. When resumed, it does not backfill missed intervals. - Overlapping runs are allowed. This can cause contention if the pipeline writes to the same resource.

A Tumbling Window Trigger is a stateful trigger that runs pipelines in fixed-size, non-overlapping time intervals (windows). It is designed for scenarios where you need to process data in discrete chunks, such as hourly aggregations. Unlike Schedule Trigger, Tumbling Window ensures that each window runs exactly once and in order. It can also automatically retry failed windows and backfill past windows.

Key properties: - Window size: Must be between 5 minutes and 14 days. Can be expressed in minutes, hours, or days. - Start time: The beginning of the first window. The trigger will fire at end of each window (i.e., at start + window size). - End time: Optional. If omitted, runs indefinitely. - Delay: A - negative offset to delay the start of processing. For example, a 1-hour window with a 5-minute delay will fire at 1:05 AM for the 12:00 AM-1:00 AM window. This allows late-arriving data. - Max concurrency: Limits how many windows can run simultaneously. Default is 1 (sequential). You can increase to process windows in parallel. - Retry policy: Can automatically retry a failed window up to a specified number of times, with a backoff interval. - Dependency: Can depend on the successful completion of another tumbling window trigger (e.g., process 1-hour windows only after the corresponding 5-minute windows are done).

How it works internally: 1. The trigger divides time into fixed windows starting from the start time. 2. At the end of each window (plus any delay), the trigger fires a pipeline run for that window. 3. The pipeline receives the window start and end times as parameters (@trigger().outputs.windowStartTime and @trigger().outputs.windowEndTime). 4. If the pipeline fails, the trigger marks the window as failed and can retry based on its retry policy. 5. If the trigger is paused and later resumed, it will backfill all windows from the last successful run to the current time (unless you specify a different backfill behavior).

Example: - Window size: 1 hour - Start time: 2024-01-01T00:00:00Z - Delay: 5 minutes - The first window (00:00-01:00) fires at 01:05. The second (01:00-02:00) fires at 02:05, etc.

Event-based triggers respond to events in Azure services. The two main types are Storage Event Trigger and Custom Event Trigger.

Storage Event Trigger: - Fires when a blob is created, deleted, or updated in an Azure Storage account (Blob Storage or Azure Data Lake Storage Gen2). - Uses Azure Event Grid to listen for events. - The trigger passes the blob path (@triggerBody().folderPath and @triggerBody().fileName) to the pipeline, allowing it to process that specific file. - Supports filtering by blob name prefix and suffix (e.g., only process .csv files in the incoming folder). - Does not support ordering guarantees. If multiple blobs arrive simultaneously, the trigger fires for each event, but the pipeline runs may execute in any order. - The trigger does not automatically retry on failure. You must configure retry on the pipeline. - Limitations: The storage account must be in the same region as the data factory. The trigger cannot be used with Azure Data Lake Storage Gen1.

Custom Event Trigger: - Processes custom events from Azure Event Grid topics. - You define the event schema and the trigger fires when a matching event is published. - Useful for integrating with custom applications or third-party systems.

How Event Grid integration works: 1. ADF registers an event subscription on the storage account or custom topic. 2. When an event occurs (e.g., blob created), Event Grid sends an HTTP POST to ADF's trigger endpoint. 3. ADF validates the event and starts a pipeline run, passing the event data as parameters. 4. The pipeline executes and can use the file path to read the blob.

The DP-900 exam will ask you to identify which trigger type to use in a given scenario. Key differentiators: - Schedule Trigger: Fixed time schedule, no state, overlapping allowed. Use for periodic batch jobs that don't depend on data arrival. - Tumbling Window Trigger: Stateful, non-overlapping windows, order guarantee, retry and backfill. Use for time-series data processing where you need exactly-once processing per window. - Event-based Trigger: Reacts to events. Use for file arrival scenarios, real-time processing, or when you need to process data as soon as it appears.

Common exam traps:

Scenario 1: Hourly Sales Aggregation A retail company processes sales transactions every hour. They need to aggregate sales data from raw JSON files into a summary table. Using a Tumbling Window trigger with a 1-hour window size ensures each hour's data is processed exactly once. They set a 10-minute delay to accommodate late-arriving transactions. If a window fails, the trigger retries up to 3 times with a 5-minute interval. In production, they run 24 windows per day, each processing about 10 GB of data. Monitoring shows that less than 1% of windows require retries, usually due to transient storage errors. Misconfiguration: If they used a Schedule Trigger instead, overlapping runs could corrupt the summary table because two runs might try to write to the same partition simultaneously.

Scenario 2: Real-time File Ingestion A financial services firm receives trade files from multiple exchanges. They need to process each file as soon as it arrives. They deploy an Event-based trigger on a Blob Storage container. The trigger filters for .csv files with prefix trades/. Each file triggers a pipeline that validates, transforms, and loads the data into Azure SQL Database. In production, hundreds of files arrive per minute, so the trigger fires rapidly. However, the pipeline must be idempotent because if the same file is uploaded twice (e.g., due to retry), it will be processed again. They also set a high concurrency limit on the pipeline to handle bursts. A common mistake is using a Schedule Trigger to poll the container every minute, which introduces latency and misses files that arrive between polls.

Scenario 3: Daily Batch ETL A media company runs a nightly ETL job that processes the previous day's logs. They use a Schedule Trigger set to run once per day at 2:00 AM. The pipeline takes about 30 minutes. If the pipeline fails, they have manual intervention to rerun. Since the job is idempotent (it reads all logs from the previous day), overlapping runs are not an issue. In production, they use a Tumbling Window trigger only when they need to process data in discrete windows (e.g., hourly). For daily full refresh, Schedule Trigger is simpler and sufficient. A common error is using Tumbling Window for a daily job without needing window-based processing, which adds unnecessary complexity.