# Azure Data Factory

> Source: Courseiva IT Certification Glossary — https://courseiva.com/glossary/azure-data-factory

## Quick definition

Azure Data Factory is a service in Microsoft Azure that helps you move data from one place to another and change it into a useful format. You can think of it as a tool that automates the flow of information between different systems. It works like a central control center where you set up schedules and rules for data movement.

## Simple meaning

Imagine you run a large company with many different departments. The sales team uses one kind of software to track orders, the warehouse uses another to manage inventory, and the finance team has their own system for accounting. Every day, you need to gather data from all these systems, combine it, and produce reports that show the overall health of the business. Without a tool to handle this, someone would have to manually copy data from each system, check for errors, and paste it into a central spreadsheet or database. This is slow, error-prone, and doesn't scale well when you have millions of records.

Azure Data Factory is like hiring a highly skilled automation manager who can connect to each of these systems, pull the data you need, clean it up, transform it into a consistent format, and deliver it to a central storage location, all on a schedule you define. This manager doesn't sleep or make mistakes, and can handle huge amounts of data without breaking a sweat. In technical terms, Data Factory lets you build what are called data pipelines. A pipeline is a series of steps that define where data comes from, what should be done to it, and where it should go.

For example, you might have a pipeline that runs every night at midnight. It connects to an on-premises SQL Server database that holds yesterday's sales transactions, extracts all new records, then moves them to an Azure Blob Storage container. From there, another step in the pipeline might convert the data into a format suitable for analysis, such as Parquet or CSV, and then load it into an Azure Synapse Analytics data warehouse. This entire process is orchestrated and monitored by Data Factory, with built-in error handling and logging.

One of the key ideas behind Data Factory is that it is serverless, meaning you don't have to worry about managing any infrastructure. Microsoft takes care of the underlying servers, networking, and scaling. You simply define what you want to happen, and Data Factory allocates the necessary compute resources to run your pipelines. This makes it ideal for organizations that need to handle unpredictable data volumes without over-provisioning hardware.

Another important concept is that Data Factory uses a visual interface for designing pipelines, which is great for people who are not programmers. You can drag and drop activities onto a canvas, connect them, and configure their properties. Of course, you can also use code, JSON, PowerShell, or Azure CLI, to define and deploy pipelines programmatically if you prefer.

So, in simple terms, Azure Data Factory is a cloud service that automates the movement and transformation of data across various sources and destinations, saving you time and reducing errors.

## Technical definition

Azure Data Factory (ADF) is a fully managed, serverless cloud data integration service on Microsoft Azure that enables the creation, scheduling, and orchestration of Extract, Transform, Load (ETL) and Extract, Load, Transform (ELT) workflows. At its core, ADF provides a platform to build data-driven workflows, called pipelines, that ingest data from a vast array of supported sources, process it using various compute services, and land it in target stores for analytics, reporting, or machine learning.

Architecturally, ADF consists of several key components. The data factory itself is the top-level resource that contains all the pipelines, datasets, linked services, triggers, and integration runtimes. A linked service is essentially a connection string that tells Data Factory how to connect to a particular data source or compute resource. For instance, a linked service to Azure Blob Storage includes the storage account name and access key. Datasets define the structure of the data within the data store, such as which table or folder to use, and the schema (though ADF does not enforce schema at rest, it is schema-on-read). Pipelines are the logical groupings of activities. An activity is a single processing step in a pipeline. Activities can be data movement activities (like Copy Data), data transformation activities (like Data Flow, Hive, or Spark), or control flow activities (like ForEach, If Condition, or Wait).

Integration Runtime (IR) is the compute infrastructure used by ADF. There are three types. Azure Integration Runtime is fully managed and handles data movement between cloud data stores, as well as dispatching data flows. Self-hosted Integration Runtime is installed on on-premises machines or virtual networks to connect to on-premises sources or sources behind a firewall. Azure-SSIS Integration Runtime allows you to lift and shift SQL Server Integration Services (SSIS) packages to run in the cloud. ADF uses the IR to perform activities. For example, a Copy activity uses the Azure IR to read data from Azure SQL Database and write to Azure Data Lake Storage.

Data flows are a powerful feature within ADF that allow you to transform data graphically without writing code. They run on Spark clusters managed by Azure, and you can perform operations like joins, aggregations, window functions, and conditional splitting. Data flows are charged separately based on the compute used.

ADF supports a wide range of connectors: over 100 built-in connectors for SaaS applications (like Salesforce, Dynamics 365), databases (SQL Server, Oracle, MySQL, PostgreSQL), file systems, and Azure services. Connectors use standard protocols such as JDBC, ODBC, REST, and OData. Authentication mechanisms include service principal, managed identity, SQL authentication, OAuth, and connection strings.

Scheduling and execution are driven by triggers. Triggers can be schedule-based (e.g., run every hour) or event-based (e.g., trigger when a new file arrives in Blob Storage). You can also manually trigger pipelines. ADF monitors pipeline runs via the Azure Portal, via its monitoring and management app, or programmatically using SDKs. It provides detailed logging, error messages, and integration with Azure Monitor and Log Analytics for advanced diagnostics.

Security is handled through Azure Active Directory authentication, managed identities, and virtual network service endpoints. Data in transit is encrypted using TLS, and data at rest is encrypted by Azure Storage encryption. You can also use private endpoints for a more secure connection between ADF and other resources.

In real-world IT implementations, ADF is often used to build enterprise data warehouses. For example, an insurance company might use ADF to ingest policy data from multiple legacy systems (SQL Server, mainframes via flat files) into Azure Synapse Analytics. They would set up self-hosted IR to reach on-premises databases, use Copy activities to stage the data in Blob Storage, run Data Flows to clean and join the data, and then load it into the warehouse. They would schedule the pipeline to run daily and set up alerts if the pipeline fails.

ADF is designed to be highly available and fault-tolerant. If an activity fails, you can configure retry policies. For the Copy activity, there is a feature called fault tolerance that allows you to skip incompatible rows instead of failing the entire run. This is especially useful when dealing with messy source data.

A critical aspect for exam candidates is understanding the difference between ADF and Azure Synapse Pipelines (which is built on ADF). While they share the same engine, Synapse Pipelines are integrated into the Synapse Studio for a more data-warehouse-centric experience. Also, note that ADF is not a data storage service; it is an orchestration service. The actual data resides in sources and destinations, and ADF only moves or transforms it.

## Real-life example

Let's imagine you are the event manager for a large music festival. You have ticket sales coming from an online ticketing platform (like Ticketmaster), social media engagement data from Twitter and Instagram, cash register sales from food and beverage vendors, and weather data from a meteorological service. All this information is in different formats and stored in different places. At the end of each day, you need to bring all this data together into a single dashboard to see how many tickets were sold, which food items are most popular, and how the weather is affecting attendance.

Without automation, you would have to manually log into each system, export the data, open it in Excel, copy and paste it into a master spreadsheet, and then create charts. This takes hours and is prone to human error. You might forget to update a row or misformat a date, causing your dashboard to be wrong.

Now imagine you have an assistant named Data Factory. You tell Data Factory to connect to the ticketing platform's API every evening at 9 PM, pull all new sales, and save them as a CSV file in a cloud drive. Then you instruct it to connect to the social media APIs, fetch the day's mentions and engagement metrics, and save them as JSON files in the same cloud drive. Next, you tell it to go to the vendor's point-of-sale system, which is a local database in the festival office, and extract the transaction records. Because that system is not in the cloud, you have a small computer on-site running a special connector (the self-hosted integration runtime) that Data Factory talks to securely. Finally, Data Factory runs a transformation step: it takes all these files, cleans them (removing duplicates and correcting missing data), merges them into a single dataset, and loads that into a cloud database that feeds your dashboard.

All of this happens automatically, with no human intervention. If the weather service API is down, Data Factory retries three times and then sends you an email alert. If a file from the vendor is corrupted, it skips that file but continues processing the rest. In the morning, you wake up to find your dashboard updated perfectly.

This is exactly what Azure Data Factory does in the IT world. It connects to many different data sources, moves the data to a central location, transforms it into a consistent format, and runs on a schedule, all while handling errors gracefully.

## Why it matters

In modern enterprises, data is generated from dozens of sources: customer relationship management systems, enterprise resource planning software, IoT devices, social media, and third-party APIs. The ability to integrate this data into a single, clean, and usable format is critical for making informed business decisions. Without a tool like Azure Data Factory, IT teams would have to write bespoke scripts for each integration point, which is costly to maintain and scale. Data Factory provides a unified, code-first-and-no-code platform that reduces development time and operational overhead.

For IT professionals, understanding Data Factory is important because data integration is a foundational skill in cloud architecture. Companies are moving away from on-premises SQL Server Integration Services (SSIS) and towards cloud-native solutions. Data Factory is Microsoft's primary offering in this space, so cloud architects, data engineers, and DevOps practitioners must be familiar with it. It also integrates deeply with Azure's ecosystem, Azure Data Lake, Azure Synapse, Power BI, and Azure Machine Learning, making it a linchpin in end-to-end analytics solutions.

Data Factory helps enforce data governance and compliance. Since pipelines are defined, monitored, and logged centrally, organizations can audit data lineage and ensure that sensitive data is handled correctly. For example, you can configure Data Factory to mask credit card numbers during a transformation step, and you can track exactly which pipeline touched that data.

## Why it matters in exams

Azure Data Factory appears across multiple cloud certification exams because it is a core data integration service. For the DP-900 (Azure Data Fundamentals) exam, you need to understand the basic concepts of ETL and ELT and know that Data Factory is the Azure service for orchestration. Questions may ask: which service is used to copy data from blob storage to Azure SQL Database? Or which tool can create a data pipeline that runs on a schedule? The focus is on recognition and use cases, not deep configuration.

For the AZ-104 (Azure Administrator) exam, Data Factory appears in the context of managing and monitoring Azure resources. You might see questions about setting up a self-hosted integration runtime on a virtual machine, or about configuring alerts for pipeline failures. Also, because Data Factory uses Managed Identity for authentication, you may need to understand role-based access control (RBAC) configurations.

The DP-203 (Azure Data Engineer) exam goes much deeper. You must know how to design and implement data pipelines using Data Factory, including data flows, integration runtime types, trigger strategies, and optimizing copy performance. Questions often present a scenario: for example, you need to copy millions of rows from an on-premises Oracle database to Azure Data Lake Storage Gen2 with minimal latency. The correct answer might involve using a self-hosted integration runtime with parallel copy and staged copy. Another common topic is the difference between schedule trigger and tumbling window trigger.

For the AWS-certified professionals expanding into Azure, the AWS Cloud Practitioner and AWS Developer Associate exams do not cover Azure services directly, but related glossary terms (like AWS Glue) help you compare. In the cross-cloud context, you must know that Azure Data Factory is the equivalent of AWS Glue, though with different architectural details.

Google Cloud exams (Google ACE and Google Cloud Digital Leader) also have a related concept: Cloud Data Fusion or Dataproc. Understanding Azure Data Factory helps you build a mental model for how cloud ETL works in general.

for Azure-specific exams, focus on: when to use Data Factory vs. Synapse Pipelines, types of integration runtimes, copy activity optimization, data flow capabilities, and security with managed identities.

## How it appears in exam questions

Exam questions about Azure Data Factory fall into several categories. First, scenario-based questions present a business requirement and ask you to choose the correct Azure service or configuration. For example: A company runs a daily process that extracts data from an on-premises SQL Server, transforms it using a Python script, and loads it into Azure Data Lake. Which service should they use? The answer is Azure Data Factory because it can orchestrate the entire ETL process, and you can run a Python activity using Azure Batch or HDInsight.

Second, configuration questions test your knowledge of specific settings. For instance: You need to copy data from an FTP server that does not accept incoming connections from the internet. Which integration runtime should you use? The answer is the self-hosted integration runtime, which can be installed on a server inside the same network as the FTP server.

Third, troubleshooting questions present a failure scenario. For example: A Copy activity fails with a timeout error when trying to read data from an Azure SQL Database. The database is under heavy load from other applications. What is the best way to handle this? Options include increasing the copy concurrency, enabling staged copy, or increasing the database's DTU limit. The best answer is often to enable staged copy, which first writes data to blob storage and then into the database, reducing load on the database.

Fourth, concept comparison questions: What is the difference between a dataset and a linked service in Azure Data Factory? A linked service defines the connection information, while a dataset defines the actual data structure and location within that storage.

Finally, design questions around security: You need to ensure that data movement from an on-premises SQL Server to Azure remains within your corporate network and does not traverse the public internet. Which feature should you use? Answer: Use a self-hosted integration runtime and configure it with Azure ExpressRoute or a VPN connection.

## Example scenario

You work for a retail company that uses an on-premises SQL Server for inventory data and a cloud-based Salesforce instance for sales data. Your manager wants a daily report that combines inventory levels and sales figures, stored in an Azure SQL Database. How would you design this with Azure Data Factory?

First, you would create a Data Factory resource in the Azure portal. You would then create a linked service to your on-premises SQL Server using a self-hosted integration runtime installed on a server within your company's network. Next, you create another linked service to Salesforce using the appropriate connector and OAuth authentication. Then you define two datasets: one pointing to the inventory table in the SQL Server, and another pointing to the Salesforce object that contains sales data.

Now you create a new pipeline. You add a Copy activity that reads from the inventory dataset and writes to an Azure Blob Storage staging area. You add a second Copy activity that reads from the Salesforce dataset and writes to the same staging area. Then you add a Data Flow activity that joins the two sets based on product ID, calculates the remaining stock after subtracting sales, and outputs a new dataset to your Azure SQL Database. You configure the pipeline to run every morning at 6 AM using a schedule trigger. You also set up an alert to email you if the pipeline fails more than once.

This scenario shows how Data Factory can blend on-premises and cloud data, transform it, and push it to a final destination, all automatically.

## Common mistakes

- **Mistake:** Thinking Azure Data Factory stores data permanently.
  - Why it is wrong: Data Factory is an orchestration and integration service, not a data store. It moves and transforms data but does not retain it. The actual data lives in sources and destinations like Azure Blob, SQL Database, or Data Lake.
  - Fix: Remember: ADF is like a conveyor belt in a factory, it moves items but doesn't stock them. The warehouses are the data stores at each end.
- **Mistake:** Using Azure Integration Runtime to connect to on-premises data sources directly.
  - Why it is wrong: The Azure Integration Runtime runs in the cloud and cannot reach data sources behind corporate firewalls without a VPN or ExpressRoute. For on-premises sources, you must use a self-hosted integration runtime installed locally.
  - Fix: Always choose Self-hosted IR for on-premises or private network data sources. For cloud-only sources, use Azure IR.
- **Mistake:** Confusing datasets with linked services.
  - Why it is wrong: A linked service defines the connection string or authentication to a data store (e.g., a specific Azure Storage account). A dataset defines which piece of data within that store you want (e.g., a specific table or folder). They are separate and need to be created distinctly.
  - Fix: Think: linked service = how to connect; dataset = what to connect to.
- **Mistake:** Believing Data Flows run on the Integration Runtime you choose.
  - Why it is wrong: Data Flows always run on Azure Databricks or Azure Synapse Spark clusters managed by ADF, regardless of the Integration Runtime setting. The Integration Runtime for a Data Flow is only used for dispatching the job, not for executing the transformation.
  - Fix: For Data Flows, you must configure a Spark cluster via Azure IR (with a specific cluster size). Self-hosted IR cannot run Data Flows.
- **Mistake:** Overlooking the difference between schedule triggers and tumbling window triggers.
  - Why it is wrong: A schedule trigger runs pipelines at fixed intervals, but it does not guarantee that the pipeline will complete before the next trigger fires. A tumbling window trigger waits for the previous run to complete before starting the next one, and it can also handle stateful windows (e.g., processing hourly data).
  - Fix: Use tumbling window trigger when you need sequential, non-overlapping runs and when you want to ensure data from a specific time window is processed exactly once.

## Exam trap

{"trap":"The question asks: 'You need to run a Python script as part of your data transformation. Which activity should you use?' The options include MapReduce activity, Stored Procedure activity, and a Custom activity. Learners often pick the Custom activity because it sounds like you can run custom code. However, the correct answer is actually the Azure Batch activity or the HDInsight activity, depending on the specific wording-or the Custom activity is actually used to run .NET code, not arbitrary Python. Data Factory does not directly run Python; you need to execute it via Azure Batch or Databricks.","why_learners_choose_it":"The word 'Custom' seems like it can handle any custom code, including Python. Also, learners may not remember that the Custom activity is specifically for .NET, and that Python requires a different compute service.","how_to_avoid_it":"Study the documentation for each activity. Know that for Python, the recommended approach is to use an Azure Databricks activity (running a notebook) or an Azure Batch activity (running a script). The Custom activity is legacy and used for .NET assemblies."}

## Commonly confused with

- **Azure Data Factory vs Azure Synapse Pipelines:** Azure Synapse Pipelines are built on the same engine as Data Factory but are integrated into the Synapse Studio. They are optimized for data warehousing workflows and are often more convenient when you are already using Synapse for analytics. Data Factory is a standalone service that can work with any destination. (Example: If you are moving data directly into Synapse a data warehouse and planning to do all analytics there, you might prefer Synapse Pipelines. If you are sending data to multiple destinations or using other compute services, use Data Factory.)
- **Azure Data Factory vs Azure Logic Apps:** Azure Logic Apps is for workflow automation and integration between SaaS applications, while Data Factory is specialized for data integration and ETL at scale. Logic Apps is better for event-driven, stateless workflows like sending emails or approving requests, whereas Data Factory is designed for heavy data movement and transformation. (Example: If you need to copy a file from an FTP site to Azure Blob and then send an email notification, Logic Apps can do both. If you need to transform a terabyte of data with joins and aggregations, use Data Factory.)
- **Azure Data Factory vs Azure Databricks:** Azure Databricks is a big data analytics platform based on Apache Spark. It can also be used for ETL, but you have to write code in Python, Scala, or SQL. Data Factory provides a visual interface and built-in connectors, and can execute pipelines using Databricks as a compute target. They complement each other. (Example: You could use Data Factory to orchestrate a pipeline that loads data from SQL Server to Data Lake, then call a Databricks notebook for advanced transformations, and then use Data Factory to load the results into a data warehouse.)
- **Azure Data Factory vs Azure SSIS (SQL Server Integration Services):** SSIS is an older on-premises ETL tool. Azure offers SSIS Integration Runtime to run existing SSIS packages in the cloud. Data Factory is a completely different, cloud-native service. They are not the same, but you can lift-and-shift SSIS packages into Data Factory via the Azure-SSIS IR. (Example: If your company has decades-old SSIS packages that you want to move to the cloud without rewriting them, you would use the Azure-SSIS Integration Runtime within Data Factory. If you are building new ETL from scratch, you would use native Data Factory pipelines and data flows.)

## Step-by-step breakdown

1. **Create a Data Factory resource** — In the Azure Portal, you provision a Data Factory instance. This defines the top-level container for all your integration resources. You choose a name, subscription, resource group, and location. The data factory itself does not consume compute until you run pipelines.
2. **Set up linked services** — You create linked services to each data source and destination. For example, a linked service to Azure Blob Storage requires the storage account name and key. For an on-premises SQL Server, you create a linked service using a self-hosted integration runtime that provides the connection string and credentials.
3. **Define datasets** — Datasets reference the linked services and specify which table, file, or folder to work with. For example, a dataset for a SQL Server table includes the table name and schema. Datasets act as pointers to the actual data.
4. **Create a pipeline** — You create a pipeline that contains one or more activities. Pipelines are your workflows. You can add activities like Copy Data, Data Flow, Stored Procedure, and control flow activities like If Condition, ForEach, and Wait.
5. **Add a Copy activity** — You drag a Copy activity onto the pipeline canvas. You configure the source dataset and sink dataset. You can also specify column mappings and choose whether to use staged copy for large datasets. The copy activity uses the Azure Integration Runtime (or self-hosted) to move data.
6. **Add transformations with Data Flow or other compute activities** — For transformations, you can add a Data Flow activity that launches a visual designer to build transformations like joins, aggregations, and pivots. Alternatively, you can add an HDInsight Hive activity, a Databricks notebook activity, or a Stored Procedure activity to transform data using existing code.
7. **Configure triggers** — You attach a trigger to the pipeline. Triggers can be schedule-based (e.g., every hour) or event-based (e.g., when a new blob is created). You can also create tumbling window triggers for exact time windows.
8. **Publish and validate** — Before running, you must publish your changes. Data Factory validates your pipeline for errors, such as missing linked services or mismatched schema. You can also debug interactively during development.
9. **Monitor runs and set alerts** — After publishing, you can monitor pipeline runs in the Azure Portal monitor tab. You can view status, duration, and error messages. You can also set up alerts via Azure Monitor to notify you of failures.

## Practical mini-lesson

When you work with Azure Data Factory in a real job, you quickly learn that the most important decisions revolve around performance and security. For performance, one of the biggest factors is the choice of copy method. The default serial copy is fine for small datasets, but for large ones, you should enable parallel copy. This splits the source data into chunks and copies them concurrently. The number of parallel copies you can set depends on your integration runtime and the source/destination capabilities. However, be mindful that too many parallel copies can overwhelm the source system, causing throttling. A good practice is to start with a moderate degree of copy parallelism (like 4 or 8) and monitor the impact.

Another performance consideration is the use of staged copy. When copying from a source to a sink that has throughput limitations (like Azure SQL Database with low DTU), you can first copy the data to a staging area (like Azure Blob) and then write to the final sink in a second step. This allows you to use PolyBase or COPY INTO for bulk loads, which are much faster.

Security is another critical area. When connecting to on-premises sources, you must install a self-hosted integration runtime on a machine that can access both the data source and the internet (outbound only). You should never store credentials in plain text. Instead, use Azure Key Vault to store secrets like SQL passwords or storage account keys, and reference them in your linked services. Also, for cloud sources, always use Managed Identity when possible, as it eliminates the need to manage credentials.

Monitoring in production is essential. You should set up diagnostic settings to send pipeline run logs to a Log Analytics workspace. This allows you to create custom queries and dashboard to track failures, performance bottlenecks, and data volume. You can also set up alerts so that you are notified immediately when a critical pipeline fails. Common failure reasons include source/destination unavailability, credential expiration, or schema changes in the source data. A good practice is to include error handling in your pipelines using the 'on failure' path with email notifications via Azure Logic Apps.

Finally, version control and DevOps integration is a must for team collaboration. You can use Azure DevOps or GitHub to manage your Data Factory code. Each pipeline and dataset is defined in JSON files that you can store in a git repository. You can then use CI/CD pipelines to deploy changes from development to test to production. This ensures that changes are reviewed and tested before they hit production.

## Commands

```
az datafactory factory create --resource-group myRG --name myADF --location eastus
```
Creates a new Azure Data Factory instance in a specified resource group and location.

*Exam note: Tests understanding of resource creation with required parameters like --resource-group, --name, and --location.*

```
az datafactory linked-service create --resource-group myRG --factory-name myADF --linked-service-name AzureStorageLinkedService --properties @linkedService.json
```
Creates a linked service to connect Azure Data Factory to an external data store (e.g., Azure Blob Storage) using a JSON file for properties.

*Exam note: Exams test knowledge of linked services as connection definitions and the use of --properties with JSON files.*

```
az datafactory pipeline create --resource-group myRG --factory-name myADF --pipeline-name MyPipeline --pipeline @pipeline.json
```
Creates a pipeline in Azure Data Factory defined by a JSON file containing activities and dependencies.

*Exam note: Frequently tested: pipelines are sets of activities; JSON structure defines them in CLI/ARM templates.*

```
az datafactory pipeline run create --resource-group myRG --factory-name myADF --pipeline-name MyPipeline --parameters "@{\"param1\":\"value1\"}"
```
Triggers a manual pipeline run with specified parameter values.

*Exam note: Exams test parameterization in pipeline runs and the --parameters syntax in PowerShell/CLI.*

```
az datafactory trigger create --resource-group myRG --factory-name myADF --trigger-name MyTrigger --properties @trigger.json
```
Creates a trigger (e.g., scheduled or event-based) for pipeline execution, defined in a JSON file.

*Exam note: Triggers are key for automation; exams ask about trigger types like schedule, tumbling window, and event.*

```
az datafactory dataset create --resource-group myRG --factory-name myADF --dataset-name MyDataset --properties @dataset.json
```
Creates a dataset that represents the data structure within a linked service, used by pipelines.

*Exam note: Datasets are inputs/outputs of activities; exams test how they link to linked services and pipelines.*

```
az datafactory integration-runtime create --resource-group myRG --factory-name myADF --integration-runtime-name MyIR --type SelfHosted --properties @ir.json
```
Creates a self-hosted integration runtime for hybrid data movement, typically used with on-premises or VNet data sources.

*Exam note: Integration runtimes (Azure, Self-Hosted, SSIS) are common exam topics, especially for hybrid scenarios.*

## Troubleshooting clues

- **Pipeline run fails with 'UserError' due to data source connection timeout** — symptom: Pipeline run logs show 'ErrorCode=UserError, Details=Failed to connect to source: A connection attempt failed because the connected party did not properly respond after a period of time'. Data source is unreachable due to firewall rules, network restrictions, or incorrect DNS. Azure Data Factory cannot reach the linked service endpoint within the timeout window. (Exam clue: Exams present scenarios where firewall rules block ADF; test-takers must identify adding ADF's IP ranges or using Self-Hosted IR.)
- **Trigger fails to start pipeline on schedule** — symptom: Trigger status shows 'Failed' or pipeline not executed at the specified time, with no activity logs.. Trigger may be in 'Disabled' state, or its JSON definition has invalid schedule parameters (e.g., wrong cron expression or time zone). (Exam clue: Questions test trigger configuration: differences between schedule and tumbling window triggers, and how to enable/disable them.)
- **Data copy activity with Azure Blob Storage fails due to authentication** — symptom: Error code: 'AuthenticationFailed' or '403 Forbidden' when accessing Blob Storage during Copy Data activity.. The linked service uses an invalid or expired account key, or SAS token lacks proper permissions (e.g., read/write). Alternatively, storage firewall blocks ADF. (Exam clue: Exams test use of managed identity or service principal for secure authentication instead of keys; questions on Blob Storage firewall rules.)
- **Self-hosted Integration Runtime is inactive or offline** — symptom: The IR node shows 'Offline' status in ADF portal, and pipelines using it fail with 'Type=IntegrationRuntimeNotAvailable'.. The on-premises machine running the IR is stopped, network connectivity to Azure is lost, or the IR agent is corrupted. (Exam clue: Scenarios where IR needs re-registration or restart; exams test troubleshooting connectivity and high-availability setup.)
- **Pipeline run succeeds but no data in destination** — symptom: Copy activity completes with '0 rows' written, but source data exists. No errors in logs.. Filter applied in dataset or copy activity excludes all rows (e.g., incorrect partition path pattern or query filter). Source data may be empty in selected folder. (Exam clue: Questions on Copy Activity settings like 'CopyBehavior' and 'FilterByLastModified' cause data to be skipped.)
- **Invalid JSON error when deploying pipeline via ARM template** — symptom: Deployment fails with 'InvalidTemplate' error, stating a specific property is missing or wrong type.. The JSON definition of the pipeline, dataset, or linked service has a syntax error, or references an undefined resource (e.g., imported linked service name mismatch). (Exam clue: Exams test common JSON structure mistakes in ADF ARM templates, like missing '$ref' or incorrect type declaration.)
- **Data flow activity runs slowly on compute cluster** — symptom: Data flow execution takes much longer than expected, with high Data Flow Debug times or queueing.. Cluster compute options are misconfigured: too few cores, wrong core count type (e.g., General Purpose vs Memory Optimized), or insufficient Time To Live (TTL) causing cold starts. (Exam clue: Questions on optimizing data flows: selecting right cluster type, increasing TTL, and using optimized partitioning.)
- **Publish button grayed out in ADF UI after editing** — symptom: User cannot save or publish changes; a warning says validation errors exist, but no specific error shown.. A linked service, dataset, or activity has invalid configuration (e.g., missing required field, broken reference) that fails validation check before save. (Exam clue: Exams test how to use 'Validate All' and interpret error messages in ADF UI for pre-deployment checks.)

## Memory tip

Remember: ADF is the 'moving truck' for data, it doesn't store anything, it just picks up and delivers.

## FAQ

**Is Azure Data Factory free?**

No, you pay for the number of operations (activity runs) and the compute used for data flows and integration runtime. There is a free tier with limited monthly operations but it is not suitable for production.

**Can I use Azure Data Factory to move data from AWS S3 to Azure?**

Yes, Data Factory has a connector for Amazon S3. You can create a linked service to S3 using access keys or temporary security tokens, and a pipeline to copy data to Azure Blob or Data Lake.

**What is the difference between a pipeline and a data flow?**

A pipeline is the overall workflow that can contain multiple activities. A data flow is a specific type of activity that allows you to visually transform data using a Spark-based engine, similar to SSIS Data Flow.

**Do I need to code to use Azure Data Factory?**

No, you can build pipelines using the drag-and-drop interface. However, if you want to perform advanced transformations, you may need to write expressions in the Data Flow expression language or use custom code activities.

**Can I run Azure Data Factory pipelines on-premises?**

Yes, you can install a self-hosted integration runtime on an on-premises machine, but the pipeline orchestration happens in the cloud. The actual data movement can be executed locally if the source and sink are on-premises.

**How does Azure Data Factory handle retries and failures?**

You can configure retry count and retry interval at the activity level. You can set up error handling using the 'on failure' path in the pipeline to run alternative activities or send notifications.

**What is the maximum file size that Data Factory can copy?**

Data Factory can handle essentially unlimited size. It uses chunking for large files. For example, you can copy petabytes of data if you use appropriate staging and parallel copy settings.

## Summary

Azure Data Factory (ADF) is Microsoft's flagship cloud data integration service, designed to automate the movement and transformation of data across a vast array of sources and destinations. It is a fully managed, serverless platform that eliminates the need to provision or manage infrastructure, making it an ideal choice for organizations building modern data pipelines. Understanding ADF is crucial for cloud certification seekers, especially those targeting Azure data-focused exams like DP-900 and DP-203, as well as administrators (AZ-104). The service's key components, linked services, datasets, pipelines, activities, integration runtimes, and triggers, form the vocabulary of any data integration discussion. The most common exam pitfalls include confusing linked services with datasets, misunderstanding the role of the self-hosted integration runtime, and choosing the wrong trigger type. ADF is not a storage service; it orchestrates data flow. By mastering its capabilities, you can efficiently build scalable ETL and ELT workflows, and confidently answer scenario-based questions on certification exams that ask how to move data from point A to point B, transform it, and keep it running reliably. Remember that ADF is the mover, not the warehouse.

---

Practice questions and the full interactive page: https://courseiva.com/glossary/azure-data-factory