A manufacturing company has a streaming data pipeline that ingests sensor data from factory equipment into Azure Event Hubs. The data must be prepared for reporting by cleaning invalid records, removing duplicates, and aggregating readings into 5-minute windows. The transformed data needs to be stored in a columnar format in a data lake to support efficient querying by data analysts using SQL. Which Azure service should perform the data transformation and loading?
Azure Stream Analytics is a serverless real-time analytics service that can ingest data from Event Hubs, perform time-windowed aggregations, clean data, and output to Azure Data Lake Storage in the desired columnar format. It is the most straightforward and cost-effective choice for this streaming ETL scenario.
Why this answer
Azure Stream Analytics is the correct choice because it is designed for real-time stream processing, directly consuming data from Azure Event Hubs, performing transformations like cleaning invalid records, removing duplicates, and aggregating over tumbling windows (e.g., 5-minute windows), and outputting the results in a columnar format (e.g., Parquet) to Azure Data Lake Storage. This aligns perfectly with the requirement for a low-latency, continuous transformation pipeline without needing additional orchestration or compute clusters.
Exam trap
The trap here is that candidates often confuse Azure Data Factory or Synapse Pipelines as suitable for streaming transformations because they see 'pipeline' or 'data movement' keywords, but these services are batch-oriented and cannot perform real-time windowed aggregations directly from Event Hubs.
Why the other options are wrong
Azure Data Factory is primarily an orchestration and data movement service, not a real-time stream processing engine. It lacks native capabilities for windowed aggregations, deduplication, and cleaning of streaming data before loading into a data lake.
Azure Databricks is a general-purpose analytics platform for batch and streaming, but for this specific requirement of cleaning, deduplicating, and aggregating streaming data in 5-minute windows with direct output to columnar storage in a data lake, Azure Stream Analytics provides a simpler, fully managed service optimized for real-time stream processing without the overhead of cluster management.
Azure Synapse Pipelines is designed for orchestrating data movement and transformation in batch scenarios, not for real-time streaming transformations like cleaning, deduplication, and windowed aggregation on live sensor data.