DEA-C01 Data Ingestion and Transformation Practice Question

Correct answer & explanation

Amazon Kinesis Data Firehose can ingest high-throughput streaming data (200,000 messages/sec) and deliver it to S3 in near-real-time (typically under 60 seconds). By using Redshift Spectrum to query the data directly in S3, the company avoids the latency and memory issues associated with AWS Glue batch transformations and Redshift bulk loads. This approach reduces data availability latency significantly. Option A is incorrect because Amazon Kinesis Data Analytics adds processing overhead and does not directly solve the Glue memory issue or reduce latency to Redshift; it is more suitable for real-time streaming analytics, not for minimizing data ingestion latency. Option B is incorrect because increasing the Redshift cluster size improves query performance and load speed but does not address the root cause: the Glue jobs are failing due to memory issues, and the data is not being transformed quickly enough. The bottleneck is upstream of Redshift. Option D is incorrect because increasing DPUs and memory for the Glue job might resolve memory issues but does not significantly reduce latency; Glue batch processing still incurs minutes of delay, whereas Firehose provides near-real-time delivery.