DEA-C01 Data Ingestion and Transformation Practice Test 9 — 15 Questions

Question 1

A company uses Amazon S3 Event Notifications to trigger a Lambda function that processes incoming files. Recently, the Lambda function has been timing out for large files (>100 MB). The data engineer wants to improve the pipeline to handle large files reliably. Which solution is the MOST scalable and cost-effective?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Use S3 Event Notification to send to an SQS queue, then have Lambda poll the queue. Decoupling S3 event notifications via an SQS queue allows Lambda to poll messages at its own pace, preventing timeouts from large files. The SQS queue acts as a buffer, enabling Lambda to process files asynchronously and scale based on the queue depth without being constrained by the synchronous S3 trigger timeout (typically 15 minutes for Lambda, but large files can still cause issues with concurrent execution limits). This approach is both scalable and cost-effective, as it avoids idle wait time and allows Lambda to process files in smaller chunks or with longer execution times as needed.

Answer

Use Amazon SNS to fan out the event to multiple Lambda functions

Answer

Use AWS Step Functions to orchestrate multiple Lambda functions for parallel processing

Answer

Increase the Lambda timeout to 15 minutes

Question 2

A data engineer is designing a streaming pipeline that ingests data from an Amazon Kinesis Data Stream (with 5 shards) into Amazon S3. The data must be transformed using a complex stateful operation that cannot be done in a Lambda function (limited to 15 minutes). The engineer needs a solution that can maintain state across multiple records. Which service should be used?

Accepted Answer

Amazon Kinesis Data Analytics for Apache Flink. Amazon Kinesis Data Analytics for Apache Flink is the correct choice because it supports stateful stream processing with exactly-once semantics, can maintain state across multiple records, and has no 15-minute execution limit like AWS Lambda. It natively integrates with Kinesis Data Streams and can sink transformed data to S3, meeting all requirements for complex stateful operations.

Answer

Amazon EMR running Spark Structured Streaming

Answer

Amazon Kinesis Data Firehose with Lambda transformation

Answer

AWS Glue streaming ETL job

Question 3

A company is streaming clickstream data from a website into Amazon Kinesis Data Streams. The data must be transformed in near real-time and stored in Amazon S3 for analytics. Which AWS service should be used to transform the data as it is ingested?

Accepted Answer

Amazon Kinesis Data Analytics. Amazon Kinesis Data Analytics is the correct choice because it can process and transform streaming data in near real-time using SQL or Apache Flink, and then output the transformed data to destinations like Amazon S3. This service is specifically designed for real-time stream processing, making it ideal for transforming clickstream data as it is ingested into Kinesis Data Streams.

Answer

AWS Lambda (streaming function)

Answer

Amazon EMR (Spark Streaming)

Answer

AWS Glue (ETL jobs)

Question 4

A company uses Amazon Kinesis Data Firehose to deliver data to Amazon S3. The Firehose delivery stream has a buffer size of 64 MB and a buffer interval of 300 seconds. The data volume is 1 GB per minute, and the average record size is 1 KB. The data must be delivered to S3 within 5 minutes of ingestion. The engineer notices that some files are being delivered after 10 minutes. What is the most likely cause?

Accepted Answer

The S3 bucket is throttling PUT requests due to high throughput. Amazon S3 buckets have a default limit of 3,500 PUT requests per second per prefix. With a data volume of 1 GB per minute and an average record size of 1 KB, Firehose generates approximately 1,000,000 records per minute, resulting in roughly 16,667 PUT requests per second (since each 64 MB buffer yields about 65,536 records, and 1 GB/min ÷ 64 MB = ~15.6 buffers per minute, each requiring a PUT). This far exceeds the S3 PUT request limit, causing throttling (HTTP 503 Slow Down errors) and delivery delays beyond the 5-minute target.

Answer

The buffer size of 64 MB is too small for the data volume

Answer

The data is not compressed, causing larger file sizes

Answer

The buffer interval of 300 seconds is too long

Question 5

A company needs to ingest data from multiple SaaS applications (Salesforce, Marketo) and load it into Amazon Redshift. The data must be transformed before loading. Which AWS service should be used to build the ingestion pipelines?

Accepted Answer

AWS Glue (crawlers and ETL jobs). AWS Glue is the correct choice because it provides a fully managed ETL service that can connect to various data sources (including SaaS applications via JDBC or custom connectors), transform the data using Apache Spark or Python scripts, and load it into Amazon Redshift. Glue crawlers can catalog the source schemas, and Glue ETL jobs handle the transformation logic required before loading into Redshift, making it ideal for building ingestion pipelines from multiple SaaS sources.

Answer

AWS Database Migration Service (DMS)

Answer

AWS Data Pipeline

Answer

Amazon AppFlow

Question 6

A data engineer is using Amazon EMR to transform large datasets stored in S3. The cluster runs once a day and takes 3 hours. The engineer notices that the cluster is idle for 30 minutes at the start while waiting for resources. What is the most cost-effective way to reduce the idle time?

Accepted Answer

Configure a larger initial core instance count and enable managed scaling. Enabling managed scaling allows the EMR cluster to automatically adjust its core instance count based on workload demands, reducing the initial idle time caused by waiting for resource allocation. By configuring a larger initial core instance count, the cluster can start processing immediately with sufficient capacity, and managed scaling then optimizes resource usage throughout the job, making it the most cost-effective solution for a daily 3-hour transformation job.

Answer

Increase the instance type size

Answer

Use Spot Instances for all nodes

Answer

Purchase Reserved Instances for the cluster

Question 7

A company ingests IoT data into an S3 bucket using AWS IoT Core rules. The data is in JSON format, and each record is about 500 bytes. The data volume is 5 GB per day. The company wants to convert the data to Parquet format and partition it by year/month/day. Which TWO AWS services can be used together to achieve this with minimal operational overhead?

Accepted Answer

AWS Glue ETL job triggered by S3 event. (AWS Glue ETL job triggered by S3 event) is correct because it provides a serverless, fully managed ETL solution that can convert JSON to Parquet and partition by year/month/day with minimal operational overhead. The S3 event trigger automatically invokes the Glue job when new data arrives, eliminating the need for manual scheduling or infrastructure management.

Answer

Amazon Athena CTAS query

Answer

Amazon EMR with Spark job

Answer

Amazon Kinesis Data Firehose with Parquet conversion

Question 8

A company runs a real-time analytics platform using Amazon Kinesis Data Streams. The data is consumed by multiple consumers: one for real-time dashboard (using Lambda) and one for long-term storage (using Firehose to S3). The Kinesis stream has 10 shards. Each record is 1 KB, and the total incoming data rate is 5 MB/s. The Lambda consumer is falling behind and processing latency exceeds 10 seconds. Which TWO actions should be taken to resolve the issue?

Accepted Answer

Increase the number of shards to 20. Increasing the number of shards from 10 to 20 doubles the stream's read capacity, allowing the Lambda consumer to poll more data per second and reduce backlog. Option C is correct because enabling enhanced fan-out provides each consumer with a dedicated 2 MB/s read throughput per shard, eliminating contention between the Lambda consumer and the Firehose consumer, which is critical when multiple consumers read from the same stream.

Answer

Increase the Lambda function's memory allocation

Answer

Switch to using Kinesis Client Library (KCL) instead of Lambda

Answer

Decrease the batch size in the Lambda event source mapping

Question 9

A company is designing a data ingestion pipeline for real-time clickstream data. The data must be ingested with low latency (< 1 second) and then processed for real-time analytics. The processed data should be stored in Amazon S3 for batch analytics. Which THREE services should be used together?

Accepted Answer

Amazon Kinesis Data Analytics. Amazon Kinesis Data Analytics is correct because it enables real-time processing of streaming data using SQL or Apache Flink, allowing the company to perform real-time analytics on clickstream data with sub-second latency. It directly consumes data from Kinesis Data Streams and can output results to Kinesis Data Firehose for storage in Amazon S3, fulfilling the requirement for both real-time analytics and batch analytics storage.

Answer

Amazon Managed Streaming for Apache Kafka (MSK)

Answer

AWS Glue ETL job

Question 10

A financial services company ingests real-time stock trade data from multiple exchanges into Amazon Kinesis Data Streams. Each trade record is a JSON object with fields: trade_id, symbol, price, quantity, timestamp. The stream has 5 shards. The data is consumed by an AWS Lambda function that aggregates trades per symbol every minute and writes the results to an Amazon DynamoDB table for a real-time dashboard. Recently, the dashboard has been showing outdated data, and the Lambda function is experiencing high error rates. The CloudWatch logs show 'ProvisionedThroughputExceededException' errors from DynamoDB. The DynamoDB table has 10 read capacity units (RCU) and 10 write capacity units (WCU). The average trade volume is 5,000 trades per second across all symbols, and there are 100 symbols. The Lambda function is configured with a batch size of 100 and a 1-minute window. The data volume is expected to double in the next month. As a data engineer, what is the most appropriate course of action?

Accepted Answer

Increase the DynamoDB write capacity units to 100 and enable auto scaling. The ProvisionedThroughputExceededException from DynamoDB indicates the table's write capacity is insufficient for the Lambda function's write load. With 5,000 trades per second and a batch size of 100, Lambda processes 50 batches per second, each batch potentially writing aggregated results per symbol. Even if writes are aggregated per symbol per minute, the current 10 WCU is far too low. Increasing WCU to 100 provides immediate headroom, and enabling auto scaling allows the table to adapt to the expected doubling of data volume, preventing throttling and ensuring the dashboard receives timely updates.

Answer

Switch the storage from DynamoDB to Amazon S3 and use Amazon Athena for the dashboard

Answer

Increase the number of Kinesis shards to 10 to increase Lambda concurrency

Answer

Use Amazon Kinesis Data Firehose to deliver data to S3 and use Amazon QuickSight for the dashboard

Question 11

A retail company uses AWS Glue ETL jobs to process sales data from an S3 data lake. The source data is partitioned by year/month/day in CSV format. The Glue job reads the latest day's data, performs transformations (e.g., cleaning, aggregating), and writes the results to a separate S3 bucket. The job runs on a schedule every day at 2 AM. Recently, the job has been failing intermittently with the error 'AnalysisException: Path does not exist: s3://source-bucket/year=2024/month=02/day=30/'. The engineer verifies that the folder 'day=30' does not exist because February has only 28 days in 2024. The job is reading data from a hardcoded path. The company expects the job to handle variable days per month automatically. What should the engineer do to fix the issue?

Accepted Answer

Use a Glue crawler to populate the Data Catalog and use dynamic frame from_catalog with partition predicates. Using a Glue crawler to populate the Data Catalog and then using dynamic frame with from_catalog allows Glue to automatically discover all existing partitions. This eliminates the need for hardcoded paths and handles variable days per month. Option A (Spark SQL with manual partition pruning) still requires manual handling of partitions. Option B (try-catch) is a workaround but does not fix the root cause. Option C (increasing retries) does not address the missing partition issue.

Answer

Modify the script to use Spark SQL with manual partition pruning based on current date

Answer

Add a try-catch block in the script to skip missing partitions

Answer

Increase the job's retry count and set a timeout

Question 12

A startup is building a data pipeline to ingest user activity logs from a mobile app. The logs are sent in real-time via HTTP POST requests. The data volume is low (a few hundred requests per second) but can spike to a few thousand during promotions. The team wants to store the logs in Amazon S3 for analysis. They also need to be able to query the data using Amazon Athena with minimal latency. The data must be transformed from JSON to Parquet and partitioned by date. The team is considering using Amazon API Gateway with AWS Lambda to receive the logs and write to S3. However, they are concerned about Lambda cold starts and the complexity of handling spikes. Which alternative solution should they choose?

Accepted Answer

Use Amazon Kinesis Data Firehose with a HTTP endpoint as source, enable Parquet conversion, and deliver to S3 with dynamic partitioning. Amazon Kinesis Data Firehose is the best choice because it can directly receive HTTP POST requests (via its HTTP endpoint or integrated with API Gateway), automatically buffer incoming data, convert JSON to Parquet, and deliver to S3 with dynamic partitioning by date. This handles traffic spikes without custom code or Lambda cold starts, meeting all requirements with minimal operational overhead. Option A (API Gateway + Lambda → SQS → Lambda) adds complexity and still involves Lambda cold starts. Option C (Kinesis Data Streams + Lambda) also requires Lambda and cold start management. Option D (EMR Spark Streaming) is overkill for this low-volume use case.

Answer

Use Amazon API Gateway with AWS Lambda that sends logs to Amazon SQS, then a separate Lambda reads from SQS and writes to S3

Answer

Use Amazon Kinesis Data Streams with AWS Lambda to process and write to S3

Answer

Use Amazon EMR with Spark Streaming to ingest logs from a custom endpoint

Question 13

A company wants to ingest data from an on-premises Oracle database into Amazon S3 on a daily basis. The data volume is 500 GB per transfer. Which AWS service is most appropriate for this batch ingestion?

Accepted Answer

AWS Glue. AWS Glue is the most appropriate service for this batch ingestion because it is purpose-built for ETL (Extract, Transform, Load) workflows, including connecting to on-premises databases via JDBC, extracting large volumes of data (500 GB per day), and writing it to Amazon S3 in a scheduled, serverless manner. Glue's built-in crawlers and job scheduler handle daily batch runs efficiently without requiring manual infrastructure management, making it ideal for this use case.

Answer

AWS Database Migration Service (DMS)

Answer

AWS Data Pipeline

Answer

Amazon Kinesis Data Firehose

Question 14

A company needs to ingest data from an on-premises Hadoop cluster into Amazon S3 for archival and analysis. The total data volume is 50 TB. The migration must be completed within one week. The on-premises network has a 1 Gbps connection to AWS. Which AWS service should be used?

Accepted Answer

AWS Snowball Edge. AWS Snowball Edge is the correct choice because it is designed for large-scale data transfers (50 TB) over slow or unreliable networks. With a 1 Gbps connection, transferring 50 TB would take approximately 5 days under ideal conditions, but real-world factors (e.g., network congestion, protocol overhead) make it risky to complete within one week. Snowball Edge physically ships the data, bypassing network constraints entirely.

Answer

AWS Transfer Family

Answer

AWS Glue

Answer

AWS DataSync

Question 15

A company is using AWS Glue to run ETL jobs that read from Amazon S3 and write to Amazon Redshift. The jobs are failing intermittently with 'Out of Memory' errors. Which TWO actions should the data engineer take to resolve this issue? (Choose TWO.)

Accepted Answer

Increase the number of DPUs allocated to the Glue job. Increasing the number of DPUs allocated to the Glue job (Option B) directly addresses the 'Out of Memory' errors by providing more memory and compute resources per executor. AWS Glue uses Apache Spark under the hood, where each DPU provides 4 vCPU and 16 GB of memory; adding more DPUs increases the total memory available for data processing, reducing the likelihood of OOM errors during shuffle or aggregation operations.

Answer

Switch the output to Amazon S3 instead of Redshift

Answer

Reduce the number of partitions in the input data

Answer

Increase the spark.sql.shuffle.partitions parameter