DEA-C01 Data Ingestion and Transformation Practice Test 19 — 15 Questions

Question 1

A company uses AWS Glue ETL jobs to process data from an S3 data lake. The job reads data in CSV format, transforms it, and writes to Parquet. The job runs daily and takes 2 hours to complete. The data volume is increasing by 20% each month. The engineer wants to reduce the job runtime. Which action is most effective?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Partition the input data in S3 by date and use partition pruning in the job. Most effective because partitioning the input data by date and using partition pruning allows the Glue ETL job to read only the relevant partitions instead of scanning the entire S3 data lake. This drastically reduces the amount of data processed, which directly addresses the growing data volume and shortens job runtime. Partition pruning is a core optimization for Spark-based Glue jobs, as it leverages Hive-style partitioning to skip unnecessary files.

Answer

Increase the number of DPUs for the Glue job

Answer

Enable compression on the input CSV files

Answer

Switch from Python Shell to Spark ETL

Question 2

A company uses Amazon Kinesis Data Streams to ingest IoT sensor data. The data is processed by an AWS Lambda function that transforms the records and writes to an Amazon S3 bucket. Recently, the Lambda function has been failing with 'Rate exceeded' errors for the S3 PUT API calls. The data volume is 10 MB/s with average record size 2 KB. What should be done to resolve this issue?

Accepted Answer

Add a random prefix to the S3 object key to distribute writes across multiple prefixes. The 'Rate exceeded' error for S3 PUT API calls indicates that the Lambda function is hitting S3 request rate limits. S3 buckets have a default limit of 3,500 PUT requests per second per prefix. With a data volume of 10 MB/s and an average record size of 2 KB, the Lambda function is generating approximately 5,000 PUT requests per second (10 MB/s ÷ 2 KB), which exceeds the per-prefix limit. Adding a random prefix to the S3 object key distributes writes across multiple prefixes, effectively increasing the aggregate request rate limit to 3,500 PUT requests per second per prefix, thereby resolving the throttling issue.

Answer

Switch to Amazon Kinesis Data Firehose to write to S3

Answer

Increase the Lambda function's reserved concurrency

Answer

Increase the number of Kinesis shards

Question 3

A company needs to ingest data from multiple SaaS applications (Salesforce, Marketo) and load it into Amazon Redshift. The data must be transformed before loading. Which AWS service should be used to build the ingestion pipelines?

Accepted Answer

AWS Glue (crawlers and ETL jobs). AWS Glue is the correct choice because it provides a fully managed ETL service that can connect to various data sources (including SaaS applications via JDBC or custom connectors), transform the data using Apache Spark or Python scripts, and load it into Amazon Redshift. Glue crawlers can catalog the source schemas, and Glue ETL jobs handle the transformation logic required before loading into Redshift, making it ideal for building ingestion pipelines from multiple SaaS sources.

Answer

AWS Database Migration Service (DMS)

Answer

AWS Data Pipeline

Answer

Amazon AppFlow

Question 4

A data engineer is using Amazon EMR to transform large datasets stored in S3. The cluster runs once a day and takes 3 hours. The engineer notices that the cluster is idle for 30 minutes at the start while waiting for resources. What is the most cost-effective way to reduce the idle time?

Accepted Answer

Configure a larger initial core instance count and enable managed scaling. Enabling managed scaling allows the EMR cluster to automatically adjust its core instance count based on workload demands, reducing the initial idle time caused by waiting for resource allocation. By configuring a larger initial core instance count, the cluster can start processing immediately with sufficient capacity, and managed scaling then optimizes resource usage throughout the job, making it the most cost-effective solution for a daily 3-hour transformation job.

Answer

Increase the instance type size

Answer

Use Spot Instances for all nodes

Answer

Purchase Reserved Instances for the cluster

Question 5

A company uses AWS Glue DataBrew for data preparation. The data source is an S3 bucket with millions of small CSV files (each < 1 MB). The DataBrew project takes a long time to load the sample data. What is the most likely cause and solution?

Accepted Answer

The large number of small files causes S3 LIST overhead; concatenate files into larger files. DataBrew loads a sample of the data by listing objects in the S3 bucket. With millions of small CSV files, the S3 LIST API call becomes a bottleneck because each list operation has a 1000-object limit per response, requiring multiple paginated requests. Concatenating the small files into larger files reduces the number of objects, dramatically decreasing LIST overhead and speeding up sample loading.

Answer

Use Amazon Athena to query the data instead of DataBrew

Answer

The DataBrew job is under-provisioned; increase the number of DPUs

Answer

Use AWS Glue ETL instead of DataBrew for this volume

Question 6

A company ingests IoT data into an S3 bucket using AWS IoT Core rules. The data is in JSON format, and each record is about 500 bytes. The data volume is 5 GB per day. The company wants to convert the data to Parquet format and partition it by year/month/day. Which TWO AWS services can be used together to achieve this with minimal operational overhead?

Accepted Answer

AWS Glue ETL job triggered by S3 event. (AWS Glue ETL job triggered by S3 event) is correct because it provides a serverless, fully managed ETL solution that can convert JSON to Parquet and partition by year/month/day with minimal operational overhead. The S3 event trigger automatically invokes the Glue job when new data arrives, eliminating the need for manual scheduling or infrastructure management.

Answer

Amazon Athena CTAS query

Answer

Amazon EMR with Spark job

Answer

Amazon Kinesis Data Firehose with Parquet conversion

Question 7

A company runs a real-time analytics platform using Amazon Kinesis Data Streams. The data is consumed by multiple consumers: one for real-time dashboard (using Lambda) and one for long-term storage (using Firehose to S3). The Kinesis stream has 10 shards. Each record is 1 KB, and the total incoming data rate is 5 MB/s. The Lambda consumer is falling behind and processing latency exceeds 10 seconds. Which TWO actions should be taken to resolve the issue?

Accepted Answer

Increase the number of shards to 20. Increasing the number of shards from 10 to 20 doubles the stream's read capacity, allowing the Lambda consumer to poll more data per second and reduce backlog. Option C is correct because enabling enhanced fan-out provides each consumer with a dedicated 2 MB/s read throughput per shard, eliminating contention between the Lambda consumer and the Firehose consumer, which is critical when multiple consumers read from the same stream.

Answer

Increase the Lambda function's memory allocation

Answer

Switch to using Kinesis Client Library (KCL) instead of Lambda

Answer

Decrease the batch size in the Lambda event source mapping

Question 8

A company is designing a data ingestion pipeline for real-time clickstream data. The data must be ingested with low latency (< 1 second) and then processed for real-time analytics. The processed data should be stored in Amazon S3 for batch analytics. Which THREE services should be used together?

Accepted Answer

Amazon Kinesis Data Analytics. Amazon Kinesis Data Analytics is correct because it enables real-time processing of streaming data using SQL or Apache Flink, allowing the company to perform real-time analytics on clickstream data with sub-second latency. It directly consumes data from Kinesis Data Streams and can output results to Kinesis Data Firehose for storage in Amazon S3, fulfilling the requirement for both real-time analytics and batch analytics storage.

Answer

Amazon Managed Streaming for Apache Kafka (MSK)

Answer

AWS Glue ETL job

Question 9

A financial services company ingests real-time stock trade data from multiple exchanges into Amazon Kinesis Data Streams. Each trade record is a JSON object with fields: trade_id, symbol, price, quantity, timestamp. The stream has 5 shards. The data is consumed by an AWS Lambda function that aggregates trades per symbol every minute and writes the results to an Amazon DynamoDB table for a real-time dashboard. Recently, the dashboard has been showing outdated data, and the Lambda function is experiencing high error rates. The CloudWatch logs show 'ProvisionedThroughputExceededException' errors from DynamoDB. The DynamoDB table has 10 read capacity units (RCU) and 10 write capacity units (WCU). The average trade volume is 5,000 trades per second across all symbols, and there are 100 symbols. The Lambda function is configured with a batch size of 100 and a 1-minute window. The data volume is expected to double in the next month. As a data engineer, what is the most appropriate course of action?

Accepted Answer

Increase the DynamoDB write capacity units to 100 and enable auto scaling. The DynamoDB table is throttling due to insufficient write capacity. With 5,000 trades/s and updating per symbol per minute, the write rate is about 100 writes per minute (one per symbol), but the aggregation may cause bursts. However, the 'ProvisionedThroughputExceededException' indicates WCU is too low. Increasing WCU to 100 resolves the immediate issue; auto scaling handles future growth. Option A (switch to S3 and Athena) changes the architecture and loses real-time capabilities. Option B (increase shards) addresses Lambda concurrency but not DynamoDB throttling. Option D (use Firehose and QuickSight) is for delivery to S3, not real-time dashboard.

Answer

Switch the storage from DynamoDB to Amazon S3 and use Amazon Athena for the dashboard

Answer

Increase the number of Kinesis shards to 10 to increase Lambda concurrency

Answer

Use Amazon Kinesis Data Firehose to deliver data to S3 and use Amazon QuickSight for the dashboard

Question 10

A retail company uses AWS Glue ETL jobs to process sales data from an S3 data lake. The source data is partitioned by year/month/day in CSV format. The Glue job reads the latest day's data, performs transformations (e.g., cleaning, aggregating), and writes the results to a separate S3 bucket. The job runs on a schedule every day at 2 AM. Recently, the job has been failing intermittently with the error 'AnalysisException: Path does not exist: s3://source-bucket/year=2024/month=02/day=30/'. The engineer verifies that the folder 'day=30' does not exist because February has only 28 days in 2024. The job is reading data from a hardcoded path. The company expects the job to handle variable days per month automatically. What should the engineer do to fix the issue?

Accepted Answer

Use a Glue crawler to populate the Data Catalog and use dynamic frame from_catalog with partition predicates. Using a Glue crawler to populate the Data Catalog and then using dynamic frame with from_catalog allows Glue to automatically discover all existing partitions. This eliminates the need for hardcoded paths and handles variable days per month. Option A (Spark SQL with manual partition pruning) still requires manual handling of partitions. Option B (try-catch) is a workaround but does not fix the root cause. Option C (increasing retries) does not address the missing partition issue.

Answer

Modify the script to use Spark SQL with manual partition pruning based on current date

Answer

Add a try-catch block in the script to skip missing partitions

Answer

Increase the job's retry count and set a timeout

Question 11

A startup is building a data pipeline to ingest user activity logs from a mobile app. The logs are sent in real-time via HTTP POST requests. The data volume is low (a few hundred requests per second) but can spike to a few thousand during promotions. The team wants to store the logs in Amazon S3 for analysis. They also need to be able to query the data using Amazon Athena with minimal latency. The data must be transformed from JSON to Parquet and partitioned by date. The team is considering using Amazon API Gateway with AWS Lambda to receive the logs and write to S3. However, they are concerned about Lambda cold starts and the complexity of handling spikes. Which alternative solution should they choose?

Accepted Answer

Use Amazon Kinesis Data Firehose with a HTTP endpoint as source, enable Parquet conversion, and deliver to S3 with dynamic partitioning. Amazon Kinesis Data Firehose is the best choice because it can directly receive HTTP POST requests (via its HTTP endpoint or integrated with API Gateway), automatically buffer incoming data, convert JSON to Parquet, and deliver to S3 with dynamic partitioning by date. This handles traffic spikes without custom code or Lambda cold starts, meeting all requirements with minimal operational overhead. Option A (API Gateway + Lambda → SQS → Lambda) adds complexity and still involves Lambda cold starts. Option C (Kinesis Data Streams + Lambda) also requires Lambda and cold start management. Option D (EMR Spark Streaming) is overkill for this low-volume use case.

Answer

Use Amazon API Gateway with AWS Lambda that sends logs to Amazon SQS, then a separate Lambda reads from SQS and writes to S3

Answer

Use Amazon Kinesis Data Streams with AWS Lambda to process and write to S3

Answer

Use Amazon EMR with Spark Streaming to ingest logs from a custom endpoint

Question 12

A company is streaming IoT data from thousands of devices into Amazon Kinesis Data Streams. The data must be transformed in real time before being stored in Amazon S3. Which service should be used to perform the transformation as the data streams through Kinesis?

Accepted Answer

Amazon Kinesis Data Analytics for Apache Flink. Amazon Kinesis Data Analytics for Apache Flink is the correct choice because it is purpose-built for running Apache Flink applications that can perform real-time transformations, filtering, and enrichment on data streaming through Kinesis Data Streams before outputting the results to destinations like Amazon S3. It integrates natively with Kinesis Data Streams as a source and can write transformed data directly to S3 using a Flink sink, making it ideal for this streaming ETL use case.

Answer

AWS Glue

Answer

Amazon EMR

Answer

AWS Lambda

Question 13

A data engineer is designing a data ingestion pipeline for clickstream data from a mobile app. The data volume varies, with occasional spikes up to 10 MB/s. The pipeline must persist the raw data in Amazon S3 and make it available for near-real-time analytics via Amazon Athena. Which combination of services minimizes cost and operational overhead?

Accepted Answer

Amazon Kinesis Data Firehose with direct delivery to Amazon S3, then Amazon Athena. Amazon Kinesis Data Firehose is the most cost-effective and low-overhead solution for ingesting variable-volume clickstream data (up to 10 MB/s) into Amazon S3 because it is a fully managed service that automatically scales, buffers, and compresses data before delivery. It integrates directly with S3 without requiring custom code or infrastructure management, and the data is immediately queryable by Amazon Athena with no additional transformation steps.

Answer

Amazon Kinesis Data Streams with Amazon Kinesis Data Analytics, then Amazon S3

Answer

Amazon SQS with an Auto Scaling group of EC2 instances writing to Amazon S3

Answer

Amazon Kinesis Data Streams with AWS Lambda for transformation, then Amazon S3

Question 14

A company wants to ingest data from an on-premises Oracle database into Amazon S3 on a daily basis. The data volume is 500 GB per transfer. Which AWS service is most appropriate for this batch ingestion?

Accepted Answer

AWS Glue. AWS Glue is the most appropriate service for this batch ingestion because it is purpose-built for ETL (Extract, Transform, Load) workflows, including connecting to on-premises databases via JDBC, extracting large volumes of data (500 GB per day), and writing it to Amazon S3 in a scheduled, serverless manner. Glue's built-in crawlers and job scheduler handle daily batch runs efficiently without requiring manual infrastructure management, making it ideal for this use case.

Answer

AWS Database Migration Service (DMS)

Answer

AWS Data Pipeline

Answer

Amazon Kinesis Data Firehose

Question 15

A company is using Amazon Kinesis Data Firehose to deliver streaming data to Amazon S3. The data must be transformed from JSON to Parquet format before landing in S3. The transformation logic is simple: convert the JSON schema to Parquet. Which approach meets the requirements with the least operational overhead?

Accepted Answer

Use the built-in data format conversion feature of Firehose with an AWS Glue Data Catalog table. Amazon Kinesis Data Firehose provides a built-in data format conversion feature that can automatically convert incoming JSON data to Parquet format using an AWS Glue Data Catalog table as the schema reference. This approach requires no custom code, no additional infrastructure, and no manual transformation logic, making it the simplest solution with the least operational overhead for a straightforward JSON-to-Parquet conversion.

Answer

Use an AWS Lambda function to transform records to Parquet before sending to Firehose

Answer

Use Amazon Kinesis Data Analytics to convert the stream to Parquet

Answer

Provision an Amazon EMR cluster to convert the data in micro-batches