DEA-C01 Data Ingestion and Transformation Practice Test 18 — 15 Questions

Question 1

A data engineer needs to ingest data from an on-premises Oracle database to Amazon S3 daily. The data volume is 500 GB per day, and the network bandwidth is 200 Mbps. The requirement is to minimize the impact on the source database and ensure data integrity. Which combination of AWS services should be used?

Accepted Answer

AWS Database Migration Service (DMS) with S3 as target. Option C is correct because AWS DMS can perform ongoing replication with minimal impact, and S3 as a target is supported. Option A (AWS Glue) is a batch ETL tool but may put load on the source. Option B (AWS Data Pipeline) is older and less efficient. Option D (Amazon Kinesis) is for streaming, not batch database loads.

Answer

AWS Glue ETL jobs with JDBC connection

Answer

Amazon Kinesis Data Firehose with Oracle as source

Answer

AWS Data Pipeline with SQLActivity

Question 2

A company uses Amazon Kinesis Data Firehose to deliver data to Amazon S3. The Firehose delivery stream has a buffer size of 64 MB and a buffer interval of 300 seconds. The data volume is 1 GB per minute, and the average record size is 1 KB. The data must be delivered to S3 within 5 minutes of ingestion. The engineer notices that some files are being delivered after 10 minutes. What is the most likely cause?

Accepted Answer

The S3 bucket is throttling PUT requests due to high throughput. Option A is correct because with 1 KB records and 64 MB buffer, it takes ~64,000 records to fill the buffer. At 1 GB/min (~1,000,000 records/min), the buffer fills in ~3.84 seconds, so the 300-second interval is not the bottleneck. However, if the S3 bucket has a PUT request rate limit, Firehose may retry and cause delays. Option B is incorrect because 64 MB is default, not too small. Option C is incorrect because 300 seconds is actually long, but data volume is high enough to trigger delivery sooner. Option D is incorrect because compression would reduce size, not cause delays.

Answer

The buffer size of 64 MB is too small for the data volume

Answer

The data is not compressed, causing larger file sizes

Answer

The buffer interval of 300 seconds is too long

Question 3

A data engineering team needs to transform CSV files to Parquet format after they land in an S3 bucket. The transformation should be triggered automatically as soon as a new file arrives. Which AWS service is best suited for this task?

Accepted Answer

AWS Lambda function triggered by S3 event. Option D is correct because AWS Lambda can be triggered by S3 events and convert files to Parquet using libraries like PyArrow. Option A (AWS Glue) is better for scheduled jobs, not event-driven. Option B (Amazon EMR) is overkill. Option C (AWS Batch) is for compute jobs but adds latency.

Answer

AWS Batch job submitted by S3 event

Answer

Amazon EMR cluster running continuously

Answer

AWS Glue ETL job scheduled every 5 minutes

Question 4

A company uses AWS Glue ETL jobs to process data from an S3 data lake. The job reads data in CSV format, transforms it, and writes to Parquet. The job runs daily and takes 2 hours to complete. The data volume is increasing by 20% each month. The engineer wants to reduce the job runtime. Which action is most effective?

Accepted Answer

Partition the input data in S3 by date and use partition pruning in the job. Option B is correct because partitioning data by date (e.g., year/month/day) allows Glue to read only the new partitions incrementally, reducing data scanned. Option A (increasing DPUs) may help but not as much as reducing data volume. Option C (using Spark) is already used by Glue. Option D (compression) is already Parquet, which is compressed.

Answer

Increase the number of DPUs for the Glue job

Answer

Enable compression on the input CSV files

Answer

Switch from Python Shell to Spark ETL

Question 5

A company uses Amazon Kinesis Data Streams to ingest IoT sensor data. The data is processed by an AWS Lambda function that transforms the records and writes to an Amazon S3 bucket. Recently, the Lambda function has been failing with 'Rate exceeded' errors for the S3 PUT API calls. The data volume is 10 MB/s with average record size 2 KB. What should be done to resolve this issue?

Accepted Answer

Add a random prefix to the S3 object key to distribute writes across multiple prefixes. Option A is correct because S3 has a PUT request rate limit of 3,500 objects/s per prefix. With 10 MB/s and 2 KB records, that's 5,000 records/s, exceeding the limit. Increasing the number of S3 prefixes distributes writes across multiple partitions. Option B (increase Lambda concurrency) would worsen the issue. Option C (increase Kinesis shards) doesn't address S3 throttling. Option D (use Firehose) may help but is a bigger change; adding prefixes is simpler.

Answer

Switch to Amazon Kinesis Data Firehose to write to S3

Answer

Increase the Lambda function's reserved concurrency

Answer

Increase the number of Kinesis shards

Question 6

A company needs to ingest data from multiple SaaS applications (Salesforce, Marketo) and load it into Amazon Redshift. The data must be transformed before loading. Which AWS service should be used to build the ingestion pipelines?

Accepted Answer

AWS Glue (crawlers and ETL jobs). Option A is correct because AWS Glue can connect to various sources via JDBC and perform ETL transformations. Option B (AppFlow) is for ingesting SaaS data but does not have built-in transformation capabilities; it can only perform simple field mappings. Option C (DMS) is for database migrations, not SaaS. Option D (Data Pipeline) is older and less flexible.

Answer

AWS Database Migration Service (DMS)

Answer

AWS Data Pipeline

Answer

Amazon AppFlow

Question 7

A data engineer is using Amazon EMR to transform large datasets stored in S3. The cluster runs once a day and takes 3 hours. The engineer notices that the cluster is idle for 30 minutes at the start while waiting for resources. What is the most cost-effective way to reduce the idle time?

Accepted Answer

Configure a larger initial core instance count and enable managed scaling. Option C is correct because using a cluster with managed scaling and a larger initial core node count can reduce the time waiting for resource allocation. Option A (spot instances) may cause interruptions. Option B (reserved instances) is for long-term use, not transient clusters. Option D (increasing instance size) may reduce processing time but not idle time.

Answer

Increase the instance type size

Answer

Use Spot Instances for all nodes

Answer

Purchase Reserved Instances for the cluster

Question 8

A company uses AWS Glue DataBrew for data preparation. The data source is an S3 bucket with millions of small CSV files (each < 1 MB). The DataBrew project takes a long time to load the sample data. What is the most likely cause and solution?

Accepted Answer

The large number of small files causes S3 LIST overhead; concatenate files into larger files. Option B is correct because DataBrew samples data by reading files, and many small files cause high overhead due to S3 LIST and GET requests. Concatenating files into fewer larger files reduces this overhead. Option A (increase DPUs) does not help with the LIST overhead. Option C (use Glue ETL) is a different service. Option D (use Athena) is for querying, not data preparation.

Answer

Use Amazon Athena to query the data instead of DataBrew

Answer

The DataBrew job is under-provisioned; increase the number of DPUs

Answer

Use AWS Glue ETL instead of DataBrew for this volume

Question 9

A company ingests IoT data into an S3 bucket using AWS IoT Core rules. The data is in JSON format, and each record is about 500 bytes. The data volume is 5 GB per day. The company wants to convert the data to Parquet format and partition it by year/month/day. Which TWO AWS services can be used together to achieve this with minimal operational overhead?

Accepted Answer

AWS Glue ETL job triggered by S3 event. Options B and E are correct. S3 Events can trigger Lambda, and Lambda can convert JSON to Parquet and write to partitioned paths. AWS Glue can also be triggered by S3 events via a workflow, but the combination of Lambda + Glue (option E) works as Glue can read from the landing zone and write partitioned Parquet. Option A (Kinesis Firehose) can convert to Parquet but cannot partition by year/month/day easily. Option C (EMR) requires cluster management. Option D (Athena) is for querying, not transformation.

Answer

Amazon Athena CTAS query

Answer

Amazon EMR with Spark job

Answer

Amazon Kinesis Data Firehose with Parquet conversion

Question 10

A company runs a real-time analytics platform using Amazon Kinesis Data Streams. The data is consumed by multiple consumers: one for real-time dashboard (using Lambda) and one for long-term storage (using Firehose to S3). The Kinesis stream has 10 shards. Each record is 1 KB, and the total incoming data rate is 5 MB/s. The Lambda consumer is falling behind and processing latency exceeds 10 seconds. Which TWO actions should be taken to resolve the issue?

Accepted Answer

Increase the number of shards to 20. Options A and D are correct. Increasing the number of shards to 20 doubles the throughput capacity, allowing Lambda to process more records per second. Using enhanced fan-out eliminates contention between consumers, giving each consumer dedicated read throughput. Option B (increase Lambda memory) may help but limited by shard throughput. Option C (decrease batch size) would increase number of invocations, possibly worsening latency. Option E (use KCL) is already used by Lambda.

Answer

Increase the Lambda function's memory allocation

Answer

Switch to using Kinesis Client Library (KCL) instead of Lambda

Answer

Decrease the batch size in the Lambda event source mapping

Question 11

A company is designing a data ingestion pipeline for real-time clickstream data. The data must be ingested with low latency (< 1 second) and then processed for real-time analytics. The processed data should be stored in Amazon S3 for batch analytics. Which THREE services should be used together?

Accepted Answer

Amazon Kinesis Data Analytics. Options A, B, and D are correct. Kinesis Data Streams ingests data with low latency. Kinesis Data Analytics processes streaming data in real-time. Kinesis Data Firehose delivers processed data to S3. Option C (AWS Glue) is batch-oriented. Option E (Amazon MSK) is an alternative but not necessary if Kinesis is used.

Answer

Amazon Managed Streaming for Apache Kafka (MSK)

Answer

AWS Glue ETL job

Question 12

A financial services company ingests real-time stock trade data from multiple exchanges into Amazon Kinesis Data Streams. Each trade record is a JSON object with fields: trade_id, symbol, price, quantity, timestamp. The stream has 5 shards. The data is consumed by an AWS Lambda function that aggregates trades per symbol every minute and writes the results to an Amazon DynamoDB table for a real-time dashboard. Recently, the dashboard has been showing outdated data, and the Lambda function is experiencing high error rates. The CloudWatch logs show 'ProvisionedThroughputExceededException' errors from DynamoDB. The DynamoDB table has 10 read capacity units (RCU) and 10 write capacity units (WCU). The average trade volume is 5,000 trades per second across all symbols, and there are 100 symbols. The Lambda function is configured with a batch size of 100 and a 1-minute window. The data volume is expected to double in the next month. As a data engineer, what is the most appropriate course of action?

Accepted Answer

Increase the DynamoDB write capacity units to 100 and enable auto scaling. Option C is correct because the DynamoDB table is throttling due to insufficient write capacity. With 5,000 trades/s and updating per symbol per minute, the write rate is about 100 writes per minute (one per symbol), but the aggregation may cause bursts. However, the 'ProvisionedThroughputExceededException' indicates WCU is too low. Increasing WCU to 100 resolves the immediate issue; auto scaling handles future growth. Option A (increase shards) addresses Lambda concurrency but not DynamoDB throttling. Option B (use S3) changes the architecture and loses real-time capabilities. Option D (use Firehose) is for delivery to S3, not real-time dashboard.

Answer

Switch the storage from DynamoDB to Amazon S3 and use Amazon Athena for the dashboard

Answer

Increase the number of Kinesis shards to 10 to increase Lambda concurrency

Answer

Use Amazon Kinesis Data Firehose to deliver data to S3 and use Amazon QuickSight for the dashboard

Question 13

A retail company uses AWS Glue ETL jobs to process sales data from an S3 data lake. The source data is partitioned by year/month/day in CSV format. The Glue job reads the latest day's data, performs transformations (e.g., cleaning, aggregating), and writes the results to a separate S3 bucket. The job runs on a schedule every day at 2 AM. Recently, the job has been failing intermittently with the error 'AnalysisException: Path does not exist: s3://source-bucket/year=2024/month=02/day=30/'. The engineer verifies that the folder 'day=30' does not exist because February has only 28 days in 2024. The job is reading data from a hardcoded path. The company expects the job to handle variable days per month automatically. What should the engineer do to fix the issue?

Accepted Answer

Use a Glue crawler to populate the Data Catalog and use dynamic frame from_catalog with partition predicates. Option B is correct because using a Glue crawler to update the partition metadata and then using dynamic frame with from_catalog allows Glue to automatically discover all existing partitions. This eliminates the need for hardcoded paths. Option A (try-catch) is a workaround but not a proper solution. Option C (increase retries) does not fix the root cause. Option D (use Spark SQL with manual partition pruning) still requires knowing which partitions exist.

Answer

Modify the script to use Spark SQL with manual partition pruning based on current date

Answer

Add a try-catch block in the script to skip missing partitions

Answer

Increase the job's retry count and set a timeout

Question 14

A startup is building a data pipeline to ingest user activity logs from a mobile app. The logs are sent in real-time via HTTP POST requests. The data volume is low (a few hundred requests per second) but can spike to a few thousand during promotions. The team wants to store the logs in Amazon S3 for analysis. They also need to be able to query the data using Amazon Athena with minimal latency. The data must be transformed from JSON to Parquet and partitioned by date. The team is considering using Amazon API Gateway with AWS Lambda to receive the logs and write to S3. However, they are concerned about Lambda cold starts and the complexity of handling spikes. Which alternative solution should they choose?

Accepted Answer

Use Amazon Kinesis Data Firehose with a HTTP endpoint as source, enable Parquet conversion, and deliver to S3 with dynamic partitioning. Option A is correct because Kinesis Data Firehose can be used as a HTTP endpoint (via API Gateway or directly with Firehose API), automatically buffers data, converts to Parquet, and writes to S3 with partitioning by date. This handles spikes without custom code. Option B (Lambda + SQS) adds complexity and still faces cold starts. Option C (EMR) is overkill. Option D (Kinesis Data Streams + Lambda) still requires Lambda and has cold start issues.

Answer

Use Amazon API Gateway with AWS Lambda that sends logs to Amazon SQS, then a separate Lambda reads from SQS and writes to S3

Answer

Use Amazon Kinesis Data Streams with AWS Lambda to process and write to S3

Answer

Use Amazon EMR with Spark Streaming to ingest logs from a custom endpoint

Question 15

A company is streaming IoT data from thousands of devices into Amazon Kinesis Data Streams. The data must be transformed in real time before being stored in Amazon S3. Which service should be used to perform the transformation as the data streams through Kinesis?

Accepted Answer

Amazon Kinesis Data Analytics for Apache Flink. Option A is correct because Amazon Kinesis Data Analytics for Apache Flink can process streaming data in real time using Flink, which is ideal for transformations. Option B (Lambda) can process records but is better for lightweight transformations and may not handle complex stateful operations. Option C (Glue) is batch-oriented, not real-time. Option D (EMR) is for big data processing but adds latency.

Answer

AWS Glue

Answer

Amazon EMR

Answer

AWS Lambda