DEA-C01 Data Ingestion and Transformation Practice Test 19 — 15 Questions

Question 1

A data engineer is designing a data ingestion pipeline for clickstream data from a mobile app. The data volume varies, with occasional spikes up to 10 MB/s. The pipeline must persist the raw data in Amazon S3 and make it available for near-real-time analytics via Amazon Athena. Which combination of services minimizes cost and operational overhead?

Accepted Answer

Amazon Kinesis Data Firehose with direct delivery to Amazon S3, then Amazon Athena. Option C is correct because Firehose can buffer and deliver data to S3 with minimal configuration, and Athena can query directly from S3. Option A (Kinesis Data Streams + Lambda) incurs higher costs and management overhead. Option B (SQS + EC2) requires managing EC2 instances and auto-scaling. Option D (Kinesis Data Streams + Kinesis Data Analytics) adds complexity and cost.

Answer

Amazon Kinesis Data Streams with Amazon Kinesis Data Analytics, then Amazon S3

Answer

Amazon SQS with an Auto Scaling group of EC2 instances writing to Amazon S3

Answer

Amazon Kinesis Data Streams with AWS Lambda for transformation, then Amazon S3

Question 2

A company wants to ingest data from an on-premises Oracle database into Amazon S3 on a daily basis. The data volume is 500 GB per transfer. Which AWS service is most appropriate for this batch ingestion?

Accepted Answer

AWS Glue. Option B is correct because AWS Glue can run scheduled ETL jobs to extract data from JDBC sources like Oracle and write to S3. Option A (DMS) is for ongoing replication, not scheduled batch. Option C (Firehose) is for streaming, not batch. Option D (Data Pipeline) is a legacy service.

Answer

AWS Database Migration Service (DMS)

Answer

AWS Data Pipeline

Answer

Amazon Kinesis Data Firehose

Question 3

A company is using Amazon Kinesis Data Firehose to deliver streaming data to Amazon S3. The data must be transformed from JSON to Parquet format before landing in S3. The transformation logic is simple: convert the JSON schema to Parquet. Which approach meets the requirements with the least operational overhead?

Accepted Answer

Use the built-in data format conversion feature of Firehose with an AWS Glue Data Catalog table. Option A is correct because Firehose can natively convert JSON to Parquet using a schema from AWS Glue Data Catalog, without custom code. Option B (Lambda) requires writing and maintaining code. Option C (Kinesis Data Analytics) is overkill. Option D (EC2) adds management overhead.

Answer

Use an AWS Lambda function to transform records to Parquet before sending to Firehose

Answer

Use Amazon Kinesis Data Analytics to convert the stream to Parquet

Answer

Provision an Amazon EMR cluster to convert the data in micro-batches

Question 4

A data engineer is troubleshooting a Kinesis Data Firehose delivery stream that is experiencing high error rates when writing to an S3 bucket. The error logs indicate 'AccessDenied' errors. The S3 bucket policy allows access from the Firehose service, but the errors persist. What is the most likely cause?

Accepted Answer

The IAM role assumed by Firehose does not have the s3:PutObject permission. Option C is correct because Firehose uses a service-linked IAM role to write to S3; if the role lacks proper permissions, AccessDenied occurs even if the bucket policy allows it. Option A (encryption) would cause different errors. Option B (KMS key) is needed only if SSE-KMS is used. Option D (S3 event notifications) is unrelated.

Answer

The S3 bucket has a lifecycle policy that is deleting objects too quickly

Answer

The S3 bucket has default encryption enabled

Answer

The S3 bucket uses an AWS KMS key for encryption and Firehose does not have kms:Decrypt permission

Question 5

A company needs to ingest data from multiple SaaS sources (e.g., Salesforce, Marketo) into Amazon S3 for analytics. Which AWS service is designed for this purpose?

Accepted Answer

Amazon AppFlow. Option B is correct because AppFlow is a fully managed service for securely transferring data from SaaS applications to AWS. Option A (Glue) can connect to JDBC sources but not directly to many SaaS APIs. Option C (DataSync) is for file-based transfers. Option D (Transfer Family) is for SFTP.

Answer

AWS Transfer Family

Answer

AWS Glue

Answer

AWS DataSync

Question 6

A company is ingesting log files from multiple EC2 instances into Amazon S3 using the CloudWatch agent. The logs are delivered to a CloudWatch Logs group, and a subscription filter sends them to a Lambda function for transformation, then to Firehose. The Firehose stream is configured with a buffer interval of 60 seconds and buffer size of 5 MB. The logs are critical and must be available in S3 within 5 minutes. What is the most cost-effective way to reduce the delivery latency?

Accepted Answer

Decrease the buffer interval to 10 seconds. Option D is correct because reducing the buffer interval from 60 to 10 seconds directly reduces latency without adding significant cost. Option A (increase buffer size) would increase latency. Option B (increase buffer interval) worsens latency. Option C (change to Kinesis Data Streams) adds cost and complexity.

Answer

Replace Firehose with Amazon Kinesis Data Streams

Answer

Increase the buffer size to 10 MB

Answer

Increase the buffer interval to 120 seconds

Question 7

A data engineer is designing a data ingestion pipeline for real-time financial transactions. The pipeline must ensure exactly-once processing semantics and must handle duplicate records that may occur due to retries. Which combination of AWS services can achieve exactly-once processing?

Accepted Answer

Amazon Kinesis Data Streams with Amazon Kinesis Data Analytics for Apache Flink. Option A is correct because Kinesis Data Streams provides sequence numbers, and Kinesis Data Analytics can use Flink's exactly-once semantics with idempotent sinks. Option B (Firehose) provides at-least-once. Option C (SQS) provides at-least-once. Option D (MSK) can achieve exactly-once but requires more configuration and is not as straightforward.

Answer

Amazon SQS with AWS Lambda

Answer

Amazon MSK with AWS Lambda

Answer

Amazon Kinesis Data Firehose with AWS Lambda

Question 8

A company needs to ingest data from an on-premises Hadoop cluster into Amazon S3 for archival and analysis. The total data volume is 50 TB. The migration must be completed within one week. The on-premises network has a 1 Gbps connection to AWS. Which AWS service should be used?

Accepted Answer

AWS Snowball Edge. Option C is correct because AWS Snowball is a physical device that can transfer large amounts of data faster than over the network given the bandwidth limitation. Option A (DataSync) can use the network but may not complete within a week due to bandwidth constraints. Option B (Glue) is for ETL, not raw transfer. Option D (Transfer Family) is for SFTP.

Answer

AWS Transfer Family

Answer

AWS Glue

Answer

AWS DataSync

Question 9

A company is using AWS Glue to run ETL jobs that read from Amazon S3 and write to Amazon Redshift. The jobs are failing intermittently with 'Out of Memory' errors. Which TWO actions should the data engineer take to resolve this issue? (Choose TWO.)

Accepted Answer

Increase the number of DPUs allocated to the Glue job. Option A is correct because increasing the DPU count provides more memory and processing power. Option B is correct because enabling job metrics helps identify memory bottlenecks. Option C (increasing shuffle threshold) may help but is not a primary solution. Option D (reducing parallelism) could reduce memory but may slow down the job. Option E (changing Redshift) is unrelated to memory in Glue.

Answer

Switch the output to Amazon S3 instead of Redshift

Answer

Reduce the number of partitions in the input data

Answer

Increase the spark.sql.shuffle.partitions parameter

Question 10

A company is ingesting streaming data from social media feeds using Amazon Kinesis Data Streams. The data is consumed by multiple applications: one for real-time sentiment analysis and another for archival to S3. The data must be processed in order for each social media post. Which TWO approaches meet the requirements? (Choose TWO.)

Accepted Answer

Use a single shard in the Kinesis Data Streams and have all consumers read from that shard. Option A is correct because using a single shard ensures ordering for all records. Option C is correct because using a partition key that groups related records ensures they go to the same shard, preserving order. Option B (multiple shards) does not guarantee global ordering. Option D (Firehose) does not guarantee ordering. Option E (SQS FIFO) can guarantee order but adds another service.

Answer

Use Amazon Kinesis Data Firehose to buffer and deliver to S3

Answer

Use Amazon SQS FIFO queues between the stream and consumers

Answer

Use multiple shards and assign each consumer to a specific shard

Question 11

A company is designing a data lake on Amazon S3. The data ingestion pipeline must handle both structured and unstructured data. The data must be cataloged for easy discovery. Which THREE services should be included in the solution? (Choose THREE.)

Accepted Answer

Amazon S3. Option A is correct because S3 is the storage layer. Option B is correct because Glue Crawlers can catalog data. Option C is correct because Athena can query the cataloged data. Option D (RDS) is a relational database, not for a data lake. Option E (Redshift) is a data warehouse, not a data lake.

Answer

Amazon RDS

Answer

Amazon Redshift

Question 12

A company runs a real-time analytics platform that ingests data from thousands of sensors via Amazon Kinesis Data Streams. Each sensor sends a JSON payload every second. The data is consumed by a fleet of EC2 instances running a custom consumer application. Recently, the consumer has been falling behind, with the iterator age exceeding 10 minutes. The company has already increased the number of shards to 100, but the problem persists. The consumer application is single-threaded per shard and uses the Kinesis Client Library (KCL). The CPU utilization on the EC2 instances is below 30%. What should the data engineer do to reduce the iterator age?

Accepted Answer

Modify the consumer to use multiple worker threads per shard. Option B is correct because the consumer is not processing data fast enough due to being single-threaded per shard; using multiple worker threads per shard can increase processing throughput. Option A (more shards) has already been tried without success. Option C (larger instance) may not help if CPU is low. Option D (Lambda) may not handle the volume efficiently and adds complexity.

Answer

Increase the number of shards to 200

Answer

Use larger EC2 instances with more vCPUs

Answer

Replace the EC2 consumer with AWS Lambda functions

Question 13

A media company ingests video files from content partners into an Amazon S3 bucket. Each video file is 10-50 GB. Upon upload, an AWS Lambda function is triggered to extract metadata (e.g., resolution, duration) and store it in DynamoDB. The company now wants to also generate a thumbnail image for each video. The thumbnail generation is CPU-intensive and can take up to 5 minutes per video. The Lambda function has a maximum execution time of 15 minutes. The company has noticed that some thumbnail generation tasks are timing out. What should the data engineer do to reliably generate thumbnails for all videos?

Accepted Answer

Use AWS Batch to run a containerized thumbnail generation job triggered by S3 events. Option C is correct because AWS Batch is designed for long-running, compute-intensive jobs like video processing. Option A (increase Lambda timeout) is not a good practice for such heavy tasks. Option B (EC2 instance) requires manual management. Option D (Glue) is for ETL, not video processing.

Answer

Provision an EC2 instance to run a script that polls S3 for new videos and generates thumbnails

Answer

Use AWS Glue with a Python shell job to generate thumbnails

Answer

Increase the Lambda timeout to 15 minutes and allocate more memory

Question 14

A small startup is building a data pipeline to ingest customer orders from a web application into Amazon Redshift for analytics. The orders are written to an Amazon RDS MySQL database. The startup wants to replicate the orders to Redshift in near-real time (within 5 minutes) with minimal operational overhead. The data volume is low, averaging 100 new orders per minute. The startup has a single data engineer who is also responsible for other tasks. What is the simplest solution?

Accepted Answer

Use AWS Database Migration Service (DMS) with continuous replication. Option D is correct because AWS DMS can continuously replicate from MySQL to Redshift with minimal setup and low overhead. Option A (Lambda) requires custom code. Option B (Glue) is batch-oriented and may not meet the 5-minute latency. Option C (EMR) is overkill.

Answer

Use AWS Glue with a scheduled job every 5 minutes to copy data from MySQL to Redshift

Answer

Use Amazon EMR with Spark streaming to read from MySQL and write to Redshift

Answer

Use an AWS Lambda function to query MySQL every minute and insert into Redshift

Question 15

A data engineer needs to ingest streaming data from thousands of IoT devices into AWS for near-real-time analytics. The data volume varies significantly and can spike unpredictably. The engineer wants to minimize operational overhead and ensure that data is durably stored as soon as it arrives. Which AWS service combination should the engineer use?

Accepted Answer

Use Amazon Kinesis Data Streams to ingest the data and AWS Lambda to process records in real-time with automatic scaling.. Option D is correct because Kinesis Data Streams provides durable, scalable ingestion for streaming data, and Lambda can process records in near-real-time with automatic scaling. Option A is wrong because S3 Transfer Acceleration is for accelerating uploads to S3, not for streaming ingestion. Option B is wrong because Kinesis Data Firehose is designed for loading streaming data into destinations like S3 but does not offer sub-second latency and has buffering delays. Option C is wrong because SQS is a message queue that decouples producers and consumers but does not natively support streaming data partitioning or replay.

Answer

Use Amazon S3 Transfer Acceleration with S3 Event Notifications to trigger AWS Lambda for processing.

Answer

Use Amazon Kinesis Data Firehose to ingest data into Amazon S3 and use AWS Lambda to transform data during delivery.

Answer

Use Amazon Simple Queue Service (SQS) to buffer the streaming data and configure an Auto Scaling group of EC2 instances to poll and process the data.