DEA-C01 Data Ingestion and Transformation Practice Test 20 — 15 Questions

Question 1

A data engineer is troubleshooting a Kinesis Data Firehose delivery stream that is experiencing high error rates when writing to an S3 bucket. The error logs indicate 'AccessDenied' errors. The S3 bucket policy allows access from the Firehose service, but the errors persist. What is the most likely cause?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

The IAM role assumed by Firehose does not have the s3:PutObject permission. The most likely cause is that the IAM role assumed by Kinesis Data Firehose lacks the `s3:PutObject` permission. Even if the S3 bucket policy allows access from the Firehose service, the IAM role must explicitly grant the necessary S3 write permissions for Firehose to deliver data. Without this permission, Firehose receives 'AccessDenied' errors when attempting to write objects to the bucket.

Answer

The S3 bucket has a lifecycle policy that is deleting objects too quickly

Answer

The S3 bucket has default encryption enabled

Answer

The S3 bucket uses an AWS KMS key for encryption and Firehose does not have kms:Decrypt permission

Question 2

A company needs to ingest data from multiple SaaS sources (e.g., Salesforce, Marketo) into Amazon S3 for analytics. Which AWS service is designed for this purpose?

Accepted Answer

Amazon AppFlow. Amazon AppFlow is a fully managed integration service specifically designed to securely transfer data between SaaS applications (like Salesforce, Marketo, Slack, and Zendesk) and AWS services such as Amazon S3 and Amazon Redshift. It supports scheduled, event-driven, or on-demand data ingestion with built-in transformations, filtering, and validation, making it the ideal choice for this use case.

Answer

AWS Transfer Family

Answer

AWS Glue

Answer

AWS DataSync

Question 3

A company is ingesting log files from multiple EC2 instances into Amazon S3 using the CloudWatch agent. The logs are delivered to a CloudWatch Logs group, and a subscription filter sends them to a Lambda function for transformation, then to Firehose. The Firehose stream is configured with a buffer interval of 60 seconds and buffer size of 5 MB. The logs are critical and must be available in S3 within 5 minutes. What is the most cost-effective way to reduce the delivery latency?

Accepted Answer

Decrease the buffer interval to 10 seconds. Decreasing the Firehose buffer interval to 10 seconds directly reduces the maximum time data waits in the buffer before being delivered to S3, ensuring logs reach S3 within the required 5-minute window. Since the current 60-second buffer interval is the primary contributor to latency, lowering it to 10 seconds minimizes delivery delay without incurring additional costs, as Firehose charges are based on data volume, not buffer frequency.

Answer

Replace Firehose with Amazon Kinesis Data Streams

Answer

Increase the buffer size to 10 MB

Answer

Increase the buffer interval to 120 seconds

Question 4

A data engineer is designing a data ingestion pipeline for real-time financial transactions. The pipeline must ensure exactly-once processing semantics and must handle duplicate records that may occur due to retries. Which combination of AWS services can achieve exactly-once processing?

Accepted Answer

Amazon Kinesis Data Streams with Amazon Kinesis Data Analytics for Apache Flink. Amazon Kinesis Data Streams with Kinesis Data Analytics for Apache Flink enables exactly-once processing by leveraging Flink's built-in checkpointing and two-phase commit protocols. Kinesis Data Streams provides a durable, ordered record store, while Flink uses its internal state and idempotent sinks to deduplicate records that arise from retries, ensuring each record is processed exactly once.

Answer

Amazon SQS with AWS Lambda

Answer

Amazon MSK with AWS Lambda

Answer

Amazon Kinesis Data Firehose with AWS Lambda

Question 5

A company needs to ingest data from an on-premises Hadoop cluster into Amazon S3 for archival and analysis. The total data volume is 50 TB. The migration must be completed within one week. The on-premises network has a 1 Gbps connection to AWS. Which AWS service should be used?

Accepted Answer

AWS Snowball Edge. AWS Snowball Edge is the correct choice because it is designed for large-scale data transfers (50 TB) over slow or unreliable networks. With a 1 Gbps connection, transferring 50 TB would take approximately 5 days under ideal conditions, but real-world factors (e.g., network congestion, protocol overhead) make it risky to complete within one week. Snowball Edge physically ships the data, bypassing network constraints entirely.

Answer

AWS Transfer Family

Answer

AWS Glue

Answer

AWS DataSync

Question 6

A company is using AWS Glue to run ETL jobs that read from Amazon S3 and write to Amazon Redshift. The jobs are failing intermittently with 'Out of Memory' errors. Which TWO actions should the data engineer take to resolve this issue? (Choose TWO.)

Accepted Answer

Increase the number of DPUs allocated to the Glue job. Increasing the number of DPUs allocated to the Glue job (Option B) directly addresses the 'Out of Memory' errors by providing more memory and compute resources per executor. AWS Glue uses Apache Spark under the hood, where each DPU provides 4 vCPU and 16 GB of memory; adding more DPUs increases the total memory available for data processing, reducing the likelihood of OOM errors during shuffle or aggregation operations.

Answer

Switch the output to Amazon S3 instead of Redshift

Answer

Reduce the number of partitions in the input data

Answer

Increase the spark.sql.shuffle.partitions parameter

Question 7

A company is ingesting streaming data from social media feeds using Amazon Kinesis Data Streams. The data is consumed by multiple applications: one for real-time sentiment analysis and another for archival to S3. The data must be processed in order for each social media post. Which TWO approaches meet the requirements? (Choose TWO.)

Accepted Answer

Use a single shard in the Kinesis Data Streams and have all consumers read from that shard. Using a single shard ensures that all records are processed in the exact order they are ingested, as Kinesis Data Streams guarantees ordering within a shard. Since both consumers (real-time sentiment analysis and archival to S3) read from the same shard, they will each receive records in the same sequence, meeting the requirement for ordered processing of each social media post.

Answer

Use Amazon Kinesis Data Firehose to buffer and deliver to S3

Answer

Use Amazon SQS FIFO queues between the stream and consumers

Answer

Use multiple shards and assign each consumer to a specific shard

Question 8

A company is designing a data lake on Amazon S3. The data ingestion pipeline must handle both structured and unstructured data. The data must be cataloged for easy discovery. Which THREE services should be included in the solution? (Choose THREE.)

Accepted Answer

Amazon S3. Amazon S3 is the core storage layer for the data lake, providing scalable, durable, and cost-effective object storage for both structured and unstructured data. AWS Glue Data Catalog acts as the central metadata repository, enabling data discovery and schema management across the data lake. Amazon Athena allows serverless querying of data directly from S3 using standard SQL, leveraging the Glue Data Catalog for schema-on-read, which is essential for easy discovery and analysis.

Answer

Amazon RDS

Answer

Amazon Redshift

Question 9

A company runs a real-time analytics platform that ingests data from thousands of sensors via Amazon Kinesis Data Streams. Each sensor sends a JSON payload every second. The data is consumed by a fleet of EC2 instances running a custom consumer application. Recently, the consumer has been falling behind, with the iterator age exceeding 10 minutes. The company has already increased the number of shards to 100, but the problem persists. The consumer application is single-threaded per shard and uses the Kinesis Client Library (KCL). The CPU utilization on the EC2 instances is below 30%. What should the data engineer do to reduce the iterator age?

Accepted Answer

Modify the consumer to use multiple worker threads per shard. The correct solution. The consumer is single-threaded per shard, which limits processing throughput despite low CPU utilization. Using multiple worker threads per shard allows concurrent processing of records from the same shard, reducing iterator age. Option A (more shards) was already tried and did not resolve the issue. Option B (larger EC2 instances) is unlikely to help because CPU is not the bottleneck. Option D (Lambda) may not handle the high-frequency sensor data efficiently and can introduce additional latency.

Answer

Increase the number of shards to 200

Answer

Use larger EC2 instances with more vCPUs

Answer

Replace the EC2 consumer with AWS Lambda functions

Question 10

A media company ingests video files from content partners into an Amazon S3 bucket. Each video file is 10-50 GB. Upon upload, an AWS Lambda function is triggered to extract metadata (e.g., resolution, duration) and store it in DynamoDB. The company now wants to also generate a thumbnail image for each video. The thumbnail generation is CPU-intensive and can take up to 5 minutes per video. The Lambda function has a maximum execution time of 15 minutes. The company has noticed that some thumbnail generation tasks are timing out. What should the data engineer do to reliably generate thumbnails for all videos?

Accepted Answer

Use AWS Batch to run a containerized thumbnail generation job triggered by S3 events. Use AWS Batch to run a containerized thumbnail generation job triggered by S3 events. AWS Batch is designed for long-running, compute-intensive workloads like video processing, and it can automatically scale resources based on the job queue. Option A (EC2 instance) requires manual management and scaling. Option B (Glue Python shell) is intended for ETL jobs, not CPU-intensive video processing. Option C (increasing Lambda timeout and memory) may still not be reliable for 5-minute thumbnail generation since Lambda has a maximum execution time of 15 minutes and is not optimized for heavy compute tasks; it could still time out or be cost-inefficient.

Answer

Provision an EC2 instance to run a script that polls S3 for new videos and generates thumbnails

Answer

Use AWS Glue with a Python shell job to generate thumbnails

Answer

Increase the Lambda timeout to 15 minutes and allocate more memory

Question 11

A small startup is building a data pipeline to ingest customer orders from a web application into Amazon Redshift for analytics. The orders are written to an Amazon RDS MySQL database. The startup wants to replicate the orders to Redshift in near-real time (within 5 minutes) with minimal operational overhead. The data volume is low, averaging 100 new orders per minute. The startup has a single data engineer who is also responsible for other tasks. What is the simplest solution?

Accepted Answer

Use AWS Database Migration Service (DMS) with continuous replication. AWS DMS can continuously replicate from MySQL to Redshift with minimal setup and low overhead. Option A (AWS Glue) is batch-oriented and may not meet the 5-minute latency. Option B (Amazon EMR) is overkill for low data volumes. Option C (AWS Lambda) requires custom code and may not efficiently handle the replication.

Answer

Use AWS Glue with a scheduled job every 5 minutes to copy data from MySQL to Redshift

Answer

Use Amazon EMR with Spark streaming to read from MySQL and write to Redshift

Answer

Use an AWS Lambda function to query MySQL every minute and insert into Redshift

Question 12

A data engineer needs to ingest streaming data from thousands of IoT devices into AWS for near-real-time analytics. The data volume varies significantly and can spike unpredictably. The engineer wants to minimize operational overhead and ensure that data is durably stored as soon as it arrives. Which AWS service combination should the engineer use?

Accepted Answer

Use Amazon Kinesis Data Streams to ingest the data and AWS Lambda to process records in real-time with automatic scaling.. Amazon Kinesis Data Streams (KDS) is designed for ingesting large volumes of streaming data with automatic scaling (via shard splitting/merging) and provides durable storage (default 24-hour retention, extendable to 365 days) as soon as records are received. AWS Lambda can be subscribed to the stream to process records in near-real-time, scaling automatically based on the number of shards, which minimizes operational overhead and handles unpredictable spikes without manual intervention.

Answer

Use Amazon S3 Transfer Acceleration with S3 Event Notifications to trigger AWS Lambda for processing.

Answer

Use Amazon Kinesis Data Firehose to ingest data into Amazon S3 and use AWS Lambda to transform data during delivery.

Answer

Use Amazon Simple Queue Service (SQS) to buffer the streaming data and configure an Auto Scaling group of EC2 instances to poll and process the data.

Question 13

A data engineer is troubleshooting a Lambda function that reads from a Kinesis Data Stream, processes records, and writes to a Kinesis Data Firehose delivery stream. The Firehose delivery stream is configured to deliver data to an S3 bucket. The Lambda function is failing with an access denied error. The IAM policy attached to the Lambda execution role is shown in the exhibit. Which permission is missing?

Accepted Answer

firehose:PutRecord on the Firehose delivery stream. The Lambda function is failing due to a missing permission to write to the Kinesis Data Firehose delivery stream. The required permission is firehose:PutRecord (or firehose:PutRecordBatch). The IAM policy likely lacks this permission, causing an access denied error when the Lambda attempts to write records. Option A correctly identifies this missing permission.

Answer

firehose:DescribeDeliveryStream on the Firehose delivery stream

Answer

logs:CreateLogGroup and logs:CreateLogStream on the CloudWatch log group

Answer

s3:PutObjectAcl on the S3 bucket

Question 14

A company wants to migrate on-premises data to Amazon S3 using AWS DataSync. The data is stored on an NFS file server and the total volume is 50 TB. The network bandwidth between the on-premises data center and AWS is 1 Gbps (gigabit per second). What is the primary factor that will determine the total time required for the initial data transfer?

Accepted Answer

The available network bandwidth between on-premises and AWS. AWS DataSync transfers data over the network, so the primary constraint is the available bandwidth between the on-premises NFS server and AWS. With 50 TB of data and a 1 Gbps link, the theoretical minimum transfer time is approximately 50 TB * 8 / 1 Gbps = 400,000 seconds (~111 hours), but real-world throughput is lower due to protocol overhead, latency, and competing traffic. The network bandwidth directly dictates the maximum data transfer rate, making it the dominant factor for the initial transfer duration.

Answer

The number of S3 buckets used as the destination

Answer

The average file size in the dataset

Answer

The IOPS (I/O operations per second) of the on-premises NFS server

Question 15

A data engineer runs an AWS Glue ETL job that reads from a large Amazon S3 source (several terabytes of CSV files) and writes transformed data to an S3 bucket in Parquet format. The job fails with the error shown in the exhibit. The job uses the Standard worker type with 10 workers (G.1X). The engineer needs to resolve the failure with minimal cost increase. What should the engineer do?

Accepted Answer

Change the worker type to G.2X with 10 workers.. The error indicates that the Glue job is running out of memory during the shuffle phase, which is common when processing large datasets with transformations that require data redistribution. Changing the worker type to G.2X doubles the memory per worker (from 16 GB to 32 GB) without increasing the number of workers, providing the necessary memory headroom for the shuffle operation at a minimal cost increase compared to scaling out with more workers.

Answer

Increase the number of workers to 20 while keeping G.1X worker type.

Answer

Change the worker type to G.4X with 10 workers.

Answer

Set the 'coalesce' parameter to reduce the number of output files.