DEA-C01 Data Ingestion and Transformation Practice Test 10 — 15 Questions

Question 1

A company is ingesting streaming data from social media feeds using Amazon Kinesis Data Streams. The data is consumed by multiple applications: one for real-time sentiment analysis and another for archival to S3. The data must be processed in order for each social media post. Which TWO approaches meet the requirements? (Choose TWO.)

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Use a single shard in the Kinesis Data Streams and have all consumers read from that shard. Using a single shard ensures that all records are processed in the exact order they are ingested, as Kinesis Data Streams guarantees ordering within a shard. Since both consumers (real-time sentiment analysis and archival to S3) read from the same shard, they will each receive records in the same sequence, meeting the requirement for ordered processing of each social media post.

Answer

Use Amazon Kinesis Data Firehose to buffer and deliver to S3

Answer

Use Amazon SQS FIFO queues between the stream and consumers

Answer

Use multiple shards and assign each consumer to a specific shard

Question 2

A company is designing a data lake on Amazon S3. The data ingestion pipeline must handle both structured and unstructured data. The data must be cataloged for easy discovery. Which THREE services should be included in the solution? (Choose THREE.)

Accepted Answer

Amazon S3. Amazon S3 is the core storage layer for the data lake, providing scalable, durable, and cost-effective object storage for both structured and unstructured data. AWS Glue Data Catalog acts as the central metadata repository, enabling data discovery and schema management across the data lake. Amazon Athena allows serverless querying of data directly from S3 using standard SQL, leveraging the Glue Data Catalog for schema-on-read, which is essential for easy discovery and analysis.

Answer

Amazon RDS

Answer

Amazon Redshift

Question 3

A company runs a real-time analytics platform that ingests data from thousands of sensors via Amazon Kinesis Data Streams. Each sensor sends a JSON payload every second. The data is consumed by a fleet of EC2 instances running a custom consumer application. Recently, the consumer has been falling behind, with the iterator age exceeding 10 minutes. The company has already increased the number of shards to 100, but the problem persists. The consumer application is single-threaded per shard and uses the Kinesis Client Library (KCL). The CPU utilization on the EC2 instances is below 30%. What should the data engineer do to reduce the iterator age?

Accepted Answer

Modify the consumer to use multiple worker threads per shard. The correct solution. The consumer is single-threaded per shard, which limits processing throughput despite low CPU utilization. Using multiple worker threads per shard allows concurrent processing of records from the same shard, reducing iterator age. Option A (more shards) was already tried and did not resolve the issue. Option B (larger EC2 instances) is unlikely to help because CPU is not the bottleneck. Option D (Lambda) may not handle the high-frequency sensor data efficiently and can introduce additional latency.

Answer

Increase the number of shards to 200

Answer

Use larger EC2 instances with more vCPUs

Answer

Replace the EC2 consumer with AWS Lambda functions

Question 4

A media company ingests video files from content partners into an Amazon S3 bucket. Each video file is 10-50 GB. Upon upload, an AWS Lambda function is triggered to extract metadata (e.g., resolution, duration) and store it in DynamoDB. The company now wants to also generate a thumbnail image for each video. The thumbnail generation is CPU-intensive and can take up to 5 minutes per video. The Lambda function has a maximum execution time of 15 minutes. The company has noticed that some thumbnail generation tasks are timing out. What should the data engineer do to reliably generate thumbnails for all videos?

Accepted Answer

Use AWS Batch to run a containerized thumbnail generation job triggered by S3 events. Use AWS Batch to run a containerized thumbnail generation job triggered by S3 events. AWS Batch is designed for long-running, compute-intensive workloads like video processing, and it can automatically scale resources based on the job queue. Option A (EC2 instance) requires manual management and scaling. Option B (Glue Python shell) is intended for ETL jobs, not CPU-intensive video processing. Option C (increasing Lambda timeout and memory) may still not be reliable for 5-minute thumbnail generation since Lambda has a maximum execution time of 15 minutes and is not optimized for heavy compute tasks; it could still time out or be cost-inefficient.

Answer

Provision an EC2 instance to run a script that polls S3 for new videos and generates thumbnails

Answer

Use AWS Glue with a Python shell job to generate thumbnails

Answer

Increase the Lambda timeout to 15 minutes and allocate more memory

Question 5

A data engineer needs to ingest streaming data from thousands of IoT devices into AWS for near-real-time analytics. The data volume varies significantly and can spike unpredictably. The engineer wants to minimize operational overhead and ensure that data is durably stored as soon as it arrives. Which AWS service combination should the engineer use?

Accepted Answer

Use Amazon Kinesis Data Streams to ingest the data and AWS Lambda to process records in real-time with automatic scaling.. Amazon Kinesis Data Streams (KDS) is designed for ingesting large volumes of streaming data with automatic scaling (via shard splitting/merging) and provides durable storage (default 24-hour retention, extendable to 365 days) as soon as records are received. AWS Lambda can be subscribed to the stream to process records in near-real-time, scaling automatically based on the number of shards, which minimizes operational overhead and handles unpredictable spikes without manual intervention.

Answer

Use Amazon S3 Transfer Acceleration with S3 Event Notifications to trigger AWS Lambda for processing.

Answer

Use Amazon Kinesis Data Firehose to ingest data into Amazon S3 and use AWS Lambda to transform data during delivery.

Answer

Use Amazon Simple Queue Service (SQS) to buffer the streaming data and configure an Auto Scaling group of EC2 instances to poll and process the data.

Question 6

A data engineer runs an AWS Glue ETL job that reads from a large Amazon S3 source (several terabytes of CSV files) and writes transformed data to an S3 bucket in Parquet format. The job fails with the error shown in the exhibit. The job uses the Standard worker type with 10 workers (G.1X). The engineer needs to resolve the failure with minimal cost increase. What should the engineer do?

Accepted Answer

Change the worker type to G.2X with 10 workers.. The error indicates that the Glue job is running out of memory during the shuffle phase, which is common when processing large datasets with transformations that require data redistribution. Changing the worker type to G.2X doubles the memory per worker (from 16 GB to 32 GB) without increasing the number of workers, providing the necessary memory headroom for the shuffle operation at a minimal cost increase compared to scaling out with more workers.

Answer

Increase the number of workers to 20 while keeping G.1X worker type.

Answer

Change the worker type to G.4X with 10 workers.

Answer

Set the 'coalesce' parameter to reduce the number of output files.

Question 7

A company wants to ingest data from a SaaS application into Amazon S3. The SaaS application supports streaming data via HTTP POST requests. The data volume is approximately 100 MB per hour, and the company needs to store the raw data in S3 for archival and later analysis. Which approach is the most cost-effective and operationally efficient?

Accepted Answer

Use Amazon API Gateway to create a REST API that receives the data and triggers an AWS Lambda function to store it in S3.. Amazon API Gateway combined with AWS Lambda is the most appropriate and cost-effective approach for this use case. API Gateway provides a fully managed HTTP endpoint that can receive the HTTP POST requests from the SaaS application. The data is then passed to a Lambda function, which writes the raw data directly to Amazon S3. This serverless architecture eliminates the need to manage servers, scales automatically, and incurs cost only when data is processed. Option B is incorrect because Amazon Kinesis Data Firehose does not natively support HTTP endpoints as a source; it can ingest data from Kinesis Data Streams, AWS IoT, or custom agents, but not directly via HTTP POST.

Answer

Launch a t3.nano EC2 instance that runs a script to receive HTTP POST requests and write to S3.

Answer

Use Amazon Kinesis Data Firehose with HTTP endpoint as the source, and configure S3 as the destination.

Answer

Use Amazon Simple Queue Service (SQS) to queue the HTTP POST data and have an AWS Lambda function read from SQS and write to S3.

Question 8

A data engineer needs to transform JSON data from a Kinesis Data Stream into Parquet format and store it in an S3 data lake. The transformation includes simple field mapping and data type conversions. Which AWS service is the most cost-effective for performing this transformation in near-real-time?

Accepted Answer

AWS Lambda function triggered by Kinesis Data Firehose. AWS Lambda functions triggered by Kinesis Data Firehose are the most cost-effective solution for near-real-time transformations because Lambda allows you to perform lightweight field mapping and data type conversions on streaming data with a pay-per-invocation model, while Firehose handles buffering, compression, and direct delivery to S3 in Parquet format. This serverless approach avoids the overhead and cost of provisioning clusters or running continuous jobs, making it ideal for simple transformations on high-frequency, low-latency streams.

Answer

Amazon Athena with CTAS (CREATE TABLE AS SELECT)

Answer

AWS Glue ETL job

Answer

Amazon EMR with Spark Streaming

Question 9

A data engineer runs an AWS Glue crawler that is configured to crawl an S3 bucket named 'my-data-lake' and update the Glue Data Catalog. The crawler fails with an access denied error. The IAM role attached to the crawler has the policy shown in the exhibit. What is the likely cause of the failure?

Accepted Answer

The policy does not allow logging to CloudWatch Logs.. AWS Glue crawlers require permissions to write logs to CloudWatch Logs for monitoring and debugging. Without the `logs:CreateLogGroup`, `logs:CreateLogStream`, and `logs:PutLogEvents` actions, the crawler fails with an access denied error even if it has S3 and Glue Data Catalog permissions. The IAM policy shown does not include these CloudWatch Logs permissions, making option C the correct answer.

Answer

The policy does not allow glue:CreateTable on the 'my-data-lake' database.

Answer

The policy does not allow s3:PutObject on the 'my-data-lake' bucket.

Answer

The policy does not allow s3:ListBucket on the 'my-data-lake' bucket.

Question 10

A data engineer needs to transform data in an S3 data lake using AWS Glue ETL. The data is in CSV format and needs to be converted to Parquet with partitioning by date. The engineer wants to minimize the number of files written to S3 to improve query performance. Which TWO configuration options should the engineer use? (Select TWO.)

Accepted Answer

Use the coalesce method to reduce the number of output partitions.. Using `coalesce` reduces the number of output partitions, which directly minimizes the number of files written to S3. Fewer, larger Parquet files improve query performance by reducing the overhead of file listing and metadata operations in engines like Amazon Athena or Redshift Spectrum.

Answer

Increase the number of workers in the Glue job to increase parallelism.

Answer

Disable compression in the Parquet output.

Answer

Set the 'groupFiles' option to 'inPartition' in the DynamicFrame writer.

Question 11

A data engineer is responsible for ingesting log files from a fleet of on-premises servers into Amazon S3 for central analysis. Each server generates log files that are rotated every hour, resulting in files of about 500 MB each. The total daily data volume is approximately 1 TB. The network connection between the on-premises data center and AWS is a 100 Mbps VPN. The engineer needs to ensure that all log files are transferred to S3 within 24 hours of generation without data loss. The engineer is considering using AWS DataSync. However, the initial setup shows that the transfer speed is insufficient to meet the 24-hour SLA. What should the engineer do to meet the requirement?

Accepted Answer

Configure AWS DataSync to run on a schedule with incremental transfers and enable data compression.. AWS DataSync can be configured for incremental transfers after an initial full sync, which reduces the amount of data transferred on a daily basis. Additionally, enabling data compression reduces the size of the log files during transfer, making it feasible to meet the 24-hour SLA over a 100 Mbps VPN. Option A is incorrect because while AWS CLI with multipart uploads can improve throughput, it still requires manual scripting and does not provide the same optimization as DataSync. Option B is incorrect because upgrading the VPN bandwidth may not be immediately feasible or cost-effective. Option C is incorrect because ordering an AWS Snowball Edge device is intended for large initial data migrations, not for ongoing daily transfers; it would introduce delays and is not suitable for recurring incremental transfers.

Answer

Use AWS CLI with multipart uploads and parallel threads to maximize throughput.

Answer

Contact the network team to upgrade the VPN bandwidth to at least 1 Gbps.

Answer

Order an AWS Snowball Edge device to transfer the initial data and then use DataSync for incremental changes.

Question 12

A data engineer is setting up a data pipeline to ingest data from an Amazon RDS for MySQL database into Amazon S3 using AWS Glue ETL. The Glue job uses a JDBC connection to read from the MySQL database. The job runs successfully, but the engineer notices that the job is taking longer than expected. The MySQL database is 500 GB in size and the Glue job uses 10 workers of type G.1X. The engineer wants to improve the performance of the extraction phase. The database is actively used by other applications, so the engineer must minimize the impact on the source database. Which approach should the engineer take?

Accepted Answer

Partition the table by a numeric column, such as the primary key, and use the 'hashex' or 'hashpar' partitioning option in the Glue JDBC connection.. Partitioning the table on a key column (e.g., primary key) allows Glue to read in parallel from multiple partitions, reducing the load on the database and improving performance. Option B is wrong because incremental extraction is for ongoing changes, not for an initial full load; it doesn't address the immediate performance issue of extracting 500 GB. Option C is wrong because using a read replica offloads read traffic but does not inherently improve parallelism; partitioning is still needed for performance. Option D is wrong because simply increasing the number of workers may overwhelm the database with more simultaneous connections without partitioning, potentially causing performance degradation.

Answer

Use an incremental extraction strategy with a watermark column to reduce the amount of data read each time.

Answer

Create a read replica of the MySQL database and configure the Glue job to read from the replica.

Answer

Increase the number of Glue workers to 20 to increase parallelism.

Question 13

A company is ingesting IoT sensor data from thousands of devices using Amazon Kinesis Data Streams. The data is consumed by a Lambda function that transforms and writes to Amazon S3. The company notices that occasionally records are dropped. The data engineer needs to identify the cause and prevent data loss. Which TWO actions should the data engineer take? (Choose TWO.)

Accepted Answer

Increase the number of shards in the Kinesis data stream.. Increasing the number of shards in the Kinesis data stream raises the total read and write capacity, reducing the likelihood of throttling that can cause records to be dropped. Option E is correct because configuring a dead-letter queue (DLQ) on the Lambda function captures records that fail processing after all retries, preventing data loss and enabling reprocessing.

Answer

Enable CloudWatch Logs on the Kinesis stream to log all records.

Answer

Decrease the Lambda batch size to process records more frequently.

Answer

Add an Amazon SQS queue between Kinesis and Lambda to buffer records.

Question 14

A data engineer needs to ingest data from multiple on-premises relational databases into Amazon S3 for analytics. The data must be transformed and loaded daily. Which THREE AWS services should the engineer use together to build this pipeline? (Choose THREE.)

Accepted Answer

AWS Glue. AWS Glue is correct because it provides a serverless ETL (Extract, Transform, Load) service that can read data from Amazon S3, apply transformations (e.g., using PySpark or Scala), and write the transformed data back to S3. In this pipeline, AWS Glue jobs can be scheduled to run daily to perform the required transformations on the ingested data.

Answer

Amazon Athena

Answer

Amazon Kinesis Data Streams

Question 15

A financial services company ingests real-time stock trade data using Amazon Kinesis Data Streams with 10 shards. Each shard receives about 500 records per second, each record approximately 1 KB. The data is consumed by a single AWS Lambda function that transforms the data and writes to Amazon S3. The Lambda function is configured with 1024 MB memory and a timeout of 5 minutes. The company notices that the Lambda function is frequently throttled, and data ingestion lags behind. The Lambda function's CloudWatch metrics show that the iterator age is increasing, and the function's concurrency is maxed out at 1000. The data engineer needs to resolve the throttling issue without changing the Lambda function code. What should the data engineer do?

Accepted Answer

Increase the number of shards in the Kinesis data stream to increase parallelism.. Increasing the number of shards increases the number of Kinesis Data Streams processing units, which directly increases the concurrency limit for Lambda consumers. With more shards, more Lambda function instances can process records in parallel, reducing the iterator age and alleviating throttling. Option B is wrong because reducing memory would likely degrade performance and not increase concurrency limit (concurrency limit is independent of memory). Option C is wrong because decreasing batch size increases the number of invocations, potentially worsening throttling and overhead. Option D is wrong because increasing memory may improve per-record processing speed, but the core issue is concurrency limit being maxed out; increasing memory does not increase concurrency limit and may not resolve throttling if CPU is not the bottleneck.

Answer

Reduce the Lambda function memory to 512 MB to increase concurrency limit.

Answer

Decrease the batch size to 10 records to reduce processing time per invocation.

Answer

Increase the Lambda function memory to 2048 MB to improve processing speed.