DEA-C01 Data Ingestion and Transformation Practice Test 11 — 15 Questions

Question 1

A retail company uses AWS Glue to process daily sales data from multiple CSV files stored in Amazon S3. The Glue job runs a PySpark script that reads the files, performs joins, and writes the output as Parquet. Recently, the job has been failing with 'Out of Memory' errors. The data volume has grown from 10 GB to 50 GB per day. The Glue job uses 10 DPUs and the standard worker type. The data engineer needs to fix the job without rewriting the script. What should the data engineer do?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Change the worker type to G.2X to get more memory per worker.. Changing the worker type to G.2X doubles the memory per worker compared to the standard worker type, directly addressing the out-of-memory errors without modifying the PySpark script. Since the data volume grew fivefold, the existing 10 DPUs with standard workers lack sufficient memory for joins and shuffles. G.2X workers provide more memory and disk per executor, allowing the same script to handle the larger dataset.

Answer

Split the input CSV files into smaller partitions.

Answer

Decrease the number of DPUs to reduce memory contention.

Answer

Increase the number of DPUs for the Glue job to 20.

Question 2

A media company uses Amazon Kinesis Data Firehose to ingest log data from web servers into Amazon S3. The data is then processed by AWS Glue jobs. The company wants to ensure that data is delivered to S3 within 5 minutes of ingestion. Currently, the Firehose delivery stream is configured with a buffer interval of 300 seconds and a buffer size of 5 MB. The log data arrives at a rate of 2 MB per second. The data engineer notices that some log files are delayed by up to 10 minutes. The company cannot change the buffer size due to downstream requirements. What should the data engineer do to meet the 5-minute delivery requirement?

Accepted Answer

Decrease the buffer interval to 120 seconds.. Decreasing the buffer interval to 120 seconds ensures that data is delivered to S3 within 2 minutes, meeting the 5-minute requirement. The current buffer interval of 300 seconds (5 minutes) is the primary cause of delays up to 10 minutes. Option A is incorrect because increasing the buffer interval would increase delay. Option B is incorrect because increasing buffer size would also increase delay, as it takes longer to fill. Option C is incorrect because compression does not affect the buffer interval; data still waits up to 300 seconds.

Answer

Increase the buffer interval to 600 seconds to reduce the number of delivery attempts.

Answer

Increase the buffer size to 10 MB to ensure data is delivered in larger chunks.

Answer

Enable GZIP compression on the Firehose stream to reduce data size.

Question 3

A gaming company collects player event data from mobile devices. The data is sent to an Amazon API Gateway endpoint, which triggers an AWS Lambda function that writes the data to an Amazon DynamoDB table. The company wants to also store the data in Amazon S3 for historical analysis. The data volume is about 100 GB per day. The data engineer needs to design a solution to copy data from DynamoDB to S3 with minimal impact on the DynamoDB table. What should the data engineer do?

Accepted Answer

Enable DynamoDB Streams on the table and configure a Lambda function to write changes to S3.. DynamoDB Streams capture every item-level change in near real-time and can trigger a Lambda function that writes the data to S3, providing continuous replication with minimal read impact on the table. This is the canonical AWS pattern for streaming DynamoDB data to S3 without consuming provisioned read capacity.

Answer

Create a global secondary index on the table and export the index to S3.

Answer

Use AWS Glue to scan the DynamoDB table and write results to S3 every hour.

Answer

Use the DynamoDB Export to S3 feature to export the entire table daily.

Question 4

A logistics company uses AWS Glue to process GPS data from delivery trucks. The data is stored in Amazon S3 as JSON files. The Glue job reads the JSON files, converts them to Parquet, and writes them back to S3. The company notices that the Glue job takes too long to complete. The data engineer wants to improve the job's performance without changing the code. What should the data engineer do?

Accepted Answer

Change the worker type to G.2X.. The G.2X worker type provides more memory and vCPUs per worker than the default G.1X, which is critical for memory-intensive operations like JSON parsing and Parquet conversion. Since the job is already taking too long, the bottleneck is likely CPU or memory per worker, not the total number of workers. Upgrading to G.2X increases the compute capacity per worker without changing the code, directly addressing the performance issue.

Answer

Increase the number of DPUs to 20.

Answer

Change the worker type to G.1X.

Answer

Decrease the number of DPUs to 5 to reduce overhead.

Question 5

An e-commerce company uses AWS Glue to process clickstream data from its website. The data is stored in Amazon S3 in partitioned Parquet format by date and hour. A recent increase in traffic has caused the Glue job to fail with 'Java heap space' errors. The job runs with 10 DPUs and uses Spark's default configurations. The data engineer needs to resolve the memory issue without modifying the ETL script. What should the data engineer do?

Accepted Answer

Increase the Spark configuration 'spark.sql.shuffle.partitions' to 500.. The 'Java heap space' error occurs when Spark executors run out of memory during shuffle operations. With 10 DPUs and default settings, spark.sql.shuffle.partitions defaults to 200, which may create partitions too large to fit in executor memory. Increasing shuffle partitions to 500 creates smaller, more manageable partitions that reduce per-partition memory pressure without requiring script changes or additional resources. This is a configuration-only fix that directly addresses the memory bottleneck.

Answer

Decrease the Spark configuration 'spark.sql.shuffle.partitions' to 50.

Answer

Change the worker type to G.1X.

Answer

Increase the number of DPUs to 20.

Question 6

A data engineer needs to ingest data from an on-premises Apache Kafka cluster into Amazon S3. The data engineer wants to minimize operational overhead and avoid managing any servers. Which AWS service should the data engineer use?

Accepted Answer

Amazon Managed Streaming for Apache Kafka (Amazon MSK). Amazon MSK is correct. It is a fully managed Apache Kafka service that can ingest data from on-premises Kafka via MirrorMaker or replication, then sink to S3, minimizing operational overhead. AWS Glue (B) is for ETL jobs, not real-time Kafka ingestion. Kinesis Data Analytics (C) is for analyzing streaming data, not ingestion. Amazon Kinesis Data Streams (D) is a different service not directly compatible with Kafka without custom connectors.

Answer

AWS Glue

Answer

Amazon Kinesis Data Analytics

Answer

Amazon Kinesis Data Streams

Question 7

A data engineer needs to ingest streaming data from an e-commerce application into Amazon S3 for near-real-time analytics. The solution must handle variable throughput and allow reprocessing of failed records. Which TWO AWS services should the engineer use? (Choose two.)

Accepted Answer

Amazon Kinesis Data Streams. Amazon Kinesis Data Streams is correct because it provides a durable, scalable, real-time data ingestion layer that can handle variable throughput by sharding, and it retains data for up to 365 days, enabling reprocessing of failed records by replaying from a specific sequence number or timestamp.

Answer

Amazon SQS

Answer

Amazon DynamoDB Streams

Answer

Amazon RDS

Question 8

A data engineering team is building a pipeline to transform CSV files uploaded to Amazon S3 into Parquet format using AWS Glue. The transformation must be serverless and handle files that arrive at irregular intervals. Which TWO actions should the team take? (Choose two.)

Accepted Answer

Use an AWS Glue ETL job to convert CSV to Parquet.. AWS Glue ETL jobs provide a serverless Spark-based environment that can directly read CSV files from S3 and write them as Parquet, meeting the requirement for serverless transformation. Glue handles schema inference and conversion without managing any infrastructure, making it ideal for irregularly scheduled data.

Answer

Configure an Amazon EMR cluster with Apache Spark for on-demand transformation.

Answer

Use AWS Data Pipeline to schedule a periodic transformation.

Answer

Use Amazon Redshift Spectrum to convert files during query execution.

Question 9

A company ingests IoT sensor data into Amazon Kinesis Data Streams. The data must be enriched with device metadata from Amazon DynamoDB and then stored in Amazon S3 in Apache Parquet format. The solution must minimize latency and cost. Which THREE steps should a data engineer implement? (Choose three.)

Accepted Answer

Deliver the enriched data to Amazon Kinesis Data Firehose and enable Parquet conversion.. The correct three steps are: using Amazon Kinesis Data Analytics for Apache Flink to enrich the stream with data from DynamoDB (null), performing a DynamoDB lookup in the Flink application for each record (E), and delivering the enriched data to Amazon Kinesis Data Firehose with Parquet conversion enabled (A). Kinesis Data Analytics for Apache Flink reads from Kinesis Data Streams, enriches each record via DynamoDB lookups, and outputs the enriched stream to Kinesis Data Firehose. Firehose automatically converts data to Apache Parquet before writing to S3, minimizing latency and cost by leveraging managed services without custom code or additional infrastructure.

Answer

Configure an AWS Lambda function to read from the stream, enrich, and write to S3.

Answer

Use AWS Glue streaming ETL to enrich and convert data to Parquet.

Answer

Use Amazon EMR with Spark Streaming to process and store the data.

Question 10

A financial services company ingests stock trade data from multiple exchanges into an Amazon S3 bucket (trade-bucket). Each exchange sends a CSV file every 5 minutes. The data must be transformed into Parquet format and partitioned by exchange and date (trade_date) for efficient querying using Amazon Athena. The pipeline must handle late-arriving data (files up to 2 hours late) and ensure exactly-once processing to avoid duplicates. Currently, a scheduled AWS Glue ETL job runs every hour, reads new CSV files, converts them to Parquet, and writes to an output bucket. However, the team is experiencing data duplication: if the job fails midway, upon retry it reprocesses all files in the input folder, causing duplicates in the output. Additionally, the job takes too long because it scans all files each run. The engineer must redesign the pipeline to eliminate duplicates and improve efficiency. What should the engineer do?

Accepted Answer

Set up an S3 event notification to invoke an AWS Lambda function that starts a Glue job with a parameter containing the S3 object key of the new file; modify the Glue job to process only that file and use the file key to avoid duplicates.. The best approach. By setting up an S3 event notification to invoke a Lambda function that triggers a Glue job with the new file's S3 key as a parameter, the job processes only that specific file. Using the file key in the job logic ensures idempotency—if the job fails and retries, it reprocesses the same file key, and deduplication can be handled (e.g., by checking if the output partition already contains that file's data or using a job bookmark on the file key). This achieves exactly-once processing and incremental processing (no full scans), improving efficiency. Option A (Glue Workflows) still processes all files each run and doesn't prevent duplicates if a file arrives after the job starts. Option C (moving CSV files to an archive folder) risks race conditions if late-arriving data comes while the job is running, and does not guarantee exactly-once if the job fails mid-way. Option D (EMR with Spark Structured Streaming) is overly complex and expensive for a 5-minute CSV batch ingestion; checkpointing can handle failures but adds significant operational overhead.

Answer

Use AWS Glue Workflows to orchestrate the job and add a condition to check for duplicates before writing.

Answer

Modify the Glue job to move processed CSV files to an archive folder after successful transformation, and process only unprocessed files.

Answer

Replace Glue with Amazon EMR and use Spark Structured Streaming with checkpointing to process files incrementally.

Question 11

A logistics company ingests real-time GPS location data from thousands of delivery vehicles into Amazon Kinesis Data Streams. Each vehicle sends a JSON payload every 10 seconds containing vehicle_id, latitude, longitude, timestamp, and speed. The data must be stored in Amazon S3 for historical analysis, but the company wants to first aggregate the data per vehicle per minute (average speed, min/max coordinates) to reduce storage costs. The solution must be serverless and handle potential duplicate records without double-counting. What should the engineer do?

Accepted Answer

Use Amazon Kinesis Data Analytics for Apache Flink to aggregate data in a 1-minute tumbling window with deduplication logic, then output to Kinesis Data Firehose for delivery to S3.. Kinesis Data Analytics for Apache Flink natively supports windowed aggregations (tumbling windows) with exactly-once semantics via checkpoints, and can apply deduplication logic on vehicle_id+timestamp keys before emitting to Firehose. Firehose then batches and delivers the reduced dataset to S3, cutting storage costs while remaining fully serverless. This is the canonical AWS pattern for stream aggregation with dedup.

Answer

Use Amazon EMR with Spark Streaming to perform the aggregation and write to S3.

Answer

Use Kinesis Data Firehose with a Lambda transformation to aggregate records in a 1-minute window.

Answer

Use an AWS Glue streaming ETL job with Spark Structured Streaming to aggregate and deduplicate.

Question 12

A media company ingests large video files (up to 100 GB each) from content creators via Amazon S3 multipart uploads. After upload, the company needs to transcode the videos into multiple formats using AWS Elemental MediaConvert. The current pipeline uses S3 event notifications to trigger an AWS Lambda function that starts a MediaConvert job. However, for very large files, the Lambda function times out (15-minute limit) before the upload completes because the event is sent when the multipart upload is initiated, not when it completes. How should the engineer fix this issue?

Accepted Answer

Change the S3 event notification to listen for s3:ObjectCreated:Put events instead of s3:ObjectCreated:*, which fires only after the object is fully written.. Use S3 event notifications for s3:ObjectCreated:Put, which triggers only after the object is completely written. Then Lambda can start MediaConvert. Option A (increase timeout) does not solve the root cause. Option B (SQS) does not address the event timing. Option C (Step Functions) adds complexity.

Answer

Increase the Lambda timeout to 30 minutes.

Answer

Use Amazon SQS to queue the event and have a Lambda function process it later.

Answer

Use AWS Step Functions to poll the S3 bucket for object existence and then start MediaConvert.

Question 13

A financial company needs to ingest real-time stock trade data from multiple sources and store it in Amazon S3 for compliance. The data must be delivered within 1 minute of the trade occurring. The data volume is approximately 10,000 records per second, with occasional spikes to 50,000 records per second. The engineer has set up Amazon Kinesis Data Streams with 10 shards and a Kinesis Data Firehose delivery stream that reads from the Kinesis stream and writes to S3. However, during spikes, the Firehose delivery stream falls behind, causing data to be delayed beyond the 1-minute SLA. What should the engineer do to meet the SLA without over-provisioning?

Accepted Answer

Increase the number of shards in the Kinesis data stream to handle peak throughput and enable auto-scaling.. Increasing the number of shards in the Kinesis data stream allows it to handle the peak throughput of 50,000 records per second, and enabling auto-scaling ensures the stream adapts to varying loads without manual intervention. This reduces the backlog that causes Firehose to fall behind, meeting the 1-minute SLA. Option A is incorrect because increasing the buffer size in Firehose would increase latency, not reduce it. Option B is incorrect because adding SQS as a buffer adds another hop, increasing latency and complexity. Option C is incorrect because AWS Lambda may not scale to 50,000 records per second and adds processing latency.

Answer

Increase the buffer size in Kinesis Data Firehose from 1 MB to 5 MB to batch more data per delivery.

Answer

Use Amazon SQS as a buffer between Kinesis and Firehose to absorb spikes.

Answer

Replace Firehose with an AWS Lambda function that writes directly to S3 for lower latency.

Question 14

A social media company ingests user activity data from multiple sources into Amazon S3. The data is in JSON format and includes fields: user_id, activity_type, timestamp, and metadata. The company wants to transform this data into a columnar format (Parquet) partitioned by date and activity_type for efficient querying with Amazon Athena. The pipeline must handle data that arrives up to 3 days late. Currently, a daily AWS Glue ETL job scans the entire S3 bucket for new files, transforms them, and writes to a separate output bucket. The job is taking longer as data volume grows, and the team wants to reduce processing time and cost. What should the engineer do?

Accepted Answer

Set up S3 event notifications to invoke an AWS Lambda function that triggers a Glue job for each new object, passing the object key so the job processes only that file.. The current job scans the entire bucket on every run, so cost and runtime scale with total data volume rather than new data. Wiring S3 event notifications to a Lambda that invokes a Glue job with the specific object key lets the job process only the newly arrived file, eliminating full-bucket scans and dramatically reducing both runtime and DPU cost. This also naturally handles late-arriving data because each new object triggers its own processing.

Answer

Increase the number of DPUs for the Glue job to process data faster.

Answer

Use AWS Glue partition projection and schema inference to reduce scan time.

Answer

Replace AWS Glue with Amazon EMR and use Spark to process data in parallel.

Question 15

A gaming company ingests player event data from mobile games into Amazon Kinesis Data Streams. Each event is a small JSON payload (<1 KB). The data must be delivered to Amazon S3 for analytics, and the company wants to minimize storage costs by aggregating events into larger files (e.g., 100 MB per file). The current setup uses Kinesis Data Firehose with a buffer size of 10 MB and a buffer interval of 60 seconds, but the resulting files are very small (average 5 MB) because the data volume is low. The engineer needs to ensure that files are at least 100 MB to reduce the number of S3 objects and lower costs. What should the engineer do?

Accepted Answer

Increase the buffer size in Kinesis Data Firehose to 100 MB and increase the buffer interval to 300 seconds.. Kinesis Data Firehose buffer size and interval work together: Firehose flushes data to S3 when either the buffer size (1–128 MB) or the buffer interval (60–900 seconds) is reached, whichever comes first. With low data volume, the 10 MB buffer size is hit before the 60-second interval, producing small files. Setting the buffer size to 100 MB and the interval to 300 seconds means Firehose will wait until 100 MB accumulates or 5 minutes pass, whichever occurs first, resulting in larger files and fewer S3 objects. This directly addresses the root cause without introducing custom code or additional services.

Answer

Use an AWS Glue streaming ETL job with a 100 MB file size threshold to write to S3.

Answer

Use an AWS Lambda function to buffer events in memory and write to S3 when buffer reaches 100 MB.

Answer

Use Amazon EMR with Spark Streaming to aggregate and write larger files to S3.