DEA-C01 Data Ingestion and Transformation Practice Test 22 — 15 Questions

Question 1

A company is ingesting IoT sensor data from thousands of devices using Amazon Kinesis Data Streams. The data is consumed by a Lambda function that transforms and writes to Amazon S3. The company notices that occasionally records are dropped. The data engineer needs to identify the cause and prevent data loss. Which TWO actions should the data engineer take? (Choose TWO.)

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Increase the number of shards in the Kinesis data stream.. Increasing the number of shards in the Kinesis data stream raises the total read and write capacity, reducing the likelihood of throttling that can cause records to be dropped. Option E is correct because configuring a dead-letter queue (DLQ) on the Lambda function captures records that fail processing after all retries, preventing data loss and enabling reprocessing.

Answer

Enable CloudWatch Logs on the Kinesis stream to log all records.

Answer

Decrease the Lambda batch size to process records more frequently.

Answer

Add an Amazon SQS queue between Kinesis and Lambda to buffer records.

Question 2

A data engineer needs to ingest data from multiple on-premises relational databases into Amazon S3 for analytics. The data must be transformed and loaded daily. Which THREE AWS services should the engineer use together to build this pipeline? (Choose THREE.)

Accepted Answer

AWS Glue. AWS Glue is correct because it provides a serverless ETL (Extract, Transform, Load) service that can read data from Amazon S3, apply transformations (e.g., using PySpark or Scala), and write the transformed data back to S3. In this pipeline, AWS Glue jobs can be scheduled to run daily to perform the required transformations on the ingested data.

Answer

Amazon Athena

Answer

Amazon Kinesis Data Streams

Question 3

A financial services company ingests real-time stock trade data using Amazon Kinesis Data Streams with 10 shards. Each shard receives about 500 records per second, each record approximately 1 KB. The data is consumed by a single AWS Lambda function that transforms the data and writes to Amazon S3. The Lambda function is configured with 1024 MB memory and a timeout of 5 minutes. The company notices that the Lambda function is frequently throttled, and data ingestion lags behind. The Lambda function's CloudWatch metrics show that the iterator age is increasing, and the function's concurrency is maxed out at 1000. The data engineer needs to resolve the throttling issue without changing the Lambda function code. What should the data engineer do?

Accepted Answer

Increase the number of shards in the Kinesis data stream to increase parallelism.. Increasing the number of shards increases the number of Kinesis Data Streams processing units, which directly increases the concurrency limit for Lambda consumers. With more shards, more Lambda function instances can process records in parallel, reducing the iterator age and alleviating throttling. Option B is wrong because reducing memory would likely degrade performance and not increase concurrency limit (concurrency limit is independent of memory). Option C is wrong because decreasing batch size increases the number of invocations, potentially worsening throttling and overhead. Option D is wrong because increasing memory may improve per-record processing speed, but the core issue is concurrency limit being maxed out; increasing memory does not increase concurrency limit and may not resolve throttling if CPU is not the bottleneck.

Answer

Reduce the Lambda function memory to 512 MB to increase concurrency limit.

Answer

Decrease the batch size to 10 records to reduce processing time per invocation.

Answer

Increase the Lambda function memory to 2048 MB to improve processing speed.

Question 4

A retail company uses AWS Glue to process daily sales data from multiple CSV files stored in Amazon S3. The Glue job runs a PySpark script that reads the files, performs joins, and writes the output as Parquet. Recently, the job has been failing with 'Out of Memory' errors. The data volume has grown from 10 GB to 50 GB per day. The Glue job uses 10 DPUs and the standard worker type. The data engineer needs to fix the job without rewriting the script. What should the data engineer do?

Accepted Answer

Change the worker type to G.2X to get more memory per worker.. Out of Memory errors in AWS Glue are typically caused by insufficient per-executor memory during operations like joins. Changing the worker type to G.2X doubles the memory per DPU (from 16 GB to 32 GB), directly addressing the OOM issue without rewriting the script. Option D is wrong because increasing the number of DPUs adds more executors but does not increase the memory per executor; it only increases parallelism, which may not resolve OOM if a single executor runs out of memory. Option A (splitting input files) does not reduce the memory footprint of joins. Option C (decreasing DPUs) reduces resources and worsens the problem.

Answer

Split the input CSV files into smaller partitions.

Answer

Decrease the number of DPUs to reduce memory contention.

Answer

Increase the number of DPUs for the Glue job to 20.

Question 5

A media company uses Amazon Kinesis Data Firehose to ingest log data from web servers into Amazon S3. The data is then processed by AWS Glue jobs. The company wants to ensure that data is delivered to S3 within 5 minutes of ingestion. Currently, the Firehose delivery stream is configured with a buffer interval of 300 seconds and a buffer size of 5 MB. The log data arrives at a rate of 2 MB per second. The data engineer notices that some log files are delayed by up to 10 minutes. The company cannot change the buffer size due to downstream requirements. What should the data engineer do to meet the 5-minute delivery requirement?

Accepted Answer

Decrease the buffer interval to 120 seconds.. Decreasing the buffer interval to 120 seconds ensures that data is delivered to S3 within 2 minutes, meeting the 5-minute requirement. The current buffer interval of 300 seconds (5 minutes) is the primary cause of delays up to 10 minutes. Option A is incorrect because increasing the buffer interval would increase delay. Option B is incorrect because increasing buffer size would also increase delay, as it takes longer to fill. Option C is incorrect because compression does not affect the buffer interval; data still waits up to 300 seconds.

Answer

Increase the buffer interval to 600 seconds to reduce the number of delivery attempts.

Answer

Increase the buffer size to 10 MB to ensure data is delivered in larger chunks.

Answer

Enable GZIP compression on the Firehose stream to reduce data size.

Question 6

A gaming company collects player event data from mobile devices. The data is sent to an Amazon API Gateway endpoint, which triggers an AWS Lambda function that writes the data to an Amazon DynamoDB table. The company wants to also store the data in Amazon S3 for historical analysis. The data volume is about 100 GB per day. The data engineer needs to design a solution to copy data from DynamoDB to S3 with minimal impact on the DynamoDB table. What should the data engineer do?

Accepted Answer

Enable DynamoDB Streams on the table and configure a Lambda function to write changes to S3.. Using DynamoDB Streams with a Lambda function that writes to S3 enables real-time replication with minimal impact on the DynamoDB table. Option B is wrong because creating a global secondary index does not export data to S3; it is used to improve query performance. Option C is wrong because using AWS Glue to scan the DynamoDB table would consume read capacity units, potentially impacting performance. Option D is wrong because the DynamoDB Export to S3 feature is a one-time or scheduled export, not continuous, and may not be suitable for near-real-time needs.

Answer

Create a global secondary index on the table and export the index to S3.

Answer

Use AWS Glue to scan the DynamoDB table and write results to S3 every hour.

Answer

Use the DynamoDB Export to S3 feature to export the entire table daily.

Question 7

A healthcare company processes patient records in near-real-time using Amazon Kinesis Data Streams. Each record contains sensitive personal health information (PHI). The data must be encrypted at rest and in transit. The company also needs to audit access to the data. The data engineer is designing the ingestion pipeline. Which combination of services and configurations meets these requirements?

Accepted Answer

Use Kinesis Data Streams with SSE-KMS and TLS, and enable CloudTrail for data events.. Kinesis Data Streams supports server-side encryption (SSE) using AWS KMS for at-rest encryption, and TLS for in-transit. CloudTrail can log Kinesis API calls for auditing. Option A lacks encryption at rest. Option B lacks auditing. Option D is wrong because S3 does not replace Kinesis for streaming.

Answer

Use Kinesis Data Firehose to deliver data to S3 with SSE-S3, and enable CloudTrail for S3.

Answer

Use Kinesis Data Streams with TLS and enable CloudTrail for auditing. Do not enable SSE.

Answer

Use Kinesis Data Streams with SSE-KMS and TLS. Do not enable any auditing.

Question 8

A logistics company uses AWS Glue to process GPS data from delivery trucks. The data is stored in Amazon S3 as JSON files. The Glue job reads the JSON files, converts them to Parquet, and writes them back to S3. The company notices that the Glue job takes too long to complete. The data engineer wants to improve the job's performance without changing the code. What should the data engineer do?

Accepted Answer

Change the worker type to G.2X.. Changing the worker type to G.2X provides more memory and CPU per worker, which improves performance for memory-intensive tasks like converting JSON to Parquet. Option A is wrong because increasing the number of DPUs can help with parallelism but may still be limited by per-worker memory; upgrading worker type is more efficient. Option C is wrong because G.1X is the default and provides less resources than G.2X. Option D is wrong because decreasing DPUs would reduce parallelism and worsen performance.

Answer

Increase the number of DPUs to 20.

Answer

Change the worker type to G.1X.

Answer

Decrease the number of DPUs to 5 to reduce overhead.

Question 9

A social media company ingests user activity data from multiple sources using Amazon Kinesis Data Firehose. The data is delivered to Amazon S3 in near-real-time. The company wants to transform the data by adding a timestamp and masking email addresses before storing it in S3. The transformation should be applied to all records. What is the most cost-effective way to implement this transformation?

Accepted Answer

Configure the Firehose delivery stream to invoke a Lambda function for data transformation.. The most cost-effective way is to configure the Kinesis Data Firehose delivery stream to invoke an AWS Lambda function for data transformation before the data is delivered to S3. This approach is serverless and only runs when data is flowing, so you pay only for the compute time used during transformation. Option A (Athena CTAS) would require querying after storage, adding cost and latency. Option B (Glue batch job) runs on a schedule regardless of data volume, leading to idle costs, and introduces latency. Option C (S3 Events with Lambda) triggers after data is already stored, meaning the data is first stored in raw form, then transformed, doubling storage costs and adding complexity. Therefore, the Firehose-integrated Lambda is the most cost-effective and low-latency option.

Answer

Use Amazon Athena to run a CTAS query that transforms the data and writes to a new location.

Answer

Use AWS Glue to schedule a batch job every 5 minutes to transform the data.

Answer

Use Amazon S3 Events to trigger a Lambda function whenever a new object is created.

Question 10

An e-commerce company uses AWS Glue to process clickstream data from its website. The data is stored in Amazon S3 in partitioned Parquet format by date and hour. A recent increase in traffic has caused the Glue job to fail with 'Java heap space' errors. The job runs with 10 DPUs and uses Spark's default configurations. The data engineer needs to resolve the memory issue without modifying the ETL script. What should the data engineer do?

Accepted Answer

Increase the Spark configuration 'spark.sql.shuffle.partitions' to 500.. Increasing 'spark.sql.shuffle.partitions' to 500 reduces the amount of data handled per partition during shuffle operations, which alleviates memory pressure and prevents 'Java heap space' errors. This is a configuration change that does not require modifying the ETL script. Option A is wrong because decreasing partitions increases data per partition, worsening memory issues. Option B is wrong because changing worker type to G.1X (which has more memory per executor) might help but does not address the root cause if the issue is due to too few shuffle partitions; it is also not a direct fix for shuffle memory. Option D is wrong because increasing DPUs adds more executors but does not solve the per-executor memory issue caused by large shuffle partitions.

Answer

Decrease the Spark configuration 'spark.sql.shuffle.partitions' to 50.

Answer

Change the worker type to G.1X.

Answer

Increase the number of DPUs to 20.

Question 11

A data engineer needs to ingest data from an on-premises Apache Kafka cluster into Amazon S3. The data engineer wants to minimize operational overhead and avoid managing any servers. Which AWS service should the data engineer use?

Accepted Answer

Amazon Managed Streaming for Apache Kafka (Amazon MSK). Amazon MSK is correct. It is a fully managed Apache Kafka service that can ingest data from on-premises Kafka via MirrorMaker or replication, then sink to S3, minimizing operational overhead. AWS Glue (B) is for ETL jobs, not real-time Kafka ingestion. Kinesis Data Analytics (C) is for analyzing streaming data, not ingestion. Amazon Kinesis Data Streams (D) is a different service not directly compatible with Kafka without custom connectors.

Answer

AWS Glue

Answer

Amazon Kinesis Data Analytics

Answer

Amazon Kinesis Data Streams

Question 12

A data engineer needs to ingest streaming data from an e-commerce application into Amazon S3 for near-real-time analytics. The solution must handle variable throughput and allow reprocessing of failed records. Which TWO AWS services should the engineer use? (Choose two.)

Accepted Answer

Amazon Kinesis Data Streams. Amazon Kinesis Data Streams is correct because it provides a durable, scalable, real-time data ingestion layer that can handle variable throughput by sharding, and it retains data for up to 365 days, enabling reprocessing of failed records by replaying from a specific sequence number or timestamp.

Answer

Amazon SQS

Answer

Amazon DynamoDB Streams

Answer

Amazon RDS

Question 13

A data engineering team is building a pipeline to transform CSV files uploaded to Amazon S3 into Parquet format using AWS Glue. The transformation must be serverless and handle files that arrive at irregular intervals. Which TWO actions should the team take? (Choose two.)

Accepted Answer

Use an AWS Glue ETL job to convert CSV to Parquet.. AWS Glue ETL jobs provide a serverless Spark-based environment that can directly read CSV files from S3 and write them as Parquet, meeting the requirement for serverless transformation. Glue handles schema inference and conversion without managing any infrastructure, making it ideal for irregularly scheduled data.

Answer

Configure an Amazon EMR cluster with Apache Spark for on-demand transformation.

Answer

Use AWS Data Pipeline to schedule a periodic transformation.

Answer

Use Amazon Redshift Spectrum to convert files during query execution.

Question 14

A company ingests IoT sensor data into Amazon Kinesis Data Streams. The data must be enriched with device metadata from Amazon DynamoDB and then stored in Amazon S3 in Apache Parquet format. The solution must minimize latency and cost. Which THREE steps should a data engineer implement? (Choose three.)

Accepted Answer

Deliver the enriched data to Amazon Kinesis Data Firehose and enable Parquet conversion.. The correct three steps are: using Amazon Kinesis Data Analytics for Apache Flink to enrich the stream with data from DynamoDB (null), performing a DynamoDB lookup in the Flink application for each record (E), and delivering the enriched data to Amazon Kinesis Data Firehose with Parquet conversion enabled (A). Kinesis Data Analytics for Apache Flink reads from Kinesis Data Streams, enriches each record via DynamoDB lookups, and outputs the enriched stream to Kinesis Data Firehose. Firehose automatically converts data to Apache Parquet before writing to S3, minimizing latency and cost by leveraging managed services without custom code or additional infrastructure.

Answer

Configure an AWS Lambda function to read from the stream, enrich, and write to S3.

Answer

Use AWS Glue streaming ETL to enrich and convert data to Parquet.

Answer

Use Amazon EMR with Spark Streaming to process and store the data.

Question 15

A financial services company ingests stock trade data from multiple exchanges into an Amazon S3 bucket (trade-bucket). Each exchange sends a CSV file every 5 minutes. The data must be transformed into Parquet format and partitioned by exchange and date (trade_date) for efficient querying using Amazon Athena. The pipeline must handle late-arriving data (files up to 2 hours late) and ensure exactly-once processing to avoid duplicates. Currently, a scheduled AWS Glue ETL job runs every hour, reads new CSV files, converts them to Parquet, and writes to an output bucket. However, the team is experiencing data duplication: if the job fails midway, upon retry it reprocesses all files in the input folder, causing duplicates in the output. Additionally, the job takes too long because it scans all files each run. The engineer must redesign the pipeline to eliminate duplicates and improve efficiency. What should the engineer do?

Accepted Answer

Set up an S3 event notification to invoke an AWS Lambda function that starts a Glue job with a parameter containing the S3 object key of the new file; modify the Glue job to process only that file and use the file key to avoid duplicates.. The best approach. By setting up an S3 event notification to invoke a Lambda function that triggers a Glue job with the new file's S3 key as a parameter, the job processes only that specific file. Using the file key in the job logic ensures idempotency—if the job fails and retries, it reprocesses the same file key, and deduplication can be handled (e.g., by checking if the output partition already contains that file's data or using a job bookmark on the file key). This achieves exactly-once processing and incremental processing (no full scans), improving efficiency. Option A (Glue Workflows) still processes all files each run and doesn't prevent duplicates if a file arrives after the job starts. Option C (moving CSV files to an archive folder) risks race conditions if late-arriving data comes while the job is running, and does not guarantee exactly-once if the job fails mid-way. Option D (EMR with Spark Structured Streaming) is overly complex and expensive for a 5-minute CSV batch ingestion; checkpointing can handle failures but adds significant operational overhead.

Answer

Use AWS Glue Workflows to orchestrate the job and add a condition to check for duplicates before writing.

Answer

Modify the Glue job to move processed CSV files to an archive folder after successful transformation, and process only unprocessed files.

Answer

Replace Glue with Amazon EMR and use Spark Structured Streaming with checkpointing to process files incrementally.