DEA-C01 Data Ingestion and Transformation Practice Test 5 — 15 Questions

Question 1

A company is using Amazon Kinesis Data Streams to process real-time stock trade data. The data is consumed by a Lambda function that calculates moving averages and stores results in Amazon DynamoDB. The Lambda function is failing with 'ProvisionedThroughputExceededException' on the DynamoDB table. The table has on-demand capacity. Which TWO actions should the engineer take to resolve this issue?

Accepted Answer

Add a dead-letter queue and configure the Lambda function to retry on failure with exponential backoff.. On-demand DynamoDB can handle bursts but may throttle if the traffic is too high. Increasing batch size reduces write frequency. Adding a retry mechanism with exponential backoff handles throttling gracefully. Option B is wrong because increasing Lambda concurrency increases write pressure. Option C is wrong because reducing batch size increases frequency. Option D is wrong because Firehose is not needed.

Answer

Decrease the batch window to 0 seconds to process records immediately.

Answer

Increase the Lambda function's reserved concurrency to process more shards.

Question 2

A data engineer is designing a data ingestion pipeline for IoT sensor data. The sensors send JSON messages every second, and the data must be stored in Amazon S3 in near real-time (within 5 minutes). The engineer also needs to transform the data by adding a timestamp and filtering out malformed records. Which THREE services should be used together?

Accepted Answer

AWS IoT Core. IoT Core can ingest sensor data, Kinesis Data Firehose can buffer and write to S3, and Lambda can transform records within Firehose. Option A is wrong because SQS is not needed. Option D is wrong because Glue is for batch ETL, not real-time. Option E is wrong because Athena is for querying.

Answer

AWS Glue

Answer

Amazon Athena

Answer

Amazon Simple Queue Service (SQS)

Question 3

The Glue job attempts to read data from 's3://my-data-bucket/input/' and write to 's3://my-data-bucket/output/'. It also tries to update a table in the Glue Data Catalog. The job fails with an access denied error. What is the MOST likely cause?

Accepted Answer

The IAM role is missing the 's3:ListBucket' permission on the bucket.. The policy allows GetObject and PutObject on the bucket, but the Glue job also needs 's3:ListBucket' permission to list objects in the bucket. Without ListBucket, the job cannot enumerate files. Option A is wrong because the resource is correct ('/*' includes all objects). Option C is wrong because the action 'glue:UpdateTable' is allowed. Option D is wrong because there is no condition key issue.

Answer

The 'glue:UpdateTable' action is not allowed on the specific table.

Answer

The policy is missing a condition key for the S3 bucket.

Answer

The resource ARN does not include the bucket itself; it only covers objects.

Question 4

The command returns an empty result, but you know there are objects in the 'logs/' prefix larger than 1000 bytes. What is the MOST likely reason?

Accepted Answer

The comparison 'Size > '1000'' uses a string instead of a number, so it never matches.. The 'Size' field in S3 is an integer, but the query compares it to a string '1000'. This causes a type mismatch, and the filter evaluates to false. Option A is wrong because the prefix is correct. Option B is wrong because the output format does not affect the query. Option D is wrong because there is no need for pagination (the command returns up to 1000 objects).

Answer

The prefix 'logs/' is incorrect; the objects are in a different prefix.

Answer

The command does not paginate, so it only checks the first 1000 objects.

Answer

The output format is set to text, but the query requires JSON.

Question 5

A data engineering team is ingesting streaming data from IoT devices into Amazon Kinesis Data Streams. The data is then consumed by an AWS Lambda function that transforms and loads it into Amazon S3. Recently, the team noticed that the Lambda function is failing with throttling errors (HTTP 429) from the Kinesis API. Which configuration change should the team make to resolve this issue?

Accepted Answer

Reduce the batch size and increase the number of shards in the Kinesis stream to increase parallelism.. The correct answer is to reduce the batch size and increase the number of shards. The Lambda function is experiencing throttling because it is trying to process too many records per invocation. Reducing the batch size lowers the number of records per invocation, and increasing shards increases parallelism. Option A is incorrect because increasing the batch size would worsen throttling. Option B is incorrect because using a DynamoDB stream is a different ingestion mechanism and doesn't address the Kinesis throttling. Option D is incorrect because disabling retries would cause data loss. Option C directly addresses the throttling by reducing load per invocation and increasing parallelism.

Answer

Disable retries on the Lambda function and configure a dead-letter queue for failed records.

Answer

Replace Kinesis Data Streams with Amazon DynamoDB Streams for ingestion.

Answer

Increase the batch size in the Lambda event source mapping to reduce the number of invocations.

Question 6

A company uses AWS Glue ETL jobs to transform data in Amazon S3. The data is partitioned by date and hour. The job reads the latest hour's data, performs aggregations, and writes results to a separate S3 bucket. The job runs every hour and processes approximately 500 MB of input data. The team notices that the job takes longer than expected, often exceeding the 1-hour window. Which action would most effectively reduce the job's runtime?

Accepted Answer

Increase the number of workers (DPUs) for the Glue job.. The correct answer is to increase the number of workers. The job processes only 500 MB, so increasing worker count (DPUs) will improve parallelism. Option B is incorrect because the job processes only one hour's data, and repartitioning would add overhead. Option C is incorrect because using Spark SQL does not inherently improve performance. Option D is incorrect because switching to a Python shell would not handle the transformation efficiently. Option A directly adds resources to speed up the job.

Answer

Use a Python shell job instead of a Spark job.

Answer

Switch from using DynamicFrame to using Spark SQL for transformations.

Answer

Repartition the input data into more partitions before reading.

Question 7

A data engineer needs to ingest data from an on-premises Oracle database into Amazon S3 on a nightly basis. The data volume is approximately 10 GB per night. The database is accessible over the internet. Which AWS service is MOST appropriate for this task?

Answer

AWS Glue ETL job with a JDBC connection

Answer

AWS DataSync

Answer

AWS Transfer Family

Answer

Amazon Kinesis Data Streams

Question 8

A company is using Amazon Kinesis Data Analytics for Apache Flink to process streaming data. The application reads from a Kinesis data stream and writes results to an Amazon S3 bucket. The team notices that the application is experiencing high latency during peak hours. The stream has 8 shards, and the application is configured with a parallelism of 4. Which action would most likely reduce the latency?

Accepted Answer

Increase the parallelism of the Flink application to 8.. The correct answer is to increase the parallelism of the Flink application to match the number of shards (8). When parallelism is lower than the number of shards, some shards are underutilized, causing backpressure and latency. Option A (increasing checkpointing interval) would reduce overhead but not address the parallelism mismatch. Option C (using a larger instance type) could help but is less effective than matching parallelism. Option D (decreasing batch size) is not applicable to Flink. Option B directly fixes the bottleneck.

Answer

Decrease the batch size in the S3 sink.

Answer

Use a larger Kinesis Data Analytics application instance type.

Answer

Increase the checkpointing interval to reduce overhead.

Question 9

A team is designing a data ingestion pipeline to load JSON files from an Amazon S3 bucket into Amazon Redshift. The files arrive every 5 minutes, and each file is between 10 MB and 50 MB. The team wants to minimize the time between file arrival and data availability in Redshift. Which approach should the team use?

Accepted Answer

Use S3 Event Notifications to trigger an AWS Lambda function that runs the COPY command to load data into Redshift.. The correct answer is to use S3 Event Notifications to invoke a Lambda function that calls the Redshift COPY command. This provides near-real-time ingestion with minimal latency. Option B (Redshift Spectrum) queries data in S3 without loading, but data is not in Redshift tables for fast querying. Option C (Kinesis Firehose) can load into Redshift but adds streaming overhead and is not optimized for batch files. Option D (AWS DMS) is for database migration, not file loading.

Answer

Schedule an AWS Glue job to run every 5 minutes to load the data.

Answer

Use Amazon Redshift Spectrum to query the data directly from S3 without loading.

Answer

Configure Amazon Kinesis Data Firehose to stream data from S3 to Redshift.

Question 10

A company uses AWS Glue ETL jobs to transform data stored in Amazon S3. The job reads data in Parquet format, applies transformations, and writes the output back to S3 in Parquet format. The team wants to improve the job's performance and reduce costs. Which action is MOST effective?

Accepted Answer

Use column pruning and predicate pushdown to read only necessary columns and filter data early.. The correct answer is to use column pruning and predicate pushdown. Reading only necessary columns and filtering early reduces data scanned and processing time. Option B (increasing worker count) increases cost and may not be needed. Option C (switching to CSV) would increase data size and slow performance. Option D (using a single large file) reduces parallelism and may harm performance.

Answer

Change the input format from Parquet to CSV to simplify parsing.

Answer

Coalesce the input data into a single large file before processing.

Answer

Increase the number of workers to maximum allowed.

Question 11

A data engineer is ingesting data from an Amazon RDS for PostgreSQL database into Amazon S3 using AWS Glue. The Glue job reads the entire table each time it runs, which takes several hours. The team wants to reduce the job duration by reading only new or updated records. Which approach should the engineer adopt?

Accepted Answer

Enable job bookmarks in AWS Glue and use a column with timestamps as the bookmark key to read only incremental data.. The correct answer is to enable Change Data Capture (CDC) using AWS DMS or enable PostgreSQL logical replication and use Glue with bookmark keys. Glue job bookmarks can track processed data if a column (like last_updated) is used. Option B (increasing workers) does not solve reading all data. Option C (partitioning) reduces data per run but still reads entire table. Option D (using Kinesis) is real-time but overkill for periodic ingestion.

Answer

Partition the table in the source database by date and read only the latest partition.

Answer

Increase the number of Glue workers to improve parallel reads.

Answer

Use Amazon Kinesis Data Streams to capture changes from PostgreSQL.

Question 12

A company uses Amazon Kinesis Data Firehose to deliver streaming data to Amazon S3. The data is in JSON format and each record is about 2 KB. The delivery stream is configured to buffer data for 60 seconds or 5 MB, whichever comes first. The team notices that the S3 objects are very small (around 1 MB) and numerous, causing high costs due to S3 PUT requests. Which configuration change should the team make to reduce the number of S3 objects?

Accepted Answer

Increase the buffer size to 50 MB and the buffer interval to 300 seconds.. The correct answer is to increase the buffer size to a larger value (e.g., 50 MB) and increase the buffer interval to 300 seconds. Larger buffers produce fewer, larger S3 objects. Option A (reducing buffer interval) would create even more objects. Option B (compression) reduces object size but not count. Option D (switching to Kinesis Streams) changes the architecture. Option C directly addresses the issue.

Answer

Enable compression (GZIP) on the delivery stream.

Answer

Reduce the buffer interval to 30 seconds and keep buffer size at 5 MB.

Answer

Switch from Kinesis Data Firehose to Amazon Kinesis Data Streams and use a Lambda function to write to S3.

Question 13

A data engineer needs to transfer 50 TB of historical data from an on-premises Hadoop cluster to Amazon S3. The company has a slow internet connection (100 Mbps). The data must be transferred within 2 weeks. Which service should the engineer recommend?

Accepted Answer

AWS Snowball Edge. The correct answer is AWS Snowball Edge, a physical device for large data transfers over slow networks. 50 TB at 100 Mbps would take over 46 days, exceeding the 2-week deadline. Option A (AWS DataSync) uses the network and would be too slow. Option B (AWS Glue) is for ETL, not transfer. Option D (Amazon Kinesis) is for streaming.

Answer

AWS DataSync

Answer

Amazon Kinesis Data Firehose

Answer

AWS Glue ETL job with JDBC connection to Hadoop

Question 14

A company is using Amazon Kinesis Data Streams to ingest clickstream data from a website. The data is consumed by an AWS Lambda function that enriches records and writes to Amazon S3. The Lambda function is experiencing high error rates due to records exceeding the 256 KB payload limit. Which TWO actions should the team take to resolve this issue?

Accepted Answer

Enable compression on the producer side before sending records to Kinesis.. The correct answers are A and C. Enabling compression reduces record size. Using Kinesis Aggregation Library (KPL) aggregates multiple records into a single larger record while staying within limits. Option B (increasing shards) does not reduce record size. Option D (using Kinesis Firehose) changes architecture but does not solve the payload limit. Option E (increasing Lambda timeout) does not address size.

Answer

Increase the Lambda function timeout.

Answer

Switch from Kinesis Data Streams to Kinesis Data Firehose.

Answer

Increase the number of shards in the Kinesis stream.

Question 15

A data engineering team is building a data lake on Amazon S3. They need to ingest data from multiple sources: (1) streaming IoT data, (2) daily CSV exports from an on-premises system via SFTP, and (3) change data capture (CDC) from an Amazon Aurora database. Which THREE services should the team use to ingest these data sources?

Accepted Answer

Amazon Kinesis Data Streams for IoT data ingestion.. The correct answers are A, B, and D. Kinesis Data Streams for streaming IoT, Transfer Family for SFTP and CDC from Aurora via DMS. Option C (Glue ETL) is for transformation, not ingestion. Option E (EMR) is for big data processing, not ingestion.

Answer

AWS Glue ETL for CDC from Aurora.

Answer

Amazon EMR for daily CSV ingestion.