DEA-C01 Data Ingestion and Transformation Practice Test 4 — 15 Questions

Question 1

A data pipeline uses Kinesis Data Firehose to deliver streaming data to an S3 bucket. The data volume spikes occasionally, causing the Firehose buffer to fill up and leading to increased delivery latency. The latency must remain under 60 seconds. What should be done to minimize latency?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Reduce the buffer interval to 60 seconds.. Reducing the buffer interval to 60 seconds ensures that Firehose delivers data to S3 at most every 60 seconds, directly capping latency even if the buffer size is not full. This aligns with the requirement to keep latency under 60 seconds, as Firehose delivers data when either the buffer interval or buffer size threshold is met first.

Answer

Enable GZIP compression on the Firehose delivery stream.

Answer

Increase the buffer size to 128 MB to accommodate larger batches.

Answer

Switch to Kinesis Data Streams with a Lambda consumer.

Question 2

A company runs a nightly AWS Glue ETL job that reads from a JDBC source (PostgreSQL) and writes to S3 in Parquet format. The job takes over 6 hours, but the SLA requires completion within 4 hours. The source table has 500 million rows and is updated frequently. Which approach will most reliably reduce job duration?

Accepted Answer

Partition the source table by year and use pushdown predicates in the Glue job.. Partitioning the source table by year and using pushdown predicates allows AWS Glue to read only the relevant partitions from PostgreSQL, drastically reducing the data scanned and transferred. This directly addresses the 500 million row volume by minimizing the JDBC read workload, which is the primary bottleneck. Option B, using multiple JDBC connections in parallel via hash partitioning, can improve performance but is less reliable: it significantly increases load on the source database, may hit connection limits, and its effectiveness depends on the JDBC driver's support for hashfield/hashexpression. Moreover, it still reads all rows, whereas pushdown predicates reduce the data volume at the source, making option C more reliable for meeting the 4-hour SLA.

Answer

Enable job bookmark and schedule the job to run more frequently.

Answer

Use multiple JDBC connections in parallel by setting 'hashexpression' and 'hashfield'.

Answer

Increase the number of DPUs for the Glue job to 100.

Question 3

A company uses AWS Data Pipeline to copy data from DynamoDB to S3 daily. Recently, the pipeline started failing with 'ThrottlingException' errors. The DynamoDB table has on-demand capacity. Which action should be taken to resolve the issue?

Accepted Answer

Configure the pipeline to use a retry strategy with exponential backoff.. ThrottlingException errors in AWS Data Pipeline when reading from DynamoDB indicate that the pipeline's read requests are exceeding the table's available throughput. Since the table uses on-demand capacity, which can handle spikes but has a per-second throughput limit, implementing exponential backoff in the pipeline's retry strategy allows it to reduce request rate upon throttling, aligning with AWS SDK best practices for handling DynamoDB throttling.

Answer

Increase the write capacity units of the DynamoDB table.

Answer

Replace Data Pipeline with AWS Glue using a DynamoDB connector.

Answer

Disable the pipeline's retry logic and increase the timeout.

Question 4

A data engineer needs to transform JSON data from an S3 bucket using AWS Glue. The JSON contains nested arrays and objects. Which Glue transform is best suited for flattening nested structures?

Accepted Answer

Relationalize. The Relationalize transform is specifically designed to flatten nested JSON structures (arrays and objects) into a set of related tables, making it ideal for this use case. It automatically handles complex nesting by creating separate DataFrames for each nested level and linking them via foreign keys, which is exactly what is needed when ingesting JSON with nested arrays and objects into a relational format.

Answer

Unnest

Answer

ResolveChoice

Answer

Map

Question 5

A company ingests IoT sensor data into Kinesis Data Streams. The data is then processed by a Lambda function that aggregates readings and writes to DynamoDB. The Lambda function is experiencing high error rates due to throttling. Which TWO actions would reduce throttling?

Accepted Answer

Increase the batch size in the Lambda event source mapping.. Increasing the batch size in the Lambda event source mapping allows each invocation to process more records from the Kinesis stream, reducing the number of concurrent Lambda invocations and thus lowering the risk of throttling. Option D is correct because switching DynamoDB to on-demand capacity mode eliminates write capacity limits, preventing throttling on the DynamoDB side that can cause Lambda retries and backpressure.

Answer

Increase the number of shards in the Kinesis stream.

Answer

Decrease the batch window in the Lambda event source mapping.

Answer

Increase the Lambda reserved concurrency to 1000.

Question 6

A company uses Amazon RDS for MySQL as a source for AWS DMS to replicate data to S3. The replication task is failing with 'OutOfMemory' errors on the DMS instance. The source table has 10 million rows with large BLOB columns. Which THREE changes would most likely resolve the issue?

Accepted Answer

Set the LOB column settings to 'Limited LOB mode' and specify a max LOB size.. Setting LOB columns to 'Limited LOB mode' with a specified max LOB size prevents DMS from loading entire LOBs into memory. Instead, DMS truncates LOBs to the specified size, reducing memory consumption and avoiding OutOfMemory errors when replicating large BLOB columns from MySQL to S3.

Answer

Disable logging for the DMS task to free memory.

Answer

Enable Full LOB mode to handle LOBs more efficiently.

Question 7

A data engineer is troubleshooting a Kinesis Data Firehose delivery stream that ingests JSON log data from web servers. The stream is configured to transform records with an AWS Lambda function and deliver to an Amazon S3 bucket. Recently, the stream has been failing with 'InvalidData' errors. Which action should the engineer take to resolve the issue?

Accepted Answer

Check the CloudWatch Logs for the Lambda function to identify transformation errors.. The 'InvalidData' error in Kinesis Data Firehose typically indicates that the Lambda function used for data transformation is failing or returning malformed records. By checking the CloudWatch Logs for the Lambda function, the engineer can identify specific transformation errors, such as incorrect JSON parsing, missing fields, or exceptions, which cause Firehose to reject the records. This is the most direct troubleshooting step because Firehose relies on the Lambda function to return valid transformed data in the expected format.

Answer

Verify the S3 bucket policy allows Firehose to write.

Answer

Increase the buffer size and interval in the Firehose delivery stream.

Answer

Change the data format to CSV in the Firehose configuration.

Question 8

A data engineer is setting up an Amazon Kinesis Data Analytics application to process streaming data from a Kinesis data stream named "input-stream". The application uses a reference data source from an S3 bucket. The engineer has attached the IAM policy shown in the exhibit to the application's IAM role. When starting the application, the engineer receives an 'AccessDeniedException' error. Which additional permission is required?

Accepted Answer

s3:GetObject on the S3 bucket containing the reference data. The Kinesis Data Analytics application needs to read reference data from the S3 bucket, which requires the s3:GetObject permission on the bucket and its objects. The error 'AccessDeniedException' indicates the IAM role lacks this specific permission to retrieve the reference data file. Option B correctly adds the missing s3:GetObject action to allow the application to fetch the reference data from S3.

Answer

kinesis:PutRecord on the input stream

Answer

kinesis:CreateStream on the input stream

Answer

kinesis:PutRecords on the input stream

Question 9

A company runs an e-commerce platform that generates clickstream data from millions of users. The data is ingested into Amazon Kinesis Data Streams with a shard count of 10. The data is then consumed by a Kinesis Data Analytics application that runs SQL queries to aggregate metrics in real time. Recently, the application has been falling behind, and the stream's iterator age metric is increasing. The data volume has doubled over the past month. The application currently uses a single Kinesis Data Analytics application with parallelism of 1. Which action should the data engineer take to improve the processing rate and reduce the iterator age without losing data or causing duplicates?

Accepted Answer

Increase the parallelism of the Kinesis Data Analytics application to match the number of shards.. Kinesis Data Analytics (KDA) processes data from each shard in a stream using one or more parallel operators. With a parallelism of 1, the application uses only a single processing thread, which cannot keep up with the doubled data volume across 10 shards. By increasing parallelism to match the shard count (10), KDA can read from all shards concurrently, distributing the processing load and reducing the iterator age without data loss or duplicates, as KDA manages checkpointing and exactly-once semantics internally.

Answer

Change the Kinesis Data Analytics application to use a Kinesis Data Firehose delivery stream as the source.

Answer

Reduce the retention period of the Kinesis Data Streams to 24 hours.

Answer

Increase the number of shards in the Kinesis Data Streams to 20.

Question 10

A financial services company ingests real-time stock trade data from multiple exchanges into Amazon Kinesis Data Streams. Each trade record is a JSON object containing fields: trade_id, symbol, price, quantity, and timestamp. The data is consumed by an AWS Lambda function that performs data validation and enrichment, then writes the processed records to an Amazon DynamoDB table for low-latency querying. Recently, the Lambda function has been timing out and failing to process all records. The Lambda function is configured with a 5-second timeout and 128 MB memory. The average record size is 2 KB, and the stream receives about 1000 records per second. The Lambda function's concurrency limit is 1000. Which set of actions should the data engineer take to resolve the issue without losing data?

Accepted Answer

Increase the Lambda function timeout to 60 seconds and memory to 1024 MB. Set the batch size to 100 records and enable parallelization factor of 10.. Increasing the Lambda timeout and memory addresses the processing bottleneck, while setting the batch size to 100 and enabling a parallelization factor of 10 allows each shard to process up to 10 concurrent batches, dramatically increasing throughput to handle 1000 records/sec (each shard can process 10 batches of 100 records concurrently, yielding 1000 records/sec per shard if the stream has at least 1 shard). This combination ensures no data loss by keeping up with the ingestion rate without exceeding the Lambda concurrency limit of 1000.

Answer

Increase the number of shards in the Kinesis data stream to 20 and keep the Lambda configuration unchanged.

Answer

Replace the Lambda function with a Kinesis Data Firehose delivery stream that writes directly to DynamoDB using a Lambda transformation.

Answer

Increase the Lambda function timeout to 60 seconds and memory to 1024 MB. Set the batch size to 100 records.

Question 11

A company uses AWS Glue to process streaming data from Amazon Kinesis Data Streams. The data is JSON formatted and includes a timestamp field. The company wants to partition the output in Amazon S3 by date and hour, and ensure exactly-once processing semantics. Which combination of configurations should be used?

Accepted Answer

Enable checkpointing in the AWS Glue streaming job and specify an S3 location for checkpoint data.. AWS Glue streaming jobs require checkpointing to track the progress of data consumption from Kinesis Data Streams and to ensure exactly-once processing semantics. By enabling checkpointing and specifying an S3 location, Glue periodically saves the state of processed records, allowing it to resume from the last committed offset in case of failures, thus preventing duplicates or data loss.

Answer

Disable checkpointing and use the 'exactly_once' delivery option in Kinesis Data Streams.

Answer

Use Amazon DynamoDB as a checkpoint store by configuring the Glue job with a DynamoDB connection.

Answer

Use Kinesis Client Library (KCL) checkpointing with a DynamoDB table.

Question 12

A data engineer is troubleshooting an AWS Glue ETL job that reads from Amazon S3 and writes to Amazon Redshift. The job runs successfully but writes duplicate rows into Redshift. The source data is static and does not contain duplicates. Which configuration change is most likely to resolve this issue?

Accepted Answer

Modify the job to use the 'postactions' option with a SQL statement that deletes duplicates before final insert.. The job runs successfully but writes duplicate rows because AWS Glue's Spark-based ETL jobs can retry tasks on failure, and when writing to Redshift using the JDBC connector, the default behavior is to append data without deduplication. Using the 'postactions' option with a SQL DELETE statement that removes duplicates before the final INSERT ensures that only unique rows remain, resolving the duplication without altering the source data.

Answer

Enable the 'upsert' feature in the Redshift connection by setting 'update' to true.

Answer

Use partition pruning on the S3 source to reduce the number of files read.

Answer

Increase the number of DPUs (Data Processing Units) allocated to the job.

Question 13

A company is designing a data ingestion pipeline to load CSV files from an SFTP server into Amazon S3. The files are generated hourly and range from 10 MB to 500 MB. Which AWS service should be used to orchestrate the transfer with minimal operational overhead?

Accepted Answer

AWS Transfer Family. AWS Transfer Family is the correct choice because it provides a fully managed, serverless SFTP endpoint that can directly receive files from an SFTP server and automatically store them in Amazon S3. This eliminates the need to manage any compute infrastructure or write custom code for the transfer, minimizing operational overhead for hourly CSV file ingestion.

Answer

AWS Glue

Answer

Amazon AppFlow

Answer

AWS DataSync

Question 14

A company uses Amazon Kinesis Data Firehose to ingest log data from web servers into Amazon S3. The data is in JSON format and each record is approximately 2 KB. The delivery stream is configured to buffer incoming records for 60 seconds or 5 MB, whichever comes first. The company notices that the data in S3 is delayed by up to 5 minutes during peak hours. Which action would most effectively reduce the delivery latency?

Accepted Answer

Decrease the buffer interval to 15 seconds.. The observed delay of up to 5 minutes during peak hours indicates that the buffer size threshold (5 MB) is rarely reached because each record is only ~2 KB, so the delivery stream relies on the buffer interval (60 seconds) to trigger delivery. By decreasing the buffer interval to 15 seconds, Kinesis Data Firehose will push data to S3 more frequently, directly reducing the maximum latency from 60 seconds to 15 seconds per batch, which eliminates the compounding delays caused by queuing during high-throughput periods.

Answer

Increase the buffer size to 10 MB to allow more records per delivery.

Answer

Enable compression (GZIP) on the delivery stream.

Answer

Enable data transformation with AWS Lambda to convert JSON to Parquet.

Question 15

A company is ingesting real-time clickstream data into Amazon S3 using Amazon Kinesis Data Firehose. The data is semi-structured and the company wants to transform the data into Parquet format and partition it by year, month, day, and hour. Which TWO steps should be taken to achieve this? (Choose TWO.)

Accepted Answer

Enable dynamic partitioning in Kinesis Data Firehose and specify the partition keys as year, month, day, hour extracted from the data.. Kinesis Data Firehose's dynamic partitioning feature allows you to specify partition keys (year, month, day, hour) extracted from the incoming data, and Firehose will automatically create the corresponding S3 prefix structure (e.g., year=2024/month=01/day=15/hour=10/) during delivery. Option D is correct because to convert semi-structured data to Parquet format, you can attach an AWS Lambda function as a data transformation to Firehose, which converts each record to Parquet before delivery to S3. Option E is incorrect because while Kinesis Data Firehose does support converting data to Parquet format using a schema from the AWS Glue Data Catalog, this approach requires a pre-defined Glue schema and is less flexible for semi-structured data. Moreover, the question does not mention any existing Glue Data Catalog, and the Lambda transformation in option D is a more direct and customizable method for the transformation needed.

Answer

Set up an Amazon S3 event notification to trigger an AWS Lambda function that partitions the data after delivery.

Answer

Use an AWS Glue Crawler to infer the schema and automatically partition the data in S3.

Answer

Configure Kinesis Data Firehose to convert the data to Parquet format using a schema from the AWS Glue Data Catalog.