DEA-C01 Data Ingestion and Transformation Practice Test 34 — 15 Questions

Question 1

A data engineer is using AWS Glue to run an ETL job that reads data from Amazon DynamoDB and writes to Amazon Redshift. The job fails with a 'ThroughputExceededException' error. What is the most likely cause?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

The DynamoDB table's read capacity is insufficient for the Glue job's read rate. The 'ThroughputExceededException' error occurs when AWS Glue reads from DynamoDB at a rate that exceeds the table's provisioned read capacity, causing DynamoDB to throttle requests. Option A is incorrect because a timeout setting would result in a different error (e.g., 'Job run timeout'). Option B is incorrect because Redshift concurrency scaling affects query performance, not DynamoDB read throttling. Option D is incorrect because S3 permissions issues would cause 'AccessDenied' errors, not throughput exceedance.

Answer

The Glue job has a timeout setting that is too low

Answer

The Redshift cluster's concurrency scaling is insufficient

Answer

The S3 bucket where Glue writes temporary data does not have proper permissions

Question 2

A company is using Amazon Kinesis Data Streams with a Lambda consumer to process real-time events. The Lambda function is triggered by a DynamoDB stream to update a counter. Recently, the counter has been inaccurate due to duplicate processing. What is the most likely cause?

Accepted Answer

The Lambda function is not idempotent and is being retried on failures. The Lambda function is invoked at least once per record in the DynamoDB stream. If the function fails or times out, it may be retried, processing the same record again. If the function is not idempotent, this retry leads to duplicate updates to the counter. Option A is incorrect because TRIM_HORIZON is a Kinesis iterator type, not relevant to DynamoDB streams. Option B is incorrect because low reserved concurrency would cause throttling, not duplicate processing. Option D is incorrect because shard rebalancing occurs in Kinesis, not DynamoDB streams, and does not cause duplicates.

Answer

The DynamoDB stream is configured with 'TRIM_HORIZON' iterator type

Answer

The Lambda function's reserved concurrency is too low

Answer

The Kinesis stream has undergone a shard rebalance

Question 3

A data engineer is designing a data ingestion pipeline that uses Amazon Kinesis Data Firehose to deliver data to Amazon S3. The engineer wants to ensure that the data is organized in a directory structure by year, month, day, and hour. Which TWO configurations should the engineer set on the Firehose delivery stream? (Choose TWO.)

Accepted Answer

Enable dynamic partitioning. Correct options: A and B. Option A is correct because enabling dynamic partitioning allows Firehose to use partition keys from the data, such as timestamps, to automatically create folder structures in S3. Option B is correct because setting a custom prefix with the expression '!{timestamp:yyyy}/!{timestamp:MM}/!{timestamp:dd}/!{timestamp:HH}/' defines the directory hierarchy by year, month, day, and hour. Option C is incorrect because using a Lambda function to write to S3 with a desired prefix is not necessary; Firehose handles the prefix natively. Option D is incorrect because format conversion to Parquet is unrelated to directory organization. Option E is incorrect because S3 bucket versioning does not affect directory structure.

Answer

Use an AWS Lambda function to write to S3 with the desired prefix

Answer

Enable format conversion to Parquet

Answer

Configure an S3 bucket with versioning enabled

Question 4

A company is ingesting data from multiple sources into Amazon S3 using AWS Glue. The data is then transformed using Apache Spark on Amazon EMR. The data engineer wants to reduce the cost of storing and processing data by compressing the ingested files. Which THREE file formats support compression and are commonly used with Spark? (Choose THREE.)

Accepted Answer

ORC. Correct options: A, B, and E. ORC, Parquet, and Avro all support compression and are commonly used with Spark. These formats are columnar (ORC and Parquet) or row-based with efficient compression (Avro), making them suitable for analytics. JSON and CSV support compression but are not columnar and less efficient for Spark processing; they are not the best choices for reducing storage and processing costs in this context.

Answer

JSON

Answer

CSV

Question 5

A data engineer is building a data ingestion pipeline that uses AWS Lambda to process records from Amazon Kinesis Data Streams. The Lambda function writes the processed data to Amazon DynamoDB. Which TWO factors affect the maximum number of concurrent Lambda executions for this stream? (Choose TWO.)

Accepted Answer

Batch size configured for the Lambda event source mapping. Correct options: D and E. The maximum number of concurrent Lambda executions for processing a Kinesis stream is determined by the number of shards in the stream, because each shard can have at most one concurrent Lambda invocation processing records from that shard. The batch size configured for the Lambda event source mapping also affects concurrency: a smaller batch size can lead to more invocations per shard over time, as each shard processes batches sequentially. Options A (DynamoDB read capacity units) and B (Lambda memory allocation) do not directly limit concurrency from the stream. Option C (stream name) is just an identifier and has no effect on concurrency.

Answer

DynamoDB table's read capacity units

Answer

Lambda function's memory allocation

Answer

Kinesis stream name

Question 6

Refer to the exhibit. A data engineer created this IAM policy for a Lambda function that reads from a Kinesis stream and writes to an S3 bucket. The Lambda function fails with an 'AccessDenied' error when trying to write to S3. What is the missing permission?

Accepted Answer

s3:PutObjectAcl on the bucket. The IAM policy includes s3:PutObject, which is sufficient for writing objects to S3. The AccessDenied error indicates the bucket policy or the resource ARN in the policy is misconfigured. Among the given options, s3:PutObjectAcl might be required if the bucket is configured to require ACLs on write operations.

Answer

s3:ListBucket on the bucket

Answer

s3:GetObject on the bucket

Answer

s3:DeleteObject on the bucket

Question 7

Refer to the exhibit. A data engineer runs this AWS Glue job but it fails with an error that the table 'orders' does not exist in the 'sales_db' database. The engineer has verified that the table exists in the AWS Glue Data Catalog. What is the most likely cause of the error?

Accepted Answer

The IAM role used by the Glue job does not have permission to read the Data Catalog. The error 'table orders does not exist in sales_db' occurs because the IAM role associated with the Glue job does not have the necessary permissions to read the AWS Glue Data Catalog. Even though the table exists, the job requires 'glue:GetTable' permission on the 'sales_db.orders' table to discover it. Option B is incorrect because job bookmark settings control data processing state, not table discovery. Option C is incorrect because the script using 'create_dynamic_frame.from_catalog' is syntactically correct. Option D is incorrect because the error specifically indicates the table is missing, not the S3 path.

Answer

The Glue job has job bookmark enabled and is skipping the table

Answer

The script uses 'create_dynamic_frame.from_catalog' incorrectly

Answer

The S3 path 's3://data-lake/raw/' does not exist

Question 8

Refer to the exhibit. A data engineer is configuring a Kinesis Data Firehose delivery stream. The stream is expected to receive bursts of 10 MB of data every 2 minutes. What is the maximum time it will take for data to be delivered to S3 during a burst?

Accepted Answer

300 seconds. The Kinesis Data Firehose delivery stream is configured with a buffer interval of 300 seconds (5 minutes) and a buffer size of 5 MB. During a burst of 10 MB every 2 minutes, the buffer will fill to 5 MB in 1 minute, so the buffer will be flushed every minute due to reaching the size threshold. However, the question asks for the maximum time it will take for data to be delivered to S3 during a burst. The maximum time is determined by the buffer interval, which is 300 seconds. Even though the size trigger may cause earlier flushes, the maximum possible time before delivery is the interval setting of 300 seconds. Therefore, the correct answer is 300 seconds (option A).

Answer

60 seconds

Answer

1 second

Answer

600 seconds

Question 9

A company streams clickstream data from websites to Amazon Kinesis Data Streams. A Lambda function processes each record and writes it to Amazon S3. Recently, the function has been timing out under high load. Which solution should a data engineer implement to handle the increased throughput?

Accepted Answer

Increase the number of shards in the Kinesis data stream.. Increasing the number of shards in the Kinesis data stream increases the level of parallelism. Each shard can be processed by a separate Lambda invocation, allowing more concurrent processing of records. This directly addresses the high load and timeout issue. Option A is incorrect because increasing the timeout does not increase throughput; it only allows the function to run longer, but under high load it will still timeout. Option C is incorrect because increasing memory may improve performance per invocation but does not increase the number of concurrent invocations; the bottleneck is limited by the number of shards. Option D is incorrect because S3 Event Notifications are for object creation events in S3, not for real-time streaming; they do not help with Kinesis ingestion.

Answer

Increase the Lambda function's timeout value.

Answer

Increase the memory allocated to the Lambda function.

Answer

Configure Amazon S3 Event Notifications to trigger Lambda directly.

Question 10

A data engineer needs to ingest JSON files from an on-premises SFTP server into Amazon S3. The files are uploaded daily and each file is up to 500 MB. The solution must be serverless and minimize cost. Which service should the engineer use?

Accepted Answer

AWS Transfer Family (SFTP) endpoint.. AWS Transfer Family provides a fully managed SFTP endpoint that directly integrates with Amazon S3, enabling file transfers without managing servers. Option A (Kinesis Data Firehose) is designed for streaming data, not file-based SFTP ingestion. Option B (AWS DataSync with an on-premises agent) requires installing and managing an agent, which adds operational overhead and is not fully serverless. Option C (S3 Transfer Acceleration) accelerates transfers to S3 but does not support ingesting from SFTP sources.

Answer

Amazon Kinesis Data Firehose.

Answer

AWS DataSync with an on-premises agent.

Answer

Amazon S3 Transfer Acceleration.

Question 11

A company uses AWS Glue ETL to transform data from Amazon RDS for PostgreSQL to Amazon S3. The transformation includes joining several tables and aggregating millions of rows. The job runs successfully but takes over 2 hours. The data engineer wants to reduce runtime. Which action is MOST effective?

Accepted Answer

Increase the number of DPUs for the Glue job.. Increasing the number of DPUs (Data Processing Units) in AWS Glue ETL jobs allows more parallel processing of the transformation, which directly reduces runtime for CPU-bound or memory-bound tasks like joining and aggregating millions of rows. Option A: Auto Scaling adjusts DPUs based on workload but does not guarantee maximum performance; it may still be limited by the initial DPU allocation. Option B: DynamicFrames vs DataFrames performance difference is minimal for such operations; this would not significantly reduce runtime. Option D: Converting source data to Parquet applies to data in S3, but the source is Amazon RDS, so this conversion does not help with reading from the database. Therefore, increasing DPUs is the most effective action.

Answer

Enable Auto Scaling for the Glue job.

Answer

Use AWS Glue DynamicFrames instead of DataFrames.

Answer

Convert the source data to Parquet format.

Question 12

An organization uses AWS Lake Formation to manage a data lake in S3. A new data engineer needs to create a Glue ETL job that reads from a Lake Formation-managed table. The engineer has been granted SELECT permission on the table via Lake Formation. However, the job fails with an AccessDenied error. What is the MOST likely cause?

Accepted Answer

The IAM role used by the Glue job does not have Lake Formation permissions.. The IAM role used by the Glue job must have Lake Formation permissions (such as lakeformation:GetDataAccess) to access the table. Without these permissions, the job will fail with an AccessDenied error even if the S3 bucket policy allows access. Option B is incorrect: S3 bucket policies are not the primary issue because Lake Formation manages fine-grained access; the role must have Lake Formation permissions. Option C is incorrect: the table is already registered with Lake Formation if it is a Lake Formation-managed table; registration is a prerequisite for granting permissions. Option D is incorrect: Lake Formation does not require a specific VPC; Glue jobs can access Lake Formation over the internet or via a VPC endpoint, but that is not the cause of the AccessDenied error.

Answer

The S3 bucket policy does not allow the Glue job to access the data.

Answer

The table has not been registered with Lake Formation.

Answer

The Glue job is not running in the same VPC as Lake Formation.

Question 13

A company needs to ingest real-time sensor data from thousands of IoT devices into Amazon S3, with a latency of less than 1 minute. The data must be transformed (e.g., convert to Parquet) before landing in S3. Which combination of services is MOST cost-effective?

Accepted Answer

Amazon Kinesis Data Streams to Amazon Kinesis Data Firehose to S3.. The most cost-effective because Amazon Kinesis Data Firehose directly ingests data from Kinesis Data Streams, can perform transformations (e.g., convert to Parquet) using built-in or Lambda functions, and delivers to S3 with low latency (under 1 minute). Option A is incorrect because AWS Lambda, while capable of transformation, does not scale cost-effectively for thousands of devices due to per-invocation costs and concurrency limits. Option B is incorrect because Kinesis Data Analytics is designed for complex stream processing with SQL, not simple transformations, and adds unnecessary cost. Option C is incorrect because AWS Glue streaming ETL is more suited for near-real-time batch processing and can incur higher costs and latency for high-throughput ingestion.

Answer

Amazon Kinesis Data Streams to AWS Lambda to S3.

Answer

Amazon Kinesis Data Streams to Amazon Kinesis Data Analytics to S3.

Answer

Amazon Kinesis Data Streams to AWS Glue streaming ETL to S3.

Question 14

A data engineer runs an AWS Glue ETL job that writes to a table in the AWS Glue Data Catalog. The job fails occasionally with the error 'Resource Not Found' for the table. The table exists. What is a likely cause?

Accepted Answer

The job is using an outdated version of the table schema.. AWS Glue jobs can cache the table schema at job start. If the table schema is updated while the job is running, the job may still reference the old schema version, which may no longer exist in the Data Catalog, causing a 'Resource Not Found' error even though the table exists. Option B is incorrect because concurrent writes do not cause this specific error; they may cause conflicts but not a 'Resource Not Found' for the table itself. Option C is incorrect because if the table exists and the location is wrong, it would typically result in a different error (e.g., 'Access Denied' or 'PathNotFound') rather than 'Resource Not Found' for the table. Option D is incorrect because special characters in the job name do not affect table access.

Answer

Multiple Glue jobs are writing to the same table concurrently.

Answer

The table location in S3 is incorrect.

Answer

The Glue job name contains special characters.

Question 15

A company uses Amazon Kinesis Data Firehose to deliver data to an Amazon S3 bucket. The data delivery is delayed by up to 5 minutes. The engineer wants to reduce the delay to under 1 minute. Which parameter should be adjusted?

Accepted Answer

Decrease the buffer interval in Kinesis Data Firehose.. Decreasing the buffer interval reduces the time Kinesis Data Firehose waits before delivering a batch, thus lowering latency to under 1 minute. Option A is incorrect because error logging to CloudWatch does not affect delivery timing. Option B is incorrect because increasing the buffer size would actually increase the delay as Firehose waits for more data to accumulate. Option C is incorrect because enabling data compression reduces storage size but has no impact on delivery frequency.

Answer

Enable error logging to CloudWatch.

Answer

Increase the buffer size in Kinesis Data Firehose.

Answer

Enable data compression.