DEA-C01 Data Ingestion and Transformation Practice Test 17 — 15 Questions

Question 1

A company ingests streaming data from IoT devices into Amazon Kinesis Data Streams. The data must be transformed in real-time using custom Python code before being stored in Amazon S3. Which AWS service should be used to perform this transformation?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Amazon Kinesis Data Analytics for Apache Flink. Amazon Kinesis Data Analytics for Apache Flink supports custom Apache Flink applications written in Java, Scala, or Python (via the Table API and PyFlink), enabling real-time transformation of streaming data with arbitrary business logic. It is the only option that natively supports custom Python code operating on a Kinesis stream in real time before writing to S3.

Answer

Amazon EMR

Answer

Amazon Kinesis Data Firehose

Answer

AWS Glue

Question 2

A company is using Amazon Kinesis Data Streams with a Lambda consumer to process real-time events. The Lambda function is triggered by a DynamoDB stream to update a counter. Recently, the counter has been inaccurate due to duplicate processing. What is the most likely cause?

Accepted Answer

The Lambda function is not idempotent and is being retried on failures. The Lambda function is invoked at least once per record in the DynamoDB stream. If the function fails or times out, it may be retried, processing the same record again. If the function is not idempotent, this retry leads to duplicate updates to the counter. Option A is incorrect because TRIM_HORIZON is a Kinesis iterator type, not relevant to DynamoDB streams. Option B is incorrect because low reserved concurrency would cause throttling, not duplicate processing. Option D is incorrect because shard rebalancing occurs in Kinesis, not DynamoDB streams, and does not cause duplicates.

Answer

The DynamoDB stream is configured with 'TRIM_HORIZON' iterator type

Answer

The Lambda function's reserved concurrency is too low

Answer

The Kinesis stream has undergone a shard rebalance

Question 3

A data engineer is designing a data ingestion pipeline that uses Amazon Kinesis Data Firehose to deliver data to Amazon S3. The engineer wants to ensure that the data is organized in a directory structure by year, month, day, and hour. Which TWO configurations should the engineer set on the Firehose delivery stream? (Choose TWO.)

Accepted Answer

Enable dynamic partitioning. Option A is correct because enabling dynamic partitioning on the Firehose delivery stream allows Firehose to automatically route incoming records into different S3 prefixes based on record attributes or the ingestion timestamp, which is exactly what is needed to organize data into year/month/day/hour folders. Option B is correct because Firehose supports the timestamp namespace in the S3 prefix, and the expression '!{timestamp:yyyy}/!{timestamp:MM}/!{timestamp:dd}/!{timestamp:HH}/' evaluates to the record's timestamp components, producing the required year/month/day/hour directory structure. Together, dynamic partitioning plus the timestamp-based prefix expression deliver the desired partitioned layout without custom code. Option C is not needed because Firehose natively supports S3 prefixing and partitioning, so a Lambda function to write to S3 would be redundant and outside the delivery stream configuration. Option D is incorrect because format conversion to Parquet changes the data format and compression, not the directory structure. Option E is incorrect because S3 bucket versioning affects object version retention and has no bearing on how Firehose organizes prefixes by time.

Answer

Use an AWS Lambda function to write to S3 with the desired prefix

Answer

Enable format conversion to Parquet

Answer

Configure an S3 bucket with versioning enabled

Question 4

A company is ingesting data from multiple sources into Amazon S3 using AWS Glue. The data is then transformed using Apache Spark on Amazon EMR. The data engineer wants to reduce the cost of storing and processing data by compressing the ingested files. Which THREE file formats support compression and are commonly used with Spark? (Choose THREE.)

Accepted Answer

ORC. Correct options: A, B, and E. ORC, Parquet, and Avro all support compression and are commonly used with Spark. These formats are columnar (ORC and Parquet) or row-based with efficient compression (Avro), making them suitable for analytics. JSON and CSV support compression but are not columnar and less efficient for Spark processing; they are not the best choices for reducing storage and processing costs in this context.

Answer

JSON

Answer

CSV

Question 5

A data engineer is building a data ingestion pipeline that uses AWS Lambda to process records from Amazon Kinesis Data Streams. The Lambda function writes the processed data to Amazon DynamoDB. Which TWO factors affect the maximum number of concurrent Lambda executions for this stream? (Choose TWO.)

Accepted Answer

Batch size configured for the Lambda event source mapping. For Kinesis-triggered Lambda, concurrency is governed by the stream's shard topology and the event source mapping configuration. Option E is correct because each shard in the Kinesis stream can have at most one concurrent Lambda invocation per shard (one consumer per shard), so the number of shards directly caps the maximum concurrency. Option D is correct because the batch size configured on the event source mapping determines how many records are delivered per invocation, which influences how the shard's processing is chunked and thus the effective concurrency behavior across shards. Option A is not correct because DynamoDB read capacity units affect the function's ability to read from DynamoDB, not the Lambda concurrency for the Kinesis source. Option B is not correct because memory allocation affects CPU and performance per invocation, not the maximum number of concurrent executions for a Kinesis stream. Option C is not correct because the stream name is just an identifier and has no bearing on concurrency limits.

Answer

DynamoDB table's read capacity units

Answer

Lambda function's memory allocation

Answer

Kinesis stream name

Question 6

Refer to the exhibit. A data engineer created this IAM policy for a Lambda function that reads from a Kinesis stream and writes to an S3 bucket. The Lambda function fails with an 'AccessDenied' error when trying to write to S3. What is the missing permission?

Accepted Answer

s3:PutObjectAcl on the bucket. The Lambda function needs to write objects to S3, and the missing permission is s3:PutObjectAcl, which is required when the write operation includes setting an ACL on the object (common when the SDK or application specifies a canned ACL like 'bucket-owner-full-control'). Without s3:PutObjectAcl, the PutObject call fails with AccessDenied even if s3:PutObject is granted. This is a classic IAM policy gap where the action is implicitly required by the API call parameters.

Answer

s3:ListBucket on the bucket

Answer

s3:GetObject on the bucket

Answer

s3:DeleteObject on the bucket

Question 7

Refer to the exhibit. A data engineer runs this AWS Glue job but it fails with an error that the table 'orders' does not exist in the 'sales_db' database. The engineer has verified that the table exists in the AWS Glue Data Catalog. What is the most likely cause of the error?

Accepted Answer

The IAM role used by the Glue job does not have permission to read the Data Catalog. The error 'table orders does not exist in sales_db' occurs because the IAM role associated with the Glue job does not have the necessary permissions to read the AWS Glue Data Catalog. Even though the table exists, the job requires 'glue:GetTable' permission on the 'sales_db.orders' table to discover it. Option B is incorrect because job bookmark settings control data processing state, not table discovery. Option C is incorrect because the script using 'create_dynamic_frame.from_catalog' is syntactically correct. Option D is incorrect because the error specifically indicates the table is missing, not the S3 path.

Answer

The Glue job has job bookmark enabled and is skipping the table

Answer

The script uses 'create_dynamic_frame.from_catalog' incorrectly

Answer

The S3 path 's3://data-lake/raw/' does not exist

Question 8

Refer to the exhibit. A data engineer is configuring a Kinesis Data Firehose delivery stream. The stream is expected to receive bursts of 10 MB of data every 2 minutes. What is the maximum time it will take for data to be delivered to S3 during a burst?

Accepted Answer

300 seconds. The Kinesis Data Firehose delivery stream is configured with a buffer interval of 300 seconds (5 minutes) and a buffer size of 5 MB. During a burst of 10 MB every 2 minutes, the buffer will fill to 5 MB in 1 minute, so the buffer will be flushed every minute due to reaching the size threshold. However, the question asks for the maximum time it will take for data to be delivered to S3 during a burst. The maximum time is determined by the buffer interval, which is 300 seconds. Even though the size trigger may cause earlier flushes, the maximum possible time before delivery is the interval setting of 300 seconds. Therefore, the correct answer is 300 seconds (option A).

Answer

60 seconds

Answer

1 second

Answer

600 seconds

Question 9

A company streams clickstream data from websites to Amazon Kinesis Data Streams. A Lambda function processes each record and writes it to Amazon S3. Recently, the function has been timing out under high load. Which solution should a data engineer implement to handle the increased throughput?

Accepted Answer

Increase the number of shards in the Kinesis data stream.. Increasing the number of shards in the Kinesis data stream increases the level of parallelism. Each shard can be processed by a separate Lambda invocation, allowing more concurrent processing of records. This directly addresses the high load and timeout issue. Option A is incorrect because increasing the timeout does not increase throughput; it only allows the function to run longer, but under high load it will still timeout. Option C is incorrect because increasing memory may improve performance per invocation but does not increase the number of concurrent invocations; the bottleneck is limited by the number of shards. Option D is incorrect because S3 Event Notifications are for object creation events in S3, not for real-time streaming; they do not help with Kinesis ingestion.

Answer

Increase the Lambda function's timeout value.

Answer

Increase the memory allocated to the Lambda function.

Answer

Configure Amazon S3 Event Notifications to trigger Lambda directly.

Question 10

A data engineer needs to ingest JSON files from an on-premises SFTP server into Amazon S3. The files are uploaded daily and each file is up to 500 MB. The solution must be serverless and minimize cost. Which service should the engineer use?

Accepted Answer

AWS Transfer Family (SFTP) endpoint.. AWS Transfer Family provides a fully managed SFTP endpoint that directly integrates with Amazon S3, enabling file transfers without managing servers. Option A (Kinesis Data Firehose) is designed for streaming data, not file-based SFTP ingestion. Option B (AWS DataSync with an on-premises agent) requires installing and managing an agent, which adds operational overhead and is not fully serverless. Option C (S3 Transfer Acceleration) accelerates transfers to S3 but does not support ingesting from SFTP sources.

Answer

Amazon Kinesis Data Firehose.

Answer

AWS DataSync with an on-premises agent.

Answer

Amazon S3 Transfer Acceleration.

Question 11

A company uses AWS Glue ETL to transform data from Amazon RDS for PostgreSQL to Amazon S3. The transformation includes joining several tables and aggregating millions of rows. The job runs successfully but takes over 2 hours. The data engineer wants to reduce runtime. Which action is MOST effective?

Accepted Answer

Increase the number of DPUs for the Glue job.. Increasing the number of DPUs (Data Processing Units) for the Glue job directly scales the compute resources allocated to the job, allowing more parallel processing of the join and aggregation operations. This is the most effective action to reduce runtime for a CPU-intensive transformation on millions of rows.

Answer

Enable Auto Scaling for the Glue job.

Answer

Use AWS Glue DynamicFrames instead of DataFrames.

Answer

Convert the source data to Parquet format.

Question 12

An organization uses AWS Lake Formation to manage a data lake in S3. A new data engineer needs to create a Glue ETL job that reads from a Lake Formation-managed table. The engineer has been granted SELECT permission on the table via Lake Formation. However, the job fails with an AccessDenied error. What is the MOST likely cause?

Accepted Answer

The IAM role used by the Glue job does not have Lake Formation permissions.. The IAM role used by the Glue job must have Lake Formation permissions (such as lakeformation:GetDataAccess) to access the table. Without these permissions, the job will fail with an AccessDenied error even if the S3 bucket policy allows access. Option B is incorrect: S3 bucket policies are not the primary issue because Lake Formation manages fine-grained access; the role must have Lake Formation permissions. Option C is incorrect: the table is already registered with Lake Formation if it is a Lake Formation-managed table; registration is a prerequisite for granting permissions. Option D is incorrect: Lake Formation does not require a specific VPC; Glue jobs can access Lake Formation over the internet or via a VPC endpoint, but that is not the cause of the AccessDenied error.

Answer

The S3 bucket policy does not allow the Glue job to access the data.

Answer

The table has not been registered with Lake Formation.

Answer

The Glue job is not running in the same VPC as Lake Formation.

Question 13

A company needs to ingest real-time sensor data from thousands of IoT devices into Amazon S3, with a latency of less than 1 minute. The data must be transformed (e.g., convert to Parquet) before landing in S3. Which combination of services is MOST cost-effective?

Accepted Answer

Amazon Kinesis Data Streams to Amazon Kinesis Data Firehose to S3.. The most cost-effective because Amazon Kinesis Data Firehose directly ingests data from Kinesis Data Streams, can perform transformations (e.g., convert to Parquet) using built-in or Lambda functions, and delivers to S3 with low latency (under 1 minute). Option A is incorrect because AWS Lambda, while capable of transformation, does not scale cost-effectively for thousands of devices due to per-invocation costs and concurrency limits. Option B is incorrect because Kinesis Data Analytics is designed for complex stream processing with SQL, not simple transformations, and adds unnecessary cost. Option C is incorrect because AWS Glue streaming ETL is more suited for near-real-time batch processing and can incur higher costs and latency for high-throughput ingestion.

Answer

Amazon Kinesis Data Streams to AWS Lambda to S3.

Answer

Amazon Kinesis Data Streams to Amazon Kinesis Data Analytics to S3.

Answer

Amazon Kinesis Data Streams to AWS Glue streaming ETL to S3.

Question 14

A data engineer runs an AWS Glue ETL job that writes to a table in the AWS Glue Data Catalog. The job fails occasionally with the error 'Resource Not Found' for the table. The table exists. What is a likely cause?

Accepted Answer

The job is using an outdated version of the table schema.. A 'Resource Not Found' error for a table that exists typically occurs when the Glue job's cached or referenced table version is stale — for example, the job was created against an older schema version and the catalog metadata has since changed. Refreshing the job's table reference or re-running with the current catalog version resolves it. This is a known Glue behavior when schema evolution or catalog updates occur between job definition and execution.

Answer

Multiple Glue jobs are writing to the same table concurrently.

Answer

The table location in S3 is incorrect.

Answer

The Glue job name contains special characters.

Question 15

A data engineer is building a streaming pipeline using Amazon Kinesis Data Streams. The data must be enriched with reference data from a DynamoDB table before being written to S3. The engineer wants to minimize latency. Which architecture is BEST?

Accepted Answer

Use Kinesis Data Analytics for Apache Flink to enrich and output to Firehose.. (Kinesis Data Analytics for Apache Flink) is the best choice because it supports low-latency enrichment using external sources like DynamoDB via asynchronous I/O, meeting the requirement to minimize latency. Option A (AWS Glue streaming ETL) is designed for batch-oriented processing and introduces higher latency. Option C (Kinesis Data Firehose with a Lambda function) may experience cold starts and limited concurrency, increasing latency. Option D (Lambda polling the stream) also suffers from cold starts and scalability issues, making it less suitable for low-latency enrichment.

Answer

Use AWS Glue streaming ETL to read from Kinesis, enrich, and write to S3.

Answer

Use Kinesis Data Firehose with a Lambda function for enrichment.

Answer

Use a Lambda function to poll the stream, enrich, and write to Firehose.