DEA-C01 Data Ingestion and Transformation — Timed 100 Questions

Question 1

A company uses AWS Lambda to process records from an Amazon Kinesis Data Stream. Each record is about 50 KB. The Lambda function transforms the data and writes to Amazon DynamoDB. Recently, the Lambda function has been experiencing throttling and high error rates. The Kinesis stream has 10 shards. What is the most cost-effective solution to improve processing throughput?

100 minute time limit — choose an answer to begin.

Accepted Answer

Increase the Batch Window (MaximumBatchingWindowInSeconds) for the event source mapping.. Increasing the Batch Window (MaximumBatchingWindowInSeconds) allows the Lambda function to accumulate more records from the Kinesis stream before invoking the function, reducing the number of invocations and thus lowering the chance of throttling. This is the most cost-effective solution as it does not require additional shards, memory, or parallelization, and directly addresses the high error rates caused by excessive concurrent executions.

Answer

Increase the number of shards in the Kinesis stream.

Answer

Increase the Parallelization Factor for the Lambda event source mapping.

Answer

Increase the memory allocated to the Lambda function.

Question 2

A data engineering team is ingesting streaming data from IoT devices into Amazon Kinesis Data Streams. The data is then consumed by an AWS Lambda function that transforms each record and writes it to Amazon S3. Recently, the Lambda function started failing with 'ProvisionedThroughputExceededException' errors when writing to S3. The team has already increased the Lambda function's memory and timeout. Which action should the team take to resolve the issue?

Accepted Answer

Implement retries with exponential backoff in the Lambda function for S3 put operations.. The 'ProvisionedThroughputExceededException' error indicates that the Lambda function is being throttled by S3 due to exceeding the bucket's request rate limits. Implementing retries with exponential backoff in the Lambda function for S3 put operations is the correct solution because it allows the function to gracefully handle transient throttling errors by waiting progressively longer between retries, which aligns with AWS's guidance for managing S3 request rate limits.

Answer

Use S3 Batch Operations to write data in batches.

Answer

Increase the number of shards in the Kinesis data stream.

Answer

Enable S3 Transfer Acceleration on the destination bucket.

Question 3

A company is using AWS Glue to process data from Amazon S3. The Glue job reads CSV files and writes Parquet files to a different S3 bucket. The job occasionally fails with 'java.lang.OutOfMemoryError: Java heap space'. The data size varies. Which change should the engineer make to avoid this error?

Accepted Answer

Increase the number of DPUs allocated to the Glue job.. The 'java.lang.OutOfMemoryError: Java heap space' error in AWS Glue indicates that the Spark executors ran out of memory while processing the data. Increasing the number of DPUs (Data Processing Units) allocated to the Glue job increases the total memory available across the cluster, allowing larger datasets to be processed without hitting the heap limit. Each DPU provides 4 vCPUs and 16 GB of memory, so adding more DPUs scales memory linearly.

Answer

Convert the CSV files to JSON format before processing.

Answer

Decrease the Spark shuffle partitions in the job script.

Answer

Increase the job timeout setting.

Question 4

A data engineering team uses Amazon Kinesis Data Analytics for Apache Flink to process streaming data. They notice that the application's checkpointing is failing intermittently, causing data reprocessing. The application uses a large state. Which configuration change should the team make to improve checkpoint reliability?

Accepted Answer

Increase the checkpointing interval.. Increasing the checkpointing interval reduces the frequency of checkpoint operations, giving the system more time to complete each checkpoint before the next one starts. This alleviates backpressure and resource contention, which is critical when dealing with large state, as checkpointing large state is I/O and CPU intensive and can fail if intervals are too tight.

Answer

Disable checkpointing to avoid failures.

Answer

Switch the state backend from in-memory to RocksDB.

Answer

Increase the parallelism of the application.

Question 5

A company uses AWS Glue to perform ETL on data stored in Amazon S3. The Glue job reads CSV files, converts them to Parquet, and partitions by date. The job runs daily and processes about 500 GB of data. The team wants to optimize costs and performance. Which three actions should the team take? (Select THREE.)

Accepted Answer

Use column pruning to read only necessary columns in the Glue script.. Option B is correct because column pruning (selecting only the needed columns in the Glue/Spark script) reduces the amount of data read from S3 and processed during the shuffle, lowering I/O and DPU cost for a 500 GB daily job. Option C is correct because G.1X and G.2X worker types are the current AWS Glue worker families with more memory and compute per worker than the legacy Standard type, improving Spark performance and cost-efficiency for large ETL workloads. Option D is correct because increasing the number of DPUs (i.e., adding workers) parallelizes the Spark job, which speeds up processing of the large CSV-to-Parquet conversion and can reduce total runtime and cost when the job is not already optimally parallelized. Option A is not correct because blindly setting Spark shuffle partitions to 500 is not a general best practice; the value should be tuned to data size and cluster parallelism (often 2-3x the number of cores), and a fixed 500 may cause excessive small tasks or overhead. Option E is not correct because writing JSON instead of Parquet would increase storage size and read cost, since Parquet is columnar and compressed, and JSON does not avoid compression overhead in a beneficial way.

Answer

Increase the Spark shuffle partitions to 500.

Answer

Write the output as JSON instead of Parquet to avoid compression overhead.

Question 6

A company uses AWS Glue to process streaming data from Amazon Kinesis Data Streams. The job reads JSON records and writes Parquet to Amazon S3. Recently, the job started failing with 'Out of Memory' errors. Which change is MOST likely to resolve the issue?

Accepted Answer

Increase the number of DPUs allocated to the Glue job.. The 'Out of Memory' error in AWS Glue indicates that the job's allocated resources are insufficient for the data volume or processing complexity. Increasing the number of DPUs (Data Processing Units) directly increases the available memory and compute capacity, which is the most straightforward fix for OOM errors in Glue streaming jobs. Option C is correct because it addresses the root cause—resource exhaustion—by scaling the job horizontally.

Answer

Enable compression on the Kinesis stream.

Answer

Change the output format from Parquet to ORC.

Answer

Reduce the streaming batch size in the Glue job configuration.

Question 7

A data engineer is designing a data ingestion pipeline for IoT sensor data. The data arrives as JSON via AWS IoT Core, and must be stored in Amazon S3 in partitioned Parquet format. The pipeline must handle late-arriving data (up to 1 hour) and ensure exactly-once processing. Which combination of services should the engineer use?

Accepted Answer

Amazon Kinesis Data Firehose with data transformation via AWS Lambda, delivering to Amazon S3.. Amazon Kinesis Data Firehose is the correct choice because it can directly ingest streaming data from AWS IoT Core, use a built-in AWS Lambda function to transform JSON to Parquet, and deliver the data to Amazon S3 with automatic partitioning. It also supports buffering and retry logic to handle late-arriving data (up to 1 hour) and provides exactly-once delivery to S3 when configured with the appropriate error handling and idempotent transformations.

Answer

Amazon Kinesis Data Streams with AWS Lambda for transformation and Amazon S3.

Answer

Amazon Simple Queue Service (SQS) with AWS Lambda for transformation and Amazon S3.

Answer

AWS Glue streaming jobs consuming from Amazon Kinesis Data Streams and writing to Amazon S3.

Question 8

A company has a Glue ETL job that reads from an Amazon RDS for MySQL table and writes to Amazon S3. The job runs hourly and processes new records based on a 'last_modified' timestamp column. Recently, the job started missing some records because the timestamp in MySQL is stored with microsecond precision but Glue's job bookmark only tracks second precision. Which solution addresses this issue?

Accepted Answer

Use a job parameter to store the last processed timestamp with millisecond precision and query records greater than that value.. AWS Glue job bookmarks track timestamps with only second precision, so records with microsecond differences within the same second are missed. By using a custom job parameter to store the last processed timestamp with millisecond precision and querying records greater than that value, you bypass Glue's bookmark limitation and capture all new or modified records.

Answer

Increase the job frequency to every 30 minutes.

Answer

Run a full refresh of the table each time instead of incremental.

Answer

Modify the MySQL table to use a DATE data type instead of TIMESTAMP.

Question 9

A data engineer is ingesting CSV files from an Amazon S3 bucket into a Glue Data Catalog table. The files have headers, but some files have extra columns not present in the first file. The engineer wants the Glue crawler to automatically detect the schema. Which crawler configuration option should be used?

Accepted Answer

Configure the crawler to 'Create a single schema for each S3 path' and enable 'Merge tables'.. When CSV files have varying schemas (extra columns), the Glue crawler must be configured to 'Create a single schema for each S3 path' with 'Merge tables' enabled. This configuration instructs the crawler to union the schemas from all files in the S3 path, adding new columns as they appear, rather than creating separate tables for each schema variation.

Answer

Configure the crawler to 'Inherit schema from table' and set the table name.

Answer

Configure the crawler to 'Create a single schema for each S3 path' without enabling 'Merge tables'.

Answer

Configure the crawler to 'Create a single schema for each S3 path' and set 'Each file as a separate table'.

Question 10

A company uses AWS Glue to process CSV files from an S3 bucket. The job fails intermittently with a 'SchemaDetectionError' for files that have inconsistent column counts. What is the most efficient way to handle this?

Accepted Answer

Define a fixed schema in the Glue job using 'apply_mapping' to map columns.. Defining a fixed schema using the `schema` parameter in the DynamicFrame reader forces Glue to apply that schema to all CSV files. With `apply_mapping`, you can map the actual columns present to the fixed schema, handling inconsistent column counts by ignoring extra columns and filling missing columns with nulls. This avoids schema detection errors without extra preprocessing.

Answer

Use the 'mergeSchema' option when reading the DynamicFrame.

Answer

Convert all CSV files to Parquet format using a separate preprocessing job.

Answer

Set the job to 'ignore' schema mismatches in the job parameters.

Question 11

A data pipeline uses Kinesis Data Firehose to deliver streaming data to an S3 bucket. The data volume spikes occasionally, causing the Firehose buffer to fill up and leading to increased delivery latency. The latency must remain under 60 seconds. What should be done to minimize latency?

Accepted Answer

Reduce the buffer interval to 60 seconds.. Reducing the buffer interval to 60 seconds ensures that Firehose delivers data to S3 at most every 60 seconds, directly capping latency even if the buffer size is not full. This aligns with the requirement to keep latency under 60 seconds, as Firehose delivers data when either the buffer interval or buffer size threshold is met first.

Answer

Enable GZIP compression on the Firehose delivery stream.

Answer

Increase the buffer size to 128 MB to accommodate larger batches.

Answer

Switch to Kinesis Data Streams with a Lambda consumer.

Question 12

A company runs a nightly AWS Glue ETL job that reads from a JDBC source (PostgreSQL) and writes to S3 in Parquet format. The job takes over 6 hours, but the SLA requires completion within 4 hours. The source table has 500 million rows and is updated frequently. Which approach will most reliably reduce job duration?

Accepted Answer

Partition the source table by year and use pushdown predicates in the Glue job.. Partitioning the source table by year and using pushdown predicates allows AWS Glue to read only the relevant partitions from PostgreSQL, drastically reducing the data scanned and transferred. This directly addresses the 500 million row volume by minimizing the JDBC read workload, which is the primary bottleneck. Option B, using multiple JDBC connections in parallel via hash partitioning, can improve performance but is less reliable: it significantly increases load on the source database, may hit connection limits, and its effectiveness depends on the JDBC driver's support for hashfield/hashexpression. Moreover, it still reads all rows, whereas pushdown predicates reduce the data volume at the source, making option C more reliable for meeting the 4-hour SLA.

Answer

Enable job bookmark and schedule the job to run more frequently.

Answer

Use multiple JDBC connections in parallel by setting 'hashexpression' and 'hashfield'.

Answer

Increase the number of DPUs for the Glue job to 100.

Question 13

A company ingests IoT sensor data into Kinesis Data Streams. The data is then processed by a Lambda function that aggregates readings and writes to DynamoDB. The Lambda function is experiencing high error rates due to throttling. Which TWO actions would reduce throttling?

Accepted Answer

Increase the batch size in the Lambda event source mapping.. Option B is correct because increasing the Lambda event source mapping batch size lets each Lambda invocation process more Kinesis records per call, reducing the total number of concurrent invocations and thus lowering the request rate against DynamoDB, which is the source of the throttling. Option D is correct because switching DynamoDB to on-demand capacity mode removes the provisioned read/write capacity limits and automatically scales with traffic, eliminating the DynamoDB throttling that is causing the Lambda errors. Option A is not correct because adding Kinesis shards increases parallelism and would drive more concurrent Lambda invocations, worsening DynamoDB throttling rather than reducing it. Option C is not correct because decreasing the batch window makes Lambda invoke more frequently with smaller batches, increasing the request rate to DynamoDB. Option E is not correct because raising reserved concurrency allows more simultaneous Lambda executions, which increases pressure on DynamoDB and does not address the throttling root cause.

Answer

Increase the number of shards in the Kinesis stream.

Answer

Decrease the batch window in the Lambda event source mapping.

Answer

Increase the Lambda reserved concurrency to 1000.

Question 14

A company uses Amazon RDS for MySQL as a source for AWS DMS to replicate data to S3. The replication task is failing with 'OutOfMemory' errors on the DMS instance. The source table has 10 million rows with large BLOB columns. Which THREE changes would most likely resolve the issue?

Accepted Answer

Set the LOB column settings to 'Limited LOB mode' and specify a max LOB size.. Option A is correct because switching the LOB column settings to 'Limited LOB mode' with an explicit Max LOB Size makes AWS DMS pre-allocate a fixed buffer per LOB instead of buffering unpredictable, potentially huge LOB values, which directly reduces memory pressure on the replication instance. Option D is correct because OutOfMemory errors on the DMS instance are fundamentally a resource constraint, so moving to a larger compute-optimized replication instance class provides more RAM and CPU to handle the 10 million rows with large BLOBs. Option E is correct because increasing the number of parallel threads (e.g., via the task's 'ParallelLoadThreads' setting) lets DMS process the table's data in more concurrent streams, improving throughput so the task completes faster and each thread's memory footprint stays bounded rather than one thread accumulating the entire load. Option B is not correct because disabling task logging does not free meaningful memory for LOB processing and removes the diagnostics needed to troubleshoot the failure. Option C is not correct because Full LOB mode buffers entire LOB values in memory, which would worsen the OutOfMemory condition rather than resolve it.

Answer

Disable logging for the DMS task to free memory.

Answer

Enable Full LOB mode to handle LOBs more efficiently.

Question 15

A company runs an e-commerce platform that generates clickstream data from millions of users. The data is ingested into Amazon Kinesis Data Streams with a shard count of 10. The data is then consumed by a Kinesis Data Analytics application that runs SQL queries to aggregate metrics in real time. Recently, the application has been falling behind, and the stream's iterator age metric is increasing. The data volume has doubled over the past month. The application currently uses a single Kinesis Data Analytics application with parallelism of 1. Which action should the data engineer take to improve the processing rate and reduce the iterator age without losing data or causing duplicates?

Accepted Answer

Increase the parallelism of the Kinesis Data Analytics application to match the number of shards.. Kinesis Data Analytics (KDA) processes data from each shard in a stream using one or more parallel operators. With a parallelism of 1, the application uses only a single processing thread, which cannot keep up with the doubled data volume across 10 shards. By increasing parallelism to match the shard count (10), KDA can read from all shards concurrently, distributing the processing load and reducing the iterator age without data loss or duplicates, as KDA manages checkpointing and exactly-once semantics internally.

Answer

Change the Kinesis Data Analytics application to use a Kinesis Data Firehose delivery stream as the source.

Answer

Reduce the retention period of the Kinesis Data Streams to 24 hours.

Answer

Increase the number of shards in the Kinesis Data Streams to 20.

Question 16

A financial services company ingests real-time stock trade data from multiple exchanges into Amazon Kinesis Data Streams. Each trade record is a JSON object containing fields: trade_id, symbol, price, quantity, and timestamp. The data is consumed by an AWS Lambda function that performs data validation and enrichment, then writes the processed records to an Amazon DynamoDB table for low-latency querying. Recently, the Lambda function has been timing out and failing to process all records. The Lambda function is configured with a 5-second timeout and 128 MB memory. The average record size is 2 KB, and the stream receives about 1000 records per second. The Lambda function's concurrency limit is 1000. Which set of actions should the data engineer take to resolve the issue without losing data?

Accepted Answer

Increase the Lambda function timeout to 60 seconds and memory to 1024 MB. Set the batch size to 100 records and enable parallelization factor of 10.. Increasing the Lambda timeout and memory addresses the processing bottleneck, while setting the batch size to 100 and enabling a parallelization factor of 10 allows each shard to process up to 10 concurrent batches, dramatically increasing throughput to handle 1000 records/sec (each shard can process 10 batches of 100 records concurrently, yielding 1000 records/sec per shard if the stream has at least 1 shard). This combination ensures no data loss by keeping up with the ingestion rate without exceeding the Lambda concurrency limit of 1000.

Answer

Increase the number of shards in the Kinesis data stream to 20 and keep the Lambda configuration unchanged.

Answer

Replace the Lambda function with a Kinesis Data Firehose delivery stream that writes directly to DynamoDB using a Lambda transformation.

Answer

Increase the Lambda function timeout to 60 seconds and memory to 1024 MB. Set the batch size to 100 records.

Question 17

A data engineer is troubleshooting an AWS Glue ETL job that reads from Amazon S3 and writes to Amazon Redshift. The job runs successfully but writes duplicate rows into Redshift. The source data is static and does not contain duplicates. Which configuration change is most likely to resolve this issue?

Accepted Answer

Modify the job to use the 'postactions' option with a SQL statement that deletes duplicates before final insert.. The job runs successfully but writes duplicate rows because AWS Glue's Spark-based ETL jobs can retry tasks on failure, and when writing to Redshift using the JDBC connector, the default behavior is to append data without deduplication. Using the 'postactions' option with a SQL DELETE statement that removes duplicates before the final INSERT ensures that only unique rows remain, resolving the duplication without altering the source data.

Answer

Enable the 'upsert' feature in the Redshift connection by setting 'update' to true.

Answer

Use partition pruning on the S3 source to reduce the number of files read.

Answer

Increase the number of DPUs (Data Processing Units) allocated to the job.

Question 18

A company is designing a data ingestion pipeline to load CSV files from an SFTP server into Amazon S3. The files are generated hourly and range from 10 MB to 500 MB. Which AWS service should be used to orchestrate the transfer with minimal operational overhead?

Accepted Answer

AWS Transfer Family. AWS Transfer Family is the correct choice because it provides a fully managed, serverless SFTP endpoint that can directly receive files from an SFTP server and automatically store them in Amazon S3. This eliminates the need to manage any compute infrastructure or write custom code for the transfer, minimizing operational overhead for hourly CSV file ingestion.

Answer

AWS Glue

Answer

Amazon AppFlow

Answer

AWS DataSync

Question 19

Arrange the steps to implement a data lake on Amazon S3 with AWS Lake Formation.

Accepted Answer

Create S3 bucket, then register it in Lake Formation, then set up administrators, then define schema in Glue Catalog, then grant access to users. Start by creating the S3 bucket. Then register it in Lake Formation, set up administrators, define the schema in Glue Catalog, and finally grant access to users.

Answer

Set up administrators, then create S3 bucket, then register it in Lake Formation, then define schema in Glue Catalog, then grant access to users

Answer

Create S3 bucket, then define schema in Glue Catalog, then register it in Lake Formation, then set up administrators, then grant access to users

Answer

Register S3 bucket in Lake Formation, then create S3 bucket, then set up administrators, then define schema in Glue Catalog, then grant access to users

Question 20

Order the steps to query data in Amazon Redshift Spectrum from an external table in Athena.

Accepted Answer

Create external schema in Redshift, then create external table, then grant permissions, then run the query, then verify results.. Start by creating the external schema in Redshift, then the external table, grant permissions, run the query, and verify results.

Answer

Run the query, then verify results, then create external table, then grant permissions, then create external schema.

Answer

Create external table, then create external schema, then grant permissions, then run the query, then verify results.

Answer

Create external schema, then create external table, then run the query, then grant permissions, then verify results.