DEA-C01 Data Ingestion and Transformation Practice Test 21 — 15 Questions

Question 1

A company uses AWS Glue to run ETL jobs that transform data from an Amazon S3 bucket (raw) to another S3 bucket (curated). The jobs run on a schedule and process data incrementally. The data engineer notices that the jobs are taking longer to complete each day, and the job metrics show that the number of DPUs (Data Processing Units) is underutilized. The engineer wants to improve job performance. What should the data engineer do?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Increase the number of DPUs allocated to the Glue job to enable more parallelism.. When DPUs are underutilized, the job is not parallelizing work effectively, so adding more DPUs (workers) increases the number of executors and partitions that can process data concurrently. This directly addresses the underutilization metric and improves throughput for incremental ETL workloads.

Answer

Switch from batch processing to streaming using AWS Glue Streaming.

Answer

Enable job bookmarks to skip already processed data more efficiently.

Answer

Decrease the number of DPUs to reduce resource contention.

Question 2

A logistics company ingests GPS tracking data from thousands of vehicles into Amazon S3 via AWS Direct Connect. Each vehicle sends a message every 5 seconds, resulting in about 200,000 messages per second. Each message is about 200 bytes. The company uses AWS Glue to transform the data into a parquet format and load it into Amazon Redshift for real-time analytics. However, the Glue jobs are failing due to memory issues and the data is not being loaded into Redshift quickly enough. The company needs to reduce the latency of data availability in Redshift. Which action should the data engineer take?

Accepted Answer

Use Amazon Kinesis Data Firehose to ingest the data directly into S3 and then use Redshift Spectrum to query the data without loading.. Using Amazon Kinesis Data Firehose to ingest data directly into S3 and then querying with Redshift Spectrum reduces latency by bypassing the Glue transformation and load steps. Firehose can deliver data to S3 in near real-time, and Redshift Spectrum allows querying the data directly from S3 without loading, thus making data available faster.

Answer

Use Amazon Kinesis Data Analytics to process the data in real-time and write to Redshift directly.

Answer

Increase the size of the Redshift cluster to improve load performance.

Answer

Increase the number of DPUs and allocate more memory to the Glue job.

Question 3

A data engineer is responsible for ingesting daily CSV files from an external partner into an Amazon S3 bucket. The partner uploads files to an AWS Transfer Family (SFTP) endpoint. Once a file is uploaded, an AWS Lambda function triggers an AWS Glue ETL job to transform the data and load it into an Amazon RDS database. Recently, some files have failed to trigger the Glue job because the Lambda function timed out while waiting for the Glue job to complete. The engineer needs to ensure that all files are processed reliably without manual intervention. What should the data engineer do?

Accepted Answer

Modify the Lambda function to send a message to an Amazon SQS queue after uploading, and create a separate Lambda function that reads from the queue and triggers the Glue job asynchronously.. The root cause is that Lambda is synchronously waiting for the Glue job to finish, and Glue jobs can run longer than Lambda's 15-minute maximum timeout. Decoupling the trigger from the job execution by using SQS lets Lambda enqueue a message and return immediately, while a separate consumer Lambda starts the Glue job asynchronously. This removes the timeout dependency and ensures every uploaded file is processed reliably without manual intervention.

Answer

Increase the Lambda function timeout to 15 minutes to accommodate longer Glue jobs.

Answer

Increase the Lambda function's reserved concurrency to allow multiple invocations.

Answer

Configure S3 event notifications to trigger the Glue job directly without Lambda.

Question 4

A company is using Amazon Kinesis Data Firehose to ingest log data from web servers into an Amazon S3 bucket. The data is then queried by Amazon Athena. The company has noticed that the Athena queries are slow and expensive. The data engineer wants to optimize the storage format to improve query performance and reduce costs. Which configuration change should the data engineer make to the Firehose delivery stream?

Accepted Answer

Enable data transformation in Firehose to convert JSON to Parquet format with Snappy compression.. Enable data transformation in Firehose to convert JSON to Parquet format with Snappy compression. Parquet is a columnar storage format that significantly improves query performance in Athena by reducing the amount of data scanned per query. Snappy compression provides efficient compression and decompression, reducing storage costs and improving I/O. Option A is incorrect because increasing buffer interval and size simply creates larger files but does not change the storage format; the data remains in its original format (likely JSON or CSV), which is less efficient for columnar querying. Option B is incorrect because while ORC is also a columnar format, Parquet is more commonly used with Athena and offers better integration; additionally, GZIP compression is not as efficient as Snappy for Parquet files. Option C is incorrect because enabling S3 server access logs would track requests to the S3 bucket but does not optimize the data format or improve query performance; it adds additional cost and storage overhead.

Answer

Increase the buffer interval to 600 seconds and buffer size to 128 MB to create larger files.

Answer

Change the output format to ORC and enable GZIP compression.

Answer

Enable S3 server access logs to track query patterns.

Question 5

A data engineer is designing a data ingestion pipeline for a social media company. The pipeline ingests user posts from a REST API into Amazon S3. The API returns JSON data with an array of posts. The engineer needs to transform the data into individual JSON objects per post and store them in S3 with a partition structure of year/month/day/hour. The data should be available in S3 within 15 minutes of ingestion. The engineer decides to use AWS Lambda for transformation. Which combination of services should the engineer use to meet these requirements with minimal operational overhead?

Accepted Answer

Use Amazon CloudWatch Events to trigger an AWS Lambda function every 15 minutes. The Lambda function calls the API, transforms the data, and writes individual JSON objects to S3 with the required partition structure.. Using Amazon CloudWatch Events to trigger an AWS Lambda function every 15 minutes (B) is correct because it provides a serverless, low-overhead solution. The Lambda function can call the API, transform the JSON array into individual objects, and write them to S3 with the required partition structure. This meets the 15-minute availability requirement with minimal operational overhead.

Answer

Use AWS Step Functions to orchestrate an API call and data transformation with Lambda, running every 15 minutes.

Answer

Use AWS Glue ETL jobs scheduled with AWS Glue triggers to run every 15 minutes.

Answer

Use Amazon Kinesis Data Firehose with a Lambda function for transformation. Configure Firehose to pull from the API every 15 minutes.

Question 6

A company is using AWS Glue to catalog data in Amazon S3. The data is in CSV format with varying schemas. The Data Engineering team wants to ensure the Glue Data Catalog is updated automatically when new partitions are added to S3. Which TWO actions should be taken? (Choose two.)

Accepted Answer

Enable partition indexing on the Glue Data Catalog.. Option A is correct because enabling partition indexing on the Glue Data Catalog (via the Glue console, CLI, or API) builds and maintains partition indexes that let the catalog serve partition metadata efficiently and keep partition information discoverable as new partitions are added, which supports automatic catalog updates for partitioned S3 data. Option C is correct because a scheduled AWS Glue crawler periodically scans the S3 location, detects new partitions and schema changes, and updates the Glue Data Catalog automatically, which is the standard mechanism for keeping the catalog current with varying CSV schemas. Option B is not appropriate here because an S3 event notification triggering a Lambda function requires custom code to parse partition paths and call Glue APIs, which is more complex and not the recommended managed approach for cataloging varying schemas. Option D is not correct because MSCK REPAIR TABLE is an Athena DDL command that only adds partitions to an existing table and does not handle schema evolution or automatic catalog maintenance. Option E is not correct because Glue ETL jobs write data and can update the catalog for their output, but they do not automatically detect and catalog new partitions added externally to S3.

Answer

Set up an S3 event notification to trigger a Lambda function that updates the Glue Data Catalog.

Answer

Run Amazon Athena queries with MSCK REPAIR TABLE to add partitions.

Answer

Use AWS Glue ETL jobs to write data and update the catalog simultaneously.

Question 7

A financial services company is ingesting real-time stock trade data into Amazon Kinesis Data Streams. The data is then processed by a Kinesis Data Analytics application for fraud detection. The company must ensure that the data is processed in the correct order for each stock symbol. Which TWO configuration steps should be taken? (Choose two.)

Accepted Answer

Use the stock symbol as the partition key when putting records into the stream.. Option C is correct because Kinesis Data Streams preserves ordering only within a single shard, and records sharing the same partition key are routed to the same shard; using the stock symbol as the partition key guarantees all trades for a given symbol land in one shard and are therefore processed in order. Option E is correct because the Kinesis Client Library (KCL) processes each shard's records sequentially using a shard iterator, so configuring it to consume records in arrival order per shard maintains the ordering that the partition key established. Option A is wrong because a random partition key scatters records for the same stock symbol across multiple shards, destroying per-symbol ordering. Option B is wrong because adding shards increases parallelism and throughput but does not by itself guarantee ordering, and more shards can actually spread a symbol's records further apart. Option D is wrong because replacing Kinesis Data Analytics with Lambda and DynamoDB Streams changes the architecture entirely and does not address ordering in Kinesis Data Streams.

Answer

Use a random partition key to distribute the load evenly.

Answer

Increase the number of shards to reduce latency.

Answer

Use AWS Lambda with DynamoDB Streams instead of Kinesis Data Analytics.

Question 8

A company is streaming IoT sensor data from thousands of devices into Amazon Kinesis Data Firehose. The data is then delivered to Amazon S3 for long-term storage. Occasionally, some records fail to be delivered to S3. The company must capture and analyze these failed records. Which TWO actions should be taken? (Choose two.)

Accepted Answer

Configure an AWS Lambda function as a pre-processing step to catch and log failed records.. Option A is correct because Kinesis Data Firehose supports AWS Lambda as a data transformation (pre-processing) function, and the Lambda function can inspect each record and log or route records that fail validation or processing, giving the company a way to capture problematic records before delivery. Option D is correct because Firehose publishes delivery and processing errors to Amazon CloudWatch Logs when error logging is enabled, which is the native mechanism for capturing and analyzing records that fail to be delivered to S3. Option B is not appropriate because Kinesis Data Analytics analyzes streaming data in real time and is not the Firehose error-capture mechanism for failed S3 deliveries. Option C is not correct because Firehose does not natively send failed records to a Kinesis Data Stream for reprocessing; that would require custom handling. Option E is not correct because S3 event notifications only fire for objects successfully written to S3, so they cannot detect records that failed to be delivered.

Answer

Use Amazon Kinesis Data Analytics to analyze the failed records in real time.

Answer

Send failed records to an Amazon Kinesis Data Stream for reprocessing.

Answer

Set up an S3 event notification to trigger a Lambda function to reprocess failed records.

Question 9

A company is migrating on-premises Apache Kafka clusters to Amazon MSK. The migration must be seamless with no data loss. The team is using MirrorMaker 2 to replicate data from on-premises Kafka to MSK. Which THREE configurations are necessary to ensure exactly-once semantics and minimal downtime? (Choose three.)

Accepted Answer

Set offsets.topic.replication.factor to 3 for the consumer offsets topic.. Option B is correct because the consumer offsets topic (__consumer_offsets) must have offsets.topic.replication.factor set to 3 on the MSK cluster so that committed consumer group offsets are durably replicated across brokers, which is required for a lossless migration and for MirrorMaker 2 to correctly translate and preserve consumer positions. Option C is correct because replication.factor=3 on the MSK cluster ensures every replicated topic partition has three replicas, providing the fault tolerance needed so that broker failures during the migration do not cause data loss. Option E is correct because MirrorMaker 2 must be explicitly configured to use exactly-once semantics (via its dedicated EOS/transactional producer settings, e.g., enabling exactly.once in the MM2 configuration) so that records are not duplicated or lost during replication. Option A is not required for exactly-once semantics or minimal downtime; disabling auto topic creation is an operational hygiene setting and does not by itself guarantee EOS or lossless migration. Option D is also not required for this goal; TLS encryption secures data in transit between on-premises and MSK but does not affect exactly-once semantics or downtime.

Answer

Set auto.create.topics.enable to false to prevent automatic topic creation.

Answer

Enable TLS encryption between on-premises and MSK.

Question 10

A data engineering team is responsible for ingesting streaming data from a fleet of IoT devices into Amazon S3 using Kinesis Data Firehose. The data volume spikes unpredictably, and the team has configured Kinesis Data Firehose with a buffer size of 5 MB and buffer interval of 60 seconds. During spikes, the team notices that the delivery to S3 is delayed, and some records are lost due to exceeding the service limits. The team needs to ensure no data loss and reduce delivery latency. What should the team do?

Accepted Answer

Use Amazon Kinesis Data Streams as the data source for Firehose to decouple ingestion and delivery.. Using Kinesis Data Streams as the data source for Firehose decouples ingestion and delivery. The stream acts as a durable buffer that can absorb unpredictable spikes, preventing data loss due to Firehose service limits. Firehose can then be configured with smaller buffer size/interval to reduce delivery latency, as the stream retains data until successful delivery. Option B increases buffer size and interval, which would increase latency, contradicting the requirement. Option A adds complexity and does not directly address buffering. Option D is unrelated to Firehose buffering.

Answer

Implement an AWS Lambda function to pre-process the data and send it to Firehose in a throttled manner.

Answer

Increase the buffer size to 10 MB and buffer interval to 120 seconds to allow more data accumulation before delivery.

Answer

Enable S3 Transfer Acceleration on the destination bucket.

Question 11

A company is using AWS DMS to migrate a 5 TB SQL Server database to Amazon Aurora PostgreSQL. The migration is using full load plus CDC. After the full load completes, the ongoing replication task is failing with errors related to large transactions on the source. The team needs to ensure that CDC continues without falling behind. What should the team do?

Accepted Answer

Increase the DMS replication instance size to provide more memory and CPU.. The errors related to large transactions on the source during CDC are typically caused by the DMS replication instance running out of memory or CPU when processing large transaction logs. Increasing the replication instance size provides more memory and CPU, allowing DMS to buffer and process large transactions without failing. This directly addresses the root cause of the CDC task falling behind.

Answer

Use Amazon Kinesis Data Streams as an intermediate target for CDC.

Answer

Modify the DMS task settings to increase MaxFileSize and decrease the CommitRate.

Answer

Disable foreign key constraints on the target Aurora database.

Question 12

A financial services company is ingesting trade data from multiple exchanges via Amazon Kinesis Data Streams. Each shard receives data from multiple exchanges, and a consumer application (using KCL) processes the data. The company needs to ensure that trades from the same exchange are processed in order. However, the current implementation distributes records to shards using a random partition key, causing trades from the same exchange to be spread across shards and processed out of order. The team must enforce ordering per exchange without significantly reducing throughput. What should the team do?

Accepted Answer

Use the exchange ID as the partition key when putting records into the stream.. Using the exchange ID as the partition key ensures all trades from the same exchange go to the same shard, preserving order. Option A is wrong because increasing shard count would further spread data and break ordering. Option B is wrong because using a single shard would preserve order but reduce throughput due to shard limits. Option D is wrong because implementing a custom sequencer is complex and unnecessary.

Answer

Implement a custom sequence number in the application to reorder after processing.

Answer

Use a single shard for all data to guarantee order.

Answer

Increase the number of shards to 10 per exchange.

Question 13

A data engineer is building a nightly ETL job in AWS Glue that reads from an Amazon DynamoDB table. The table is large (several TB) and the job must read the entire table efficiently without consuming excessive read capacity. Which approach should the data engineer use to read the DynamoDB table?

Accepted Answer

Use AWS Glue's DynamoDB connector with the `dynamodb.input.tableName` parameter and set `dynamodb.throughput.read.percent` to 1.0 to maximize read capacity.. The AWS Glue DynamoDB connector allows reading a DynamoDB table by specifying the table name and controlling read throughput with `dynamodb.throughput.read.percent`. Setting this to 1.0 uses the maximum available read capacity, enabling efficient parallel reads for large tables. Other options either use invalid parameters or incorrect methods that would not achieve the required performance.

Answer

Use AWS Glue's DynamoDB connector with the `dynamodb.input.tableName` parameter and set `dynamodb.input.read` to 'parallel' to enable parallel scans.

Answer

Use the DynamoDB connector in AWS Glue with the `dynamodb.scan` option set to true.

Answer

Use AWS Glue's DynamoDB connector with `dynamodb.input.tableName` and enable `dynamodb.input.throughput` to set read throughput.

Question 14

A data engineer needs to transform data in an AWS Glue ETL job. The source is a large dataset in Amazon S3, and the transformation must add a column derived from an existing timestamp column and filter out records with null values. The engineer wants to use the most efficient method for this transformation within AWS Glue. Which approach should the engineer use?

Accepted Answer

Convert the DynamicFrame to a Spark DataFrame, use Spark SQL functions for derivation and filtering, then convert back.. Converting a DynamicFrame to a Spark DataFrame enables the use of Spark SQL functions, which are highly optimized for large-scale data processing. This method efficiently handles column derivation and filtering by leveraging the Catalyst optimizer and predicate pushdown, resulting in better performance compared to row-by-row operations or visual tools that may not optimize as effectively.

Answer

Use AWS Glue Studio to visually create the transformation with a Custom Transform node.

Answer

Use the AWS Glue DynamicFrame class and apply the Map and Filter transformations.

Answer

Use AWS Glue's built-in ApplyMapping transform to rename columns and then filter nulls with a SQL query in Athena.

Question 15

A data engineer is using AWS Glue Studio to create a visual ETL job that reads from an Amazon S3 bucket containing JSON files, applies transformations, and writes to Amazon Redshift. The job must handle schema changes in the JSON files, such as new fields being added over time. Which two actions should the data engineer take to ensure the job handles schema evolution without failing? (Choose two.)

Accepted Answer

Use a DynamicFrame with the `mergeSchema` option set to true when reading the JSON data.. To handle schema evolution when reading JSON files, the engineer should use DynamicFrames with `mergeSchema` set to true to combine schemas from multiple files, and schedule an AWS Glue crawler to update the Data Catalog table with new fields. These actions ensure the job can read data with new fields without failing, while other options either do not address schema evolution or are incorrect for the read path.

Answer

Enable the 'Update the Data Catalog schema' option in the AWS Glue job bookmark settings.

Answer

Use the `resolveChoice` transform with the 'Match Catalog Schema' option to align the data with the existing schema.

Answer

Set the job's 'Schema change policy' to 'Update in the Data Catalog' in the job's output settings.