DEA-C01 Data Ingestion and Transformation — 40 Questions

Question 1

A company is using AWS Glue to run ETL jobs that transform data from Amazon S3 to Amazon Redshift. The jobs are failing intermittently with 'Out of Memory' errors. The team wants to resolve this issue without increasing costs significantly. Which TWO actions should the team take?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Increase the Spark memory overhead parameter in the Glue job configuration.. Options A and E are both correct. Option A increases the Spark memory overhead parameter (`spark.executor.memoryOverhead`), which allocates additional off-heap memory for Spark operations. This is a cost-effective tuning approach that can prevent 'Out of Memory' errors without adding more workers or changing job types. Option E changes the worker type from 'G.1x' to 'G.2x', doubling the memory per worker. While this increases per-worker cost, it avoids adding more workers (which would increase cost significantly) and is a targeted fix for memory-intensive workloads. Both options address memory pressure without the high cost of increasing the number of workers (Option C) or the performance loss of switching to Python shell (Option D).

Answer

Use DynamicFrame instead of Spark DataFrame for transformations.

Answer

Increase the number of workers to maximum allowed.

Answer

Switch from a Spark job to a Python shell job.

Question 2

A data engineer is designing a real-time analytics pipeline for clickstream data. The source is Amazon Kinesis Data Streams, and the data must be stored in Amazon S3 in partitioned Parquet format with near-real-time latency. The engineer must also handle late-arriving data (up to 1 hour). Which combination of services meets these requirements?

Accepted Answer

Use Kinesis Data Firehose with a Lambda transformation to write to S3, and a separate Lambda consumer to reprocess late data from the stream.. AWS Glue streaming ETL, while capable of handling late data via Spark Structured Streaming's watermarking, is not ideal for near-real-time Parquet partitioning to S3 at high throughput; it is more suited for complex transformations with micro-batches. Option B is incorrect: Kinesis Data Analytics tumbling windows process data in memory and output to destinations like S3, but they lack built-in late data handling and Parquet conversion. Option C is correct: Kinesis Data Firehose delivers streaming data to S3 with near-real-time latency, and a Lambda transformation converts it to Parquet. To handle late-arriving data (up to 1 hour), a separate Lambda consumer reads from the Kinesis Data Streams shard iterator to reprocess records that arrive after the Firehose delivery window, ensuring no data loss. Option D is incorrect: S3 Batch Operations are designed for batch processing of existing objects, not for near-real-time handling of late-arriving streaming data.

Answer

Use AWS Glue streaming ETL to write to S3 in real-time.

Answer

Use Kinesis Data Analytics with a tumbling window to write to S3.

Answer

Use Kinesis Data Firehose to deliver to S3, and use S3 Batch Operations to process late data.

Question 3

A company is ingesting CSV files into Amazon S3. Each file contains a header row. The pipeline uses AWS Glue to crawl the S3 bucket and create a table in the AWS Glue Data Catalog. However, the crawler is including the header as data. What is the most likely cause?

Accepted Answer

The crawler is configured to use a custom classifier that does not skip headers. The AWS Glue crawler uses built-in classifiers to infer schema of CSV files. The built-in CSV classifier automatically detects and skips header rows. However, if a custom classifier is configured and it does not have the correct setting to skip headers (e.g., `HasHeader` set to `false` or not defined), the crawler will treat the header row as data. Since the question describes the header being included as data, the most likely cause is that a custom classifier is in use that does not properly skip headers.

Answer

The S3 bucket has versioning enabled

Answer

The crawler is running in 'crawl all folders' mode

Answer

The CSV files are not compressed

Question 4

Which THREE factors should be considered when choosing between Amazon Kinesis Data Streams and Amazon Kinesis Data Firehose for a streaming ingestion architecture? (Choose 3.)

Accepted Answer

Ability to deliver data directly to Amazon Redshift [wrong]. Amazon Kinesis Data Firehose can deliver data directly to Amazon Redshift via an intermediate Amazon S3 bucket, making option A a factor to consider. Option C is a factor because Kinesis Data Firehose automatically scales its throughput, while Kinesis Data Streams requires manual shard management. Option E is a factor because Kinesis Data Streams supports custom data processing using the Kinesis Client Library (KCL) or AWS Lambda, whereas Firehose only supports Lambda for transformation.

Answer

Automatic scaling of ingestion throughput [CORRECT]

Answer

Lower cost per GB ingested [wrong]

Question 5

A data engineer is building a batch ETL pipeline using AWS Glue. The source data is in Amazon RDS for MySQL. The pipeline must run daily and process only new and modified records since the last run. The engineer needs to implement change data capture (CDC) efficiently. Which THREE steps should the engineer take? (Choose THREE.)

Accepted Answer

Set a job bookmark in Glue to track processed records.. AWS Glue job bookmarks can track processed records by using a timestamp column in the source table, enabling incremental processing (Option C). Enabling binary logging (binlog) on RDS MySQL is a necessary step for any change data capture (CDC) implementation, as it ensures all changes are recorded and can be used by AWS DMS or other services to capture changes in real time, which can then be ingested into AWS Glue via S3 or Kinesis. Although Glue does not directly read binlog via JDBC, enabling binlog is a best practice and a prerequisite for CDC from RDS MySQL. Additionally, using a full table scan with a WHERE clause on a timestamp column (Option E) is an efficient way to process only new and modified records when combined with job bookmarks. Option A is incorrect because Glue JDBC connections cannot directly read MySQL binary logs. Option B is unnecessary and adds complexity without direct benefit to the task.

Answer

Configure the Glue job to use a JDBC connection with a SQL query that reads from the binary log.

Answer

Ingest the RDS data into Kinesis Data Streams and then use Glue.

Question 6

Refer to the exhibit. A data engineer runs the describe-stream command on a Kinesis data stream. The stream has two shards. The engineer wants to increase the shard count to 4 using the UpdateShardCount API. What will be the resulting shard distribution?

Accepted Answer

Four shards with hash ranges: 0-5.67e37, 5.67e37-1.13e38, 1.13e38-1.70e38, 1.70e38-2.27e38.. When the UpdateShardCount API doubles the shard count from 2 to 4, Kinesis splits the existing hash range (0 to 2^128-1, approximately 3.40e38) into four equal contiguous sub-ranges. The precise boundaries are: 0, 5.67e37, 1.13e38, 1.70e38, and 2.27e38. Option B lists these exact boundaries, making it correct. Option D lists the same boundaries, but since it is a duplicate of the correct answer, it is considered a distractor and is not a valid separate correct choice. Options A and C contain incorrect values or descriptions.

Answer

Four shards with hash ranges: 0-5.68e37, 5.68e37-1.13e38, 1.13e38-1.70e38, 1.70e38-2.27e38.

Answer

Two new shards are created with overlapping hash ranges, and the original two shards remain as parents.

Answer

Four shards with hash ranges: 0-5.67e37, 5.67e37-1.13e38, 1.13e38-1.70e38, 1.70e38-2.27e38.

Question 7

A data engineer uses AWS DMS to migrate a 2 TB PostgreSQL database to Amazon Aurora PostgreSQL. The migration task is set to full load + CDC. After the full load completes, the CDC phase starts but shows a high latency of 5 minutes. The source database has a low write load. What should the engineer do to reduce the CDC latency?

Accepted Answer

Increase the size of the DMS replication instance.. To reduce CDC latency, increasing the size of the DMS replication instance (Option C) is the most effective action because it provides more CPU and memory to process the change data capture stream. Option A (decreasing batch size) could actually worsen latency by increasing the number of batches. Option B (disabling validation) might reduce some overhead but does not address potential resource constraints. Option D (enabling logging) adds overhead and does not reduce latency.

Answer

Decrease the batch size for the CDC task.

Answer

Disable the validation feature on the DMS task.

Answer

Enable logging for the DMS task to capture additional details.

Question 8

A media company ingests large video files (up to 100 GB each) from content creators via Amazon S3 multipart uploads. After upload, the company needs to transcode the videos into multiple formats using AWS Elemental MediaConvert. The current pipeline uses S3 event notifications to trigger an AWS Lambda function that starts a MediaConvert job. However, for very large files, the Lambda function times out (15-minute limit) before the upload completes because the event is sent when the multipart upload is initiated, not when it completes. How should the engineer fix this issue?

Accepted Answer

Change the S3 event notification to listen for s3:ObjectCreated:Put events instead of s3:ObjectCreated:*, which fires only after the object is fully written.. Use S3 event notifications for s3:ObjectCreated:Put, which triggers only after the object is completely written. Then Lambda can start MediaConvert. Option A (increase timeout) does not solve the root cause. Option B (SQS) does not address the event timing. Option C (Step Functions) adds complexity.

Answer

Increase the Lambda timeout to 30 minutes.

Answer

Use Amazon SQS to queue the event and have a Lambda function process it later.

Answer

Use AWS Step Functions to poll the S3 bucket for object existence and then start MediaConvert.

Question 9

A data engineer needs to transform JSON data from an S3 bucket into Parquet format and load it into Amazon Redshift. The transformation must be performed incrementally as new data arrives. Which AWS service is BEST suited for this task?

Accepted Answer

Use AWS Glue to create an ETL job that runs on a schedule or trigger.. AWS Glue provides a serverless ETL service that can run jobs triggered by S3 events to transform data incrementally and load into Redshift. Option A (AWS Lambda) can be used for simple transformations but may hit time limits for complex transformations. Option B (Amazon EMR) is more suited for large-scale big data processing but requires cluster management. Option D (Amazon Kinesis Data Firehose) is for streaming data, not for batch transformation of existing S3 objects.

Answer

Use AWS Lambda to transform the data on the fly and write to Redshift.

Answer

Use Amazon EMR with Apache Spark to transform the data and load it into Redshift.

Answer

Use Amazon Kinesis Data Firehose to transform and load data into Redshift in real time.

Question 10

A data engineer is ingesting data from a third-party API into Amazon S3 using AWS Lambda. The API returns a JSON payload of up to 10 MB per request. The Lambda function runs every minute. Occasionally, the function times out after 15 seconds. What is the most likely cause?

Accepted Answer

The Lambda function's timeout setting is too short.. The Lambda function times out after 15 seconds, which matches its configured timeout. Increasing the timeout value (up to 15 minutes) would resolve the occasional timeout. Option A is incorrect: being in a VPC without a NAT gateway would cause consistent failures due to lack of internet access, not occasional timeouts. Option C is incorrect: while insufficient memory can slow processing, the immediate symptom of timing out exactly at 15 seconds points to the timeout setting, not memory. Option D is incorrect: the API response size (up to 10 MB) does not affect Lambda's invocation payload limit, which applies to the event that triggers the function (a small CloudWatch Event), not the HTTP response received by the function.

Answer

The Lambda function is in a VPC without a NAT gateway, causing network timeouts.

Answer

The Lambda function's memory is too low, causing slow processing.

Answer

The API response size exceeds the Lambda invocation payload limit.

Question 11

A data engineer needs to transform JSON data from an S3 bucket into Parquet format for efficient querying with Amazon Athena. The transformation must be serverless and event-driven. Which approach meets these requirements?

Accepted Answer

Use S3 event notifications to invoke a Lambda function that runs PySpark to convert the data.. S3 event notifications can trigger a Lambda function that launches an AWS Glue ETL job. Glue ETL jobs run PySpark code natively to convert JSON to Parquet, satisfying the serverless and event-driven requirements. Athena only queries existing data, Glue crawlers do not perform transformations, and EMR clusters are not serverless.

Answer

Use AWS Glue with a scheduled crawler to convert the data.

Answer

Use Amazon Athena to convert JSON to Parquet on the fly.

Answer

Use Amazon EMR with a long-running cluster to process S3 data.

Question 12

A company uses Amazon Kinesis Data Firehose to deliver data to an S3 bucket. The data contains personally identifiable information (PII) that must be redacted before storage. Which THREE actions can achieve this requirement? (Choose THREE.)

Accepted Answer

Use AWS Glue ETL to read from the S3 bucket and write redacted data to another S3 bucket.. AWS Glue ETL can read data from the S3 bucket, apply transformations (including redacting PII), and write to another S3 bucket. Option E is correct because an AWS Lambda function can be used as a transformation in the Firehose delivery stream to redact PII before the data reaches S3. Option C is incorrect because Amazon Macie is a discovery and classification service; while it can detect PII, it cannot automatically redact it without additional automation like Lambda. Options B (Athena) and D (DMS) are not suitable for redacting PII before storage.

Answer

Use Amazon Athena to query the data and redact PII on the fly.

Answer

Use Amazon Macie to discover and automatically redact PII before storage.

Answer

Use AWS Database Migration Service (AWS DMS) to replicate data and apply transformations.

Question 13

A data engineer reviews the Glue job configuration. The job fails when processing large datasets. The error message indicates out-of-memory in the executors. Which change to the job configuration will most directly address this issue?

Accepted Answer

Change the worker type from Standard to G.2X.. The job fails due to out-of-memory errors in the executors. The current configuration uses 5 Standard workers, each with 16 GB of memory. Changing the worker type to G.2X provides 32 GB per worker, doubling the memory per executor and directly addressing the OOM issue. Increasing the number of workers (option C) adds more executors but does not increase memory per executor, which may not resolve the issue if each executor still runs out of memory. Options B and D do not affect memory.

Answer

Increase the timeout from 30 to 60 minutes.

Answer

Increase the number of workers from 5 to 10.

Answer

Set MaxRetries to 3.

Question 14

A company uses AWS Glue to catalog data in Amazon S3. The data arrives in Parquet format, but the crawler fails to update the schema when new columns are added. What is the most likely cause?

Accepted Answer

The crawler configuration has 'Update the table definition' set to 'Ignore the change'.. The most likely cause of the crawler failing to detect schema changes is that its configuration option 'Update the table definition' is set to 'Ignore the change'. This prevents the crawler from adding new columns. Option A is incorrect because the Glue Data Catalog itself supports schema changes; the issue is with the crawler's configuration. Option B is incorrect because Parquet files do support schema evolution. Option D is incorrect because S3 bucket versioning does not affect how the crawler updates the schema.

Answer

The Glue Data Catalog is not configured to accept schema changes.

Answer

Parquet files do not support schema evolution.

Answer

The S3 bucket has versioning disabled.

Question 15

A data engineer is designing a data ingestion pipeline to load data from an on-premises Oracle database into Amazon Redshift. The pipeline must capture changes (inserts, updates, deletes) with low latency and minimal impact on the source database. Which combination of AWS services should the engineer use?

Accepted Answer

AWS Database Migration Service (DMS) to Amazon S3, then COPY into Redshift. AWS DMS with ongoing replication (change data capture) can capture changes from Oracle with minimal impact and replicate to S3, then COPY into Redshift. Option B is wrong because Kinesis Data Streams requires a custom producer to capture Oracle changes and does not provide native CDC from Oracle databases. Option C is wrong because AWS Glue with JDBC is batch-oriented and does not support real-time change data capture natively. Option D is wrong because AWS Lambda can process events but is not designed for continuous, low-latency CDC from a database and would require polling or triggers.

Answer

Amazon Kinesis Data Streams with a custom producer on Oracle

Answer

AWS Glue with JDBC connection to Oracle, writing to Redshift

Answer

AWS Lambda reading from Oracle logs and writing to Redshift

Question 16

A company wants to ingest real-time streaming data from thousands of IoT devices into AWS for immediate processing. Which service is designed for ingesting large volumes of streaming data with low latency?

Accepted Answer

Amazon Kinesis Data Streams. Amazon Kinesis Data Streams is purpose-built for real-time streaming data ingestion at scale. Option A (S3) is object storage, not designed for streaming ingestion. Option B (DMS) is for database migration, not real-time data streaming. Option C (SQS) is a message queue service that is pull-based and not optimized for high-throughput streaming with low latency. Only option D (Kinesis Data Streams) meets all requirements for ingesting large volumes of streaming data with low latency.

Answer

Amazon Simple Storage Service (S3)

Answer

AWS Database Migration Service (DMS)

Answer

Amazon Simple Queue Service (SQS)

Question 17

A data engineering team needs to ingest CSV files from an S3 bucket into a Redshift cluster on a daily basis. The files are large (up to 100 GB each). Which approach is MOST cost-effective and efficient?

Accepted Answer

Use the Redshift COPY command with an IAM role to load data directly from S3.. Using the Redshift COPY command with an IAM role is the most cost-effective and efficient approach for loading large CSV files from S3 into Redshift on a daily basis. The COPY command is optimized for bulk data loading and can handle files up to 100 GB efficiently. Option A (Kinesis Data Firehose) is designed for streaming data, not batch loading of large files. Option B (AWS Data Pipeline) adds unnecessary cost and complexity for a simple scheduled copy job. Option C (Lambda function) has execution time limits (15 minutes) and memory constraints, making it unsuitable for processing 100 GB files atomically. Therefore, the COPY command is the best choice.

Answer

Use Amazon Kinesis Data Firehose to load data into Redshift.

Answer

Use AWS Data Pipeline to copy data from S3 to Redshift.

Answer

Use a Lambda function to read each file and insert rows individually.

Question 18

A company uses AWS Database Migration Service (DMS) to migrate an on-premises PostgreSQL database to Amazon RDS for PostgreSQL. The migration is ongoing and uses change data capture (CDC). The engineer notices that the target database is falling behind the source by several hours. What is the MOST likely cause?

Accepted Answer

A large full load is being transferred, which delays the start of CDC.. A large full load can delay the start of CDC, causing the target to fall behind by hours. Option A is incorrect because disabling parallel apply on the target table can reduce replication speed, but it is not the most likely cause of a multi-hour lag. Option C is incorrect because while an under-provisioned replication instance can cause performance issues, it is less likely than the full load delaying the start of CDC. Option D is incorrect because if change data capture were actually disabled, the replication would stop completely, not just fall behind.

Answer

The target table has disabled parallel apply.

Answer

The replication instance is under-provisioned and needs to be larger.

Answer

Change data capture has been disabled on the source database.

Question 19

A company runs a nightly batch ETL job using AWS Glue to transform data from Amazon RDS for MySQL to Amazon S3. The job reads 100 tables and writes Parquet files partitioned by date. Recently, the job started failing with 'ThrottlingException' from the RDS database. The data volume has increased, and the Glue job is reading large tables without any filtering. The job uses a single Glue job with multiple Spark executors. The engineer needs to reduce the load on the RDS database while maintaining the same processing time. What should the engineer do?

Accepted Answer

Modify the Glue job to use a JDBC connection with a WHERE clause to read only the latest partition.. Using a JDBC connection with a WHERE clause to filter data by date reduces the amount of data read from RDS, reducing throttling. Option A is wrong because AWS DMS is a migration tool, not designed for regular ETL workloads, and continuous replication would not reduce the load on RDS during the Glue job. Option B is wrong because reading all data into S3 first still requires reading all data from RDS, which does not reduce the load on RDS. Option C is wrong because increasing DPUs would increase parallelism and potentially increase the load on RDS, worsening the throttling.

Answer

Use AWS DMS to continuously replicate data to S3 and then run Glue on the S3 data.

Answer

Change the job to read all data from RDS into a staging table in S3 first, then transform.

Answer

Increase the number of DPUs to process data faster.

Question 20

A data engineer needs to ingest streaming data from an IoT fleet into Amazon S3 for near-real-time analytics. The data volume is approximately 5 GB per hour, and each event is less than 1 KB. Which AWS service should be used as the ingestion endpoint?

Accepted Answer

AWS IoT Core. AWS IoT Core is purpose-built for ingesting data from IoT devices, supporting MQTT, HTTP, and WebSocket protocols. It can handle millions of devices and high-throughput, small-message payloads (each event <1 KB) and integrates directly with Amazon S3 via IoT Core rules, making it the ideal ingestion endpoint for near-real-time analytics on streaming IoT data.

Answer

AWS DataSync

Answer

Amazon AppFlow

Answer

Amazon Kinesis Data Streams