DEA-C01 Data Ingestion and Transformation Practice Test 19 — 15 Questions

Question 1

A company uses S3 as a data lake. They want to ingest on-premises relational database data daily with full-load snapshots. The data volume is 500 GB per day. The database is accessible over the internet. Which service should they use for this ingestion?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

AWS Database Migration Service (DMS). AWS Database Migration Service (DMS) is designed to migrate databases to AWS, but it can also be used for ongoing replication and data ingestion. It supports full-load snapshots and change data capture (CDC) from various sources, including on-premises relational databases over the internet. For daily full-load snapshots of 500 GB, DMS is the appropriate service.

Answer

Kinesis Data Firehose

Answer

AWS Glue ETL job reading from JDBC

Answer

AWS Transfer Family

Question 2

A company ingests streaming data from multiple sources into a single Kinesis Data Streams stream. Each source produces records with a different schema. The data must be routed to different S3 prefixes based on the source. Which approach minimizes transformation overhead?

Accepted Answer

Use a single Kinesis Data Firehose with a Lambda transformation that reads schema metadata from DynamoDB to determine the S3 prefix.. Using a single Kinesis Data Firehose delivery stream with a Lambda transformation that reads schema metadata from DynamoDB to determine the S3 prefix minimizes transformation overhead by routing data in real-time without additional storage or processing steps. Option B is wrong because ingesting all data into S3 first and then using AWS Glue ETL jobs to partition and route data to different prefixes adds latency and transformation overhead due to the need for batch processing. Option C is wrong because using separate Kinesis Data Streams for each source with separate Firehose delivery streams increases cost and complexity, and does not minimize transformation overhead as it requires managing multiple streams. Option D is wrong because using Kinesis Data Analytics to run SQL queries for routing adds unnecessary complexity and overhead, as Kinesis Data Analytics is better suited for real-time analytics, not simple routing.

Answer

Ingest all data into S3 and use AWS Glue ETL jobs to partition and route data to different prefixes.

Answer

Use separate Kinesis Data Streams for each source and configure separate Firehose delivery streams.

Answer

Use Kinesis Data Analytics to run SQL queries that route data to different Firehose streams.

Question 3

A company needs to ingest data from an external FTP server into AWS S3. The FTP server is not accessible from the internet. Which AWS service should be used to securely transfer the data?

Accepted Answer

AWS Transfer Family with SFTP endpoint in a VPC. AWS Transfer Family supports SFTP and can be configured with a VPC endpoint to securely transfer data from an FTP server in a private network to S3. Option A is wrong because Kinesis Data Firehose is designed for streaming data ingestion, not for FTP transfers. Option C is wrong because AWS DataSync requires network connectivity to the source, and the FTP server is not internet-accessible. Option D is wrong because AWS Snowball Edge is intended for large-scale offline data transfers, not regular FTP-based transfers.

Answer

Kinesis Data Firehose

Answer

AWS DataSync

Answer

AWS Snowball Edge

Question 4

A company uses Kinesis Data Firehose with a Lambda function for data transformation. The transformation is failing intermittently due to Lambda timeouts. The maximum record size is 1 MB. What is the most cost-effective way to reduce failures without losing data?

Accepted Answer

Increase the Lambda function timeout and memory allocation. The most cost-effective way to reduce failures from Lambda timeouts without losing data is to increase the Lambda function's timeout and memory allocation (Option D). Lambda timeouts occur when the transformation takes longer than the allocated timeout. Increasing memory also increases CPU, which can speed up processing and reduce timeouts. This approach is cost-effective because you only pay for the increased resources when the function runs, and it avoids data loss since all records are still transformed. Option A (Kinesis Data Analytics) adds unnecessary complexity and cost. Option B (decreasing batch size) reduces the number of records per invocation, which can help but may increase costs because more invocations are needed; also, it doesn't directly address timeouts. Option C (sending failed records to an S3 dead-letter bucket) would result in data loss for those records, as they are not transformed, and the question says 'without losing data'. Thus, D is the best choice.

Answer

Use Kinesis Data Analytics to pre-process data before Firehose

Answer

Decrease the Firehose batch size to reduce the number of records per invocation

Answer

Configure the Firehose delivery stream to send failed records to an S3 dead-letter bucket

Question 5

A company wants to ingest data from an on-premises SQL Server database into Amazon Redshift. They need to transform the data during ingestion, such as masking PII columns. Which approach meets these requirements with minimal operational overhead?

Accepted Answer

Use AWS Database Migration Service (DMS) with transformation rules. AWS DMS can perform transformations, such as data masking, during the migration process and can load data directly into Amazon Redshift, minimizing operational overhead. Option A (AWS Glue ETL) adds an extra step and overhead compared to DMS. Option B (custom application on EC2) introduces high operational overhead for management and scaling. Option C (Kinesis Data Firehose) is designed for streaming data and is not suitable for batch ingestion from an on-premises SQL Server database.

Answer

Use AWS Glue ETL jobs to extract data from SQL Server, transform it, and load into Redshift

Answer

Use a custom application on EC2 to extract, transform, and load

Answer

Use Kinesis Data Firehose to stream data from SQL Server to Redshift

Question 6

A company ingests streaming data from social media APIs into Kinesis Data Streams. Each record is approximately 5 KB. The data must be enriched with geolocation information from a DynamoDB table before being stored in S3. The enrichment process takes about 200 ms per record. Which architecture minimizes latency and cost?

Accepted Answer

Use Kinesis Data Firehose with a Lambda function that queries DynamoDB. Kinesis Data Firehose with a Lambda function for enrichment is the lowest-latency, lowest-operational-cost option: Firehose handles buffering, scaling, and delivery to S3, while Lambda performs the DynamoDB lookup per record within Firehose's transformation window. This serverless combination avoids managing EC2 or Glue infrastructure and scales automatically with stream volume.

Answer

Use an EC2 instance running a custom application to consume from Kinesis, enrich, and write to S3

Answer

Use AWS Glue ETL jobs running continuously on the stream

Answer

Use Kinesis Data Analytics to perform enrichment with SQL

Question 7

A company runs a data pipeline using AWS Glue ETL jobs to process daily files from an S3 bucket. The files are in CSV format and range from 1 GB to 10 GB. The Glue job runs successfully for small files but fails with an 'Out of Memory' error for files larger than 5 GB. The job uses a single G.1X DPU (16 GB memory). The company needs to process these large files without changing the existing ETL script. Which solution should the company implement?

Accepted Answer

Increase the number of DPUs and use the G.2X worker type to provide more memory per worker.. The Glue job fails with Out of Memory because a single G.1X worker provides only 16 GB of memory, which is insufficient for processing 5-10 GB CSV files loaded into memory during transformation. Switching to G.2X workers provides 32 GB of memory per worker and adding more DPUs distributes the workload, directly resolving the memory constraint without modifying the ETL script. This is the least disruptive solution that addresses the root cause.

Answer

Convert the input files from CSV to Parquet format to reduce memory usage.

Answer

Use the Optimus format in AWS Glue to compress data.

Answer

Use Amazon EMR with Spark instead of AWS Glue.

Question 8

A company has a Kinesis Data Firehose delivery stream that receives JSON data from IoT devices. The data is delivered to an S3 bucket. The company notices that the data in S3 is delayed by up to 30 minutes. The Firehose stream is configured with a buffer size of 1 MB and a buffer interval of 60 seconds. The incoming data rate is approximately 100 KB per second. The company needs to reduce the delivery latency to under 5 minutes. Which action should the company take?

Accepted Answer

Decrease the buffer size to 256 KB.. The observed latency of up to 30 minutes is likely due to the buffer size being too large relative to the data rate, causing long waits to fill the buffer. With a data rate of 100 KB/s and a buffer size of 1 MB, the buffer fills in approximately 10 seconds, but the buffer interval of 60 seconds already limits delivery to at most 60 seconds. However, the 30-minute delay suggests additional issues such as backlog or configuration errors. Decreasing the buffer size to 256 KB will cause more frequent deliveries (every ~2.5 seconds), reducing latency. Option A (Lambda transformation) adds processing time and increases latency. Option B (increase buffer interval to 300 seconds) would increase latency. Option C (change compression) does not affect delivery frequency. Therefore, option D is correct.

Answer

Enable Lambda transformation to process data faster.

Answer

Increase the buffer interval to 300 seconds.

Answer

Change the compression format from GZIP to Snappy.

Question 9

A company needs to ingest data from an on-premises MySQL database into Amazon S3 for analytics. The database is 2 TB in size. The company has a low-bandwidth internet connection (10 Mbps). They need to perform an initial full load and then incremental updates every hour. Which approach should they use?

Accepted Answer

Use AWS Database Migration Service (DMS) to perform the full load and ongoing replication.. AWS DMS is designed for both one-time full load and ongoing change data capture (CDC) replication from supported sources like MySQL to targets like S3. It handles the 2 TB initial load over the network using compression and parallel threads, and then continuously replicates incremental changes every hour or in near real-time, satisfying both requirements without manual ETL coding.

Answer

Use Kinesis Data Firehose to stream data from MySQL to S3.

Answer

Use AWS Glue ETL jobs to extract data and load into S3.

Answer

Use AWS Snowball Edge to transfer the initial full load, then use AWS DataSync for incremental updates.

Question 10

A data engineer needs to ingest daily CSV files from an external FTP server into Amazon S3. The files are 5 GB each. Which service is MOST suitable to automate this ingestion?

Accepted Answer

AWS DataSync. AWS DataSync is purpose-built for automated, high-speed data transfer between on-premises storage (including FTP servers) and AWS storage services like S3. It handles scheduling, integrity verification, encryption in transit, and bandwidth throttling, making it the most suitable service for recurring ingestion of large CSV files.

Answer

AWS AppSync

Answer

Amazon S3 Transfer Acceleration

Answer

AWS Glue

Question 11

A data engineer is tasked with ingesting on-premises database snapshots (full load) into Amazon S3 on a daily basis. The database is PostgreSQL and the snapshot size is 50 GB. The network link is 1 Gbps. Which approach is the MOST time-efficient and cost-effective?

Accepted Answer

Use AWS Database Migration Service (DMS) with S3 as target.. AWS Database Migration Service (DMS) is a managed service that efficiently performs full loads from on-premises PostgreSQL to Amazon S3, handling schema conversion and data transfer automatically. For a 50 GB snapshot over a 1 Gbps link, the transfer takes approximately 7 minutes, making Snowball (Option B) overkill and more expensive. Option C (AWS CLI) would require manual steps and is less efficient for database snapshots. Option D (Lambda + pg_dump) adds unnecessary complexity and potential performance issues. Therefore, DMS is the most time-efficient and cost-effective approach.

Answer

Use AWS Snowball Edge to transfer the snapshot.

Answer

Use AWS CLI to copy the snapshot file directly to S3.

Answer

Write a Lambda function to run pg_dump and upload to S3.

Question 12

A company is using Amazon Kinesis Data Firehose to deliver streaming data to an S3 bucket. The data includes a timestamp field. They want to partition the S3 objects by hour dynamically. The Firehose delivery stream is configured with a prefix like 'year=!{timestamp:yyyy}/month=!{timestamp:MM}/day=!{timestamp:dd}/hour=!{timestamp:HH}/'. However, the objects are not being partitioned as expected; all files end up in a single partition. What is the MOST likely cause?

Accepted Answer

Dynamic partitioning is not enabled in the Firehose delivery stream configuration.. Firehose dynamic partitioning requires the feature to be explicitly enabled on the delivery stream and the prefix to use the !{partitionKeyFromQuery:...} or !{timestamp:...} syntax with a configured JQ expression or inline parsing. If dynamic partitioning is not enabled, Firehose treats the prefix literally and writes all objects to a single S3 prefix, which matches the symptom described. Enabling it and configuring the partition key expression resolves the issue.

Answer

The buffer size is set too large, delaying file delivery.

Answer

The timestamp is in UTC but the prefix uses local time.

Answer

The IAM role for Firehose lacks permissions to write to S3 with dynamic prefixes.

Question 13

A company uses AWS Glue to catalog and transform data in Amazon S3. The Glue ETL jobs are failing intermittently with 'ThrottlingException' errors. Which THREE actions can help mitigate this issue? (Select THREE.)

Accepted Answer

Implement exponential backoff and retry in the Glue job code.. Option A is correct because implementing exponential backoff and retry logic in the Glue job code allows the job to gracefully handle transient ThrottlingException errors by retrying API calls after progressively longer waits, which is the AWS-recommended best practice for throttling. Option C is correct because ThrottlingException often occurs when the account exceeds the default AWS Glue API request quotas (such as concurrent job runs or API call rates), so requesting a service quota increase from AWS Support directly raises those limits and reduces throttling. Option D is correct because enabling job bookmarking tracks previously processed data, so the Glue job only processes new data on subsequent runs, reducing the number of API calls and overall workload that could trigger throttling. Option B is not correct because increasing DPUs adds more compute capacity for the job but does not reduce the rate of API calls to Glue services, and may even increase concurrent requests, worsening throttling. Option E is not correct because switching from PySpark to Spark SQL changes the query language/engine used within the job but does not address the underlying API throttling issue.

Answer

Increase the number of DPUs for the Glue job.

Answer

Switch from PySpark to Spark SQL.

Question 14

A company is using Amazon Kinesis Data Firehose to ingest clickstream data from a website into an S3 bucket. The data is then analyzed using Amazon Athena. Recently, the company noticed that Athena queries are returning incomplete results for the last 30 minutes of data. The Firehose delivery stream is configured to buffer data for 60 seconds or 5 MB before delivering to S3. The S3 bucket has a lifecycle policy that transitions objects to Amazon S3 Glacier after 30 days. The IAM role for Firehose has permissions to write to S3 and access a CloudWatch Logs group. The engineer checks the Firehose monitoring and sees that the delivery rate is healthy, but the 'S3.Bytes' metric shows a spike in the last hour. The 'BackupToS3.Bytes' metric is zero. What is the MOST likely cause of the missing data?

Accepted Answer

The data is still being buffered in Firehose and has not yet been delivered to S3.. The Firehose delivery stream buffers data for up to 60 seconds or 5 MB before delivering to S3. Since the delivery rate is healthy and the S3.Bytes metric shows a spike (indicating recent deliveries), the most recent data (last 30 minutes) may still be in the buffer or in transit, not yet available for Athena queries. The BackupToS3.Bytes metric is zero, confirming that backup is not enabled, so data is not being diverted. The lifecycle policy transitions objects after 30 days, which is irrelevant to recent data. Therefore, the missing data is likely still buffered in Firehose and not yet delivered to S3.

Answer

The lifecycle policy is transitioning data before Athena can query it.

Answer

The backup is enabled and data is being sent to the backup bucket instead.

Answer

The IAM role for Firehose does not have permissions to write to the S3 bucket.

Question 15

A company uses AWS Glue to process CSV files stored in Amazon S3. The data pipeline runs daily, but recently some jobs have failed with a 'MemoryError'. The data volume has grown from 1 GB to 10 GB per day. What is the MOST cost-effective solution to resolve this issue?

Accepted Answer

Increase the number of DPUs (Data Processing Units) allocated to the Glue job.. Increasing the number of DPUs allocates more memory and processing capacity to the Glue job, directly addressing the MemoryError caused by data growth from 1 GB to 10 GB. Option A is wrong because changing the worker type to G.2X provides more memory per worker, but it is a more expensive option compared to simply increasing DPUs, which scales horizontally. Option C is wrong because converting CSV to Parquet improves performance and reduces storage but does not add memory to the Glue job; the job still runs with the same DPU allocation. Option D is wrong because migrating to Amazon EMR introduces additional operational complexity and cost; increasing DPUs in Glue is simpler and more cost-effective for scaling an existing job.

Answer

Change the Glue worker type from Standard to G.2X.

Answer

Convert the CSV files to Parquet format using an S3 batch operation.

Answer

Migrate the job to Amazon EMR with a larger cluster.