DEA-C01 Data Ingestion and Transformation Practice Test 21 — 15 Questions

Question 1

A company wants to ingest data from a SaaS application into Amazon S3. The SaaS application supports streaming data via HTTP POST requests. The data volume is approximately 100 MB per hour, and the company needs to store the raw data in S3 for archival and later analysis. Which approach is the most cost-effective and operationally efficient?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Use Amazon API Gateway to create a REST API that receives the data and triggers an AWS Lambda function to store it in S3.. Amazon API Gateway combined with AWS Lambda is the most appropriate and cost-effective approach for this use case. API Gateway provides a fully managed HTTP endpoint that can receive the HTTP POST requests from the SaaS application. The data is then passed to a Lambda function, which writes the raw data directly to Amazon S3. This serverless architecture eliminates the need to manage servers, scales automatically, and incurs cost only when data is processed. Option B is incorrect because Amazon Kinesis Data Firehose does not natively support HTTP endpoints as a source; it can ingest data from Kinesis Data Streams, AWS IoT, or custom agents, but not directly via HTTP POST.

Answer

Launch a t3.nano EC2 instance that runs a script to receive HTTP POST requests and write to S3.

Answer

Use Amazon Kinesis Data Firehose with HTTP endpoint as the source, and configure S3 as the destination.

Answer

Use Amazon Simple Queue Service (SQS) to queue the HTTP POST data and have an AWS Lambda function read from SQS and write to S3.

Question 2

A data engineer needs to transform JSON data from a Kinesis Data Stream into Parquet format and store it in an S3 data lake. The transformation includes simple field mapping and data type conversions. Which AWS service is the most cost-effective for performing this transformation in near-real-time?

Accepted Answer

AWS Lambda function triggered by Kinesis Data Firehose. AWS Lambda functions triggered by Kinesis Data Firehose are the most cost-effective solution for near-real-time transformations because Lambda allows you to perform lightweight field mapping and data type conversions on streaming data with a pay-per-invocation model, while Firehose handles buffering, compression, and direct delivery to S3 in Parquet format. This serverless approach avoids the overhead and cost of provisioning clusters or running continuous jobs, making it ideal for simple transformations on high-frequency, low-latency streams.

Answer

Amazon Athena with CTAS (CREATE TABLE AS SELECT)

Answer

AWS Glue ETL job

Answer

Amazon EMR with Spark Streaming

Question 3

A data engineer runs an AWS Glue crawler that is configured to crawl an S3 bucket named 'my-data-lake' and update the Glue Data Catalog. The crawler fails with an access denied error. The IAM role attached to the crawler has the policy shown in the exhibit. What is the likely cause of the failure?

Accepted Answer

The policy does not allow logging to CloudWatch Logs.. AWS Glue crawlers require permissions to write logs to CloudWatch Logs for monitoring and debugging. Without the `logs:CreateLogGroup`, `logs:CreateLogStream`, and `logs:PutLogEvents` actions, the crawler fails with an access denied error even if it has S3 and Glue Data Catalog permissions. The IAM policy shown does not include these CloudWatch Logs permissions, making option C the correct answer.

Answer

The policy does not allow glue:CreateTable on the 'my-data-lake' database.

Answer

The policy does not allow s3:PutObject on the 'my-data-lake' bucket.

Answer

The policy does not allow s3:ListBucket on the 'my-data-lake' bucket.

Question 4

A company is using AWS Database Migration Service (DMS) to migrate a 2 TB Oracle database to Amazon Aurora PostgreSQL. The migration must have minimal downtime. The source database is highly active with continuous writes. Which DMS migration type and additional configuration should the engineer use?

Accepted Answer

Use a full load migration task followed by ongoing replication (CDC).. A full load migration followed by ongoing replication (CDC) allows the initial data copy to complete while continuously capturing and applying incremental changes from the highly active source. This minimizes downtime by keeping the target database nearly synchronized, requiring only a brief cutover window to stop writes and finalize replication.

Answer

Use a CDC-only migration task to capture changes from the source.

Answer

Use a full load migration task and stop the source database before starting.

Answer

Use a full load migration task with task restart enabled.

Question 5

A data engineer needs to schedule an AWS Glue ETL job to run every hour and process new data that arrives in an S3 bucket. The job should only process files that have been added since the last run. Which approach should the engineer use to track which files have been processed?

Accepted Answer

Enable job bookmarks in the Glue ETL job.. AWS Glue job bookmarks automatically track the last processed data, allowing the job to incrementally process only new files since the last run. This is the native, built-in mechanism for stateful incremental processing in Glue ETL, eliminating the need for external tracking.

Answer

Configure S3 Event Notifications to trigger the Glue job on each new object creation.

Answer

Store the last processed timestamp in a DynamoDB table and query it at the start of the job.

Answer

Use S3 Inventory to list all objects and filter by last modified date in the job.

Question 6

A data engineer is designing a data ingestion pipeline for real-time clickstream data from a website. The data must be ingested with low latency (seconds) and made available for multiple consumer applications, including a dashboard that refreshes every minute and a machine learning model that processes data in near-real-time. The engineer needs to choose a streaming ingestion service. Which TWO services meet these requirements? (Select TWO.)

Accepted Answer

Amazon Managed Streaming for Apache Kafka (Amazon MSK). Amazon Kinesis Data Streams (C) provides sub-second ingestion latency and supports multiple consumer applications via its enhanced fan-out feature, enabling a dashboard and ML model to consume data concurrently with low latency. Amazon MSK (B) offers similar real-time capabilities with Apache Kafka's native pub/sub model, allowing multiple consumers to process the same stream independently and with low latency, meeting the near-real-time requirements.

Answer

Amazon Kinesis Data Firehose

Answer

Amazon Simple Queue Service (SQS)

Answer

Amazon S3

Question 7

A data engineer is troubleshooting a Kinesis Data Streams consumer application that is falling behind. The stream has 10 shards and is receiving 5 MB/s of data. The consumer uses the Kinesis Client Library (KCL) with a single worker. The worker is processing all 10 shards but is experiencing high latency and checkpointing delays. Which THREE actions should the engineer take to improve consumer performance? (Select THREE.)

Accepted Answer

Increase the number of KCL workers to match the number of shards.. The KCL worker is processing all 10 shards sequentially within a single worker, causing a bottleneck. By increasing the number of KCL workers to match the number of shards, each worker can process one shard in parallel, significantly improving throughput and reducing latency. This is a standard scaling pattern for KCL-based consumers.

Answer

Decrease the checkpoint interval to reduce checkpointing overhead.

Answer

Increase the number of shards in the stream.

Question 8

A data engineer needs to transform data in an S3 data lake using AWS Glue ETL. The data is in CSV format and needs to be converted to Parquet with partitioning by date. The engineer wants to minimize the number of files written to S3 to improve query performance. Which TWO configuration options should the engineer use? (Select TWO.)

Accepted Answer

Use the coalesce method to reduce the number of output partitions.. Using `coalesce` reduces the number of output partitions, which directly minimizes the number of files written to S3. Fewer, larger Parquet files improve query performance by reducing the overhead of file listing and metadata operations in engines like Amazon Athena or Redshift Spectrum.

Answer

Increase the number of workers in the Glue job to increase parallelism.

Answer

Disable compression in the Parquet output.

Answer

Set the 'groupFiles' option to 'inPartition' in the DynamicFrame writer.

Question 9

A company runs an e-commerce platform that generates clickstream data from user interactions on their website. The data is sent as JSON objects via HTTP POST to an API Gateway endpoint, which triggers a Lambda function that writes each record to a Kinesis Data Stream (100 shards). A second Lambda function consumes the stream, transforms the data (enriches with geolocation from a DynamoDB table), and writes to a Kinesis Data Firehose delivery stream that delivers Parquet files to an S3 data lake every 5 minutes. The system has been working for months, but recently the Firehose delivery stream started showing 'DeliveryFailed' errors for a subset of records. The errors point to 'InvalidData' from the Lambda transformation. The engineer reviews the Lambda transformation code and notices that the geolocation lookup occasionally fails because the DynamoDB table has a throttling issue. The engineer needs to handle these failures gracefully so that records that fail enrichment are still delivered to S3 with a null geolocation field, without blocking other records. Which course of action should the engineer take?

Accepted Answer

Modify the Lambda function to catch exceptions during the geolocation lookup, set the geolocation field to null, and continue processing the record.. It modifies the Lambda function to catch exceptions during the geolocation lookup, set the geolocation field to null, and continue processing. This ensures that records that fail enrichment are still delivered to S3 with a null geolocation field, without blocking other records, and without requiring additional infrastructure. Option A is incorrect because Kinesis Data Firehose does not natively support a dead-letter queue (DLQ); failed records can be sent to an S3 bucket for failed data, but that would not include the transformed data with null geolocation. Option B is incorrect because sending failed records to a separate Kinesis Data Stream adds complexity and does not ensure they are delivered to S3 with the desired null geolocation field. Option D is incorrect because increasing RCUs may reduce throttling but does not eliminate the possibility of failures, and it increases cost; the requirement is to handle failures gracefully, not prevent them entirely.

Answer

Configure the Kinesis Data Firehose delivery stream to send failed records to a dead-letter queue (DLQ) for later reprocessing.

Answer

Modify the Lambda function to send failed records to a separate Kinesis Data Stream for manual processing.

Answer

Increase the read capacity units (RCUs) on the DynamoDB table to eliminate throttling.

Question 10

A data engineer is responsible for ingesting log files from a fleet of on-premises servers into Amazon S3 for central analysis. Each server generates log files that are rotated every hour, resulting in files of about 500 MB each. The total daily data volume is approximately 1 TB. The network connection between the on-premises data center and AWS is a 100 Mbps VPN. The engineer needs to ensure that all log files are transferred to S3 within 24 hours of generation without data loss. The engineer is considering using AWS DataSync. However, the initial setup shows that the transfer speed is insufficient to meet the 24-hour SLA. What should the engineer do to meet the requirement?

Accepted Answer

Configure AWS DataSync to run on a schedule with incremental transfers and enable data compression.. AWS DataSync can be configured for incremental transfers after an initial full sync, which reduces the amount of data transferred on a daily basis. Additionally, enabling data compression reduces the size of the log files during transfer, making it feasible to meet the 24-hour SLA over a 100 Mbps VPN. Option A is incorrect because while AWS CLI with multipart uploads can improve throughput, it still requires manual scripting and does not provide the same optimization as DataSync. Option B is incorrect because upgrading the VPN bandwidth may not be immediately feasible or cost-effective. Option C is incorrect because ordering an AWS Snowball Edge device is intended for large initial data migrations, not for ongoing daily transfers; it would introduce delays and is not suitable for recurring incremental transfers.

Answer

Use AWS CLI with multipart uploads and parallel threads to maximize throughput.

Answer

Contact the network team to upgrade the VPN bandwidth to at least 1 Gbps.

Answer

Order an AWS Snowball Edge device to transfer the initial data and then use DataSync for incremental changes.

Question 11

A data engineer is setting up a data pipeline to ingest data from an Amazon RDS for MySQL database into Amazon S3 using AWS Glue ETL. The Glue job uses a JDBC connection to read from the MySQL database. The job runs successfully, but the engineer notices that the job is taking longer than expected. The MySQL database is 500 GB in size and the Glue job uses 10 workers of type G.1X. The engineer wants to improve the performance of the extraction phase. The database is actively used by other applications, so the engineer must minimize the impact on the source database. Which approach should the engineer take?

Accepted Answer

Partition the table by a numeric column, such as the primary key, and use the 'hashex' or 'hashpar' partitioning option in the Glue JDBC connection.. Partitioning the table on a key column (e.g., primary key) allows Glue to read in parallel from multiple partitions, reducing the load on the database and improving performance. Option B is wrong because incremental extraction is for ongoing changes, not for an initial full load; it doesn't address the immediate performance issue of extracting 500 GB. Option C is wrong because using a read replica offloads read traffic but does not inherently improve parallelism; partitioning is still needed for performance. Option D is wrong because simply increasing the number of workers may overwhelm the database with more simultaneous connections without partitioning, potentially causing performance degradation.

Answer

Use an incremental extraction strategy with a watermark column to reduce the amount of data read each time.

Answer

Create a read replica of the MySQL database and configure the Glue job to read from the replica.

Answer

Increase the number of Glue workers to 20 to increase parallelism.

Question 12

A company uses AWS Glue to process data from multiple sources. The data is stored in an Amazon S3 data lake. The company needs to transform the data using a custom Python library that is not available in the default Glue environment. What is the MOST efficient way to make this library available to the Glue jobs?

Accepted Answer

Upload the library as a .whl file to Amazon S3 and reference it in the Glue job's --additional-python-modules parameter.. AWS Glue supports adding custom Python libraries by uploading a .whl file to Amazon S3 and referencing it via the `--additional-python-modules` job parameter. This method is the most efficient as it requires no manual node configuration, no custom Docker images, and no runtime pip installs, ensuring the library is automatically distributed to all worker nodes before the job executes.

Answer

Manually install the library on each node in the Glue cluster by editing the bootstrap script.

Answer

Create a custom Docker image with the library and use it in AWS Glue for Ray.

Answer

Use a shell command in the Glue job script to run 'pip install <library>' before the job runs.

Question 13

A data engineer is designing a real-time streaming pipeline to ingest clickstream data from a website into Amazon S3. The data must be transformed before storage. Which TWO AWS services can be used together to build this pipeline? (Choose TWO.)

Accepted Answer

Amazon Kinesis Data Firehose. The correct combination is Amazon Kinesis Data Streams to ingest the streaming clickstream data, and Amazon Kinesis Data Firehose to read from the stream, perform transformations (e.g., via Lambda), and deliver the transformed data to Amazon S3. Kinesis Data Streams provides the durable ingestion layer, while Kinesis Data Firehose handles the delivery and transformation without managing servers.

Answer

AWS Glue

Answer

Amazon S3 Transfer Acceleration

Answer

AWS Database Migration Service (DMS)

Question 14

A company uses AWS Glue to run ETL jobs daily. The jobs consume data from an Amazon RDS for MySQL database and write results to Amazon S3. The company wants to minimize the impact on the source database during extraction. Which THREE actions should the data engineer take to achieve this? (Choose THREE.)

Accepted Answer

Schedule the Glue job to run during off-peak hours.. Scheduling the Glue job to run during off-peak hours minimizes the load on the source RDS for MySQL database by avoiding high-traffic periods, reducing contention for CPU, memory, and I/O resources. This is a straightforward operational practice to reduce impact on production databases during extraction.

Answer

Increase the number of Glue DPUs to process data faster.

Answer

Disable Glue job bookmarks to force full refresh.

Question 15

A data engineer is building a data ingestion pipeline using AWS Glue. The source is an Amazon DynamoDB table, and the target is an Amazon S3 data lake in Parquet format. The pipeline must handle large volumes and ensure exactly-once processing. Which THREE features should the engineer use together to achieve this? (Choose THREE.)

Accepted Answer

Configure the Glue job to convert data to Parquet format.. Converting data to Parquet format is a core requirement for an S3 data lake, as Parquet offers columnar storage, compression, and efficient querying via services like Amazon Athena and Amazon Redshift Spectrum. AWS Glue natively supports Parquet as an output format, enabling the engineer to specify it in the job's output schema or transformation logic.

Answer

Use Amazon Kinesis Data Streams to capture DynamoDB Streams changes.

Answer

Use Amazon S3 Object Lambda to transform data on the fly.