DEA-C01 Data Ingestion and Transformation Practice Test 11 — 15 Questions

Question 1

A data engineer is tasked with transforming JSON data from an S3 bucket into Parquet format for efficient querying. The transformation should run on a schedule every hour. Which AWS service is best suited for this task?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

AWS Glue. AWS Glue is the best choice because it is a fully managed ETL service designed specifically for transforming and cataloging data at scale. It can natively read JSON from S3, convert it to Parquet, and run on a scheduled hourly basis using a Glue job with a trigger, without requiring server management or custom infrastructure.

Answer

AWS Lambda

Answer

Amazon Athena

Answer

Amazon EMR

Question 2

A company uses Kinesis Data Streams to ingest IoT data. The data volume varies, and occasionally the shard write throughput is exceeded, causing ProvisionedThroughputExceeded exceptions. The data engineer needs to handle these spikes without losing data. Which approach is most cost-effective and requires minimal code changes?

Accepted Answer

Use Kinesis Data Firehose as a consumer with retries and buffer settings. Kinesis Data Firehose is the most cost-effective solution because it can be configured as a consumer of the Kinesis Data Stream with built-in retry logic and buffer settings (e.g., buffer size up to 128 MB or buffer interval up to 900 seconds). This handles throughput spikes by buffering data and retrying failed writes without requiring custom code, and it scales automatically without the need to manage shard counts.

Answer

Implement custom retry logic using the Kinesis Client Library with exponential backoff

Answer

Increase the number of shards to handle peak throughput

Answer

Send data to an SQS queue first, then have a Lambda function write to Kinesis

Question 3

A company wants to migrate on-premises data to Amazon S3 using AWS DataSync. The data is 10 TB and the network bandwidth is 1 Gbps. The migration must be completed within 48 hours. What should the data engineer do to meet the deadline?

Accepted Answer

Use AWS DataSync with multiple agents and enable data compression. AWS DataSync can use multiple agents in parallel to increase throughput, and enabling data compression reduces the amount of data transferred over the network. With 10 TB at 1 Gbps, the theoretical minimum transfer time is about 22.2 hours, but real-world overhead (protocol, retransmissions) often exceeds 48 hours without parallelism and compression. Multiple agents and compression together can achieve the required throughput within the deadline.

Answer

Use S3 Transfer Acceleration to speed up the transfer

Answer

Use AWS Snowball Edge to transfer the data physically

Answer

Request a bandwidth increase from the ISP

Question 4

A data engineer is designing a data pipeline that ingests streaming data from Kinesis Data Streams, transforms it using AWS Lambda, and writes to S3. The Lambda function sometimes fails due to transient errors, and the engineer wants to ensure no data is lost. Which approach should be used?

Accepted Answer

Use Kinesis Data Firehose as the delivery stream with Lambda for transformation and configure error handling with retries and a backup S3 bucket. Kinesis Data Firehose natively integrates with Lambda for data transformation and provides built-in error handling with retries. If transformation fails after the retry limit, Firehose can automatically redirect failed records to a backup S3 bucket, ensuring no data is lost. This approach handles transient errors without requiring custom checkpointing or manual retry logic.

Answer

Use the Kinesis Client Library to process records with checkpointing

Answer

Increase the Lambda function's timeout and memory

Answer

Configure a dead-letter queue (DLQ) on Lambda to capture failed records

Question 5

A company needs to ingest data from multiple SaaS applications (e.g., Salesforce, Marketo) into Amazon S3 for centralized analytics. The data volume is several GB per day. Which AWS service is most suitable for this ingestion?

Accepted Answer

AWS Glue. AWS Glue is the most suitable service because it provides a fully managed ETL (Extract, Transform, Load) capability with built-in connectors for SaaS applications like Salesforce and Marketo. Glue can extract data from these sources, transform it as needed, and write it to Amazon S3, handling the several GB per day volume efficiently without requiring custom code for each source.

Answer

Amazon Kinesis Data Firehose

Answer

Amazon Athena

Answer

AWS Data Pipeline

Question 6

A company is ingesting log files from EC2 instances into CloudWatch Logs and then wants to deliver them to S3 for long-term storage and analysis. The data engineer needs to ensure the logs are delivered to S3 within 5 minutes of being generated. Which approach meets this requirement?

Accepted Answer

Use CloudWatch Logs subscription filter with Kinesis Data Firehose to deliver to S3. CloudWatch Logs subscription filters can stream log data in near real-time to Kinesis Data Firehose, which then delivers the data to S3 with a buffer interval configurable down to 60 seconds, easily meeting the 5-minute requirement. This approach provides the lowest latency for automated, continuous delivery without custom code.

Answer

Configure a CloudWatch Logs metric filter and invoke a Lambda function to write to S3

Answer

Use CloudWatch Logs Insights to query logs and save results to S3

Answer

Use the CloudWatch Logs export to S3 feature

Question 7

A company is building a data lake on S3 and needs to ingest data from on-premises Oracle database. The data is 5 TB and changes incrementally. The ingestion must capture changes in near real-time (less than 1 minute latency) and be cost-effective. Which approach should be used?

Accepted Answer

Use AWS Database Migration Service (DMS) with ongoing replication to S3. AWS DMS with ongoing replication captures incremental changes from Oracle using its native change data capture (CDC) mechanism, such as Oracle LogMiner or binary logs, and streams them to S3 in near real-time with latency under 1 minute. This approach is cost-effective because DMS charges only for the compute resources used during replication, and S3 storage is inexpensive, making it ideal for a 5 TB dataset with continuous changes.

Answer

Use Amazon Kinesis Data Firehose with an Oracle JDBC connector

Answer

Use AWS Glue to perform a full table export daily

Answer

Use AWS DataSync to sync the Oracle data files to S3

Question 8

A data engineer needs to transform CSV files arriving in S3 into Parquet format and partition them by date. The transformation should be event-driven and run immediately after each file is uploaded. Which approach is most efficient?

Accepted Answer

Use S3 event notification to trigger an AWS Glue job. AWS Glue jobs can be triggered directly by S3 event notifications, enabling event-driven, serverless transformation of CSV to Parquet with partitioning by date. Glue is optimized for this batch ETL workload, handling schema inference and partitioning efficiently without managing infrastructure, making it the most efficient choice for immediate, per-file transformation.

Answer

Use an S3 event notification to invoke a Lambda function that converts the file

Answer

Use an Amazon EMR cluster running Spark to process files as they arrive

Answer

Use Amazon Athena CREATE TABLE AS SELECT (CTAS) on a schedule

Question 9

A data engineer is designing a data ingestion pipeline that uses AWS Lambda to process records from a Kinesis Data Stream and write to DynamoDB. Which TWO strategies can help handle increased throughput and prevent data loss? (Choose TWO.)

Accepted Answer

Configure the Lambda event source mapping with a batch window and set the number of concurrent batches per shard. Configuring a batch window allows Lambda to accumulate records from the Kinesis stream for up to 300 seconds before invoking the function, which helps smooth out traffic spikes and reduces the number of invocations. Setting the number of concurrent batches per shard (via the ParallelizationFactor, up to 10) enables Lambda to process multiple batches from the same shard in parallel, increasing throughput without data loss. This combination ensures that records are processed efficiently even under high load, as Lambda can handle more concurrent executions per shard while batching reduces the risk of throttling.

Answer

Use synchronous invocation of Lambda from the producer

Answer

Configure a dead-letter queue (DLQ) for the Lambda function

Answer

Increase the Lambda function timeout

Question 10

A company is using AWS Glue to run ETL jobs that transform data from S3 to Redshift. The jobs are failing intermittently with out-of-memory errors. Which THREE actions can help resolve this issue? (Choose THREE.)

Accepted Answer

Increase the number of DPUs allocated to the Glue job. Increasing the number of DPUs allocated to the Glue job provides more memory and compute resources for the Spark executors, directly addressing out-of-memory errors by allowing larger datasets to be processed without exceeding heap limits. This is a standard scaling approach for memory-intensive ETL workloads in AWS Glue.

Answer

Use S3 Select to filter data before reading into the Glue job

Answer

Use Spark's 'coalesce' function to reduce the number of partitions

Question 11

A data engineer needs to set up a data ingestion pipeline that reads from Amazon MSK (Managed Streaming for Kafka) and writes to Amazon S3 with transformations. The data is in Avro format and must be converted to Parquet. Which THREE components should be used together? (Choose THREE.)

Accepted Answer

AWS Lambda function to convert Avro to Parquet as a Firehose transformation. AWS Lambda can be used as a transformation function within a Kinesis Data Firehose delivery stream to convert Avro records to Parquet format before delivery to S3. This is a serverless, real-time approach that integrates directly with Firehose's transformation capabilities, avoiding the need for separate compute resources.

Answer

Amazon Athena to convert the data format

Answer

AWS Glue ETL job to read from MSK

Question 12

A data engineer needs to ingest streaming data from thousands of IoT devices into Amazon S3 in near real-time. The data must be processed with minimal latency and stored in a columnar format for analytics. Which service should the engineer use to ingest the data?

Accepted Answer

Amazon Kinesis Data Firehose. Amazon Kinesis Data Firehose is the correct choice because it is designed to ingest streaming data and deliver it directly to Amazon S3 with near-real-time latency (typically 60 seconds or more, depending on buffer settings). It automatically converts data to columnar formats like Parquet or ORC, which is essential for analytics, and requires no custom code for the delivery pipeline, minimizing operational overhead.

Answer

Amazon Kinesis Data Analytics

Answer

Amazon Simple Queue Service (SQS)

Answer

Amazon Kinesis Data Streams with a Lambda consumer

Question 13

A company uses AWS Glue ETL jobs to transform data from Amazon RDS to Amazon S3 daily. The job recently started failing with memory errors. The data volume has grown 3x in the past month. Which change should the data engineer make to resolve the issue?

Accepted Answer

Increase the number of DPUs allocated to the Glue job. The Glue job is failing with memory errors due to a 3x increase in data volume. Increasing the number of DPUs (Data Processing Units) allocated to the job provides more memory and compute resources, directly addressing the out-of-memory condition without changing the job logic or architecture.

Answer

Increase the size of the Amazon RDS instance

Answer

Switch the Glue job type from Python Shell to Spark

Answer

Partition the output data in Amazon S3 by date

Question 14

A data engineer needs to design a data ingestion pipeline that captures change data capture (CDC) events from an on-premises SQL Server database to Amazon S3 with low latency. The pipeline must handle schema changes and ensure exactly-once delivery semantics. Which combination of AWS services should the engineer use?

Accepted Answer

AWS Database Migration Service (DMS) with Amazon Kinesis Data Firehose to Amazon S3. AWS DMS can capture ongoing changes from SQL Server using its CDC capability and stream them directly into Amazon Kinesis Data Firehose, which buffers and delivers data to Amazon S3 with low latency. DMS handles schema changes by propagating them to the target, and Kinesis Data Firehose, combined with DMS's transactional integrity, supports exactly-once delivery semantics when configured with a primary key and appropriate error handling.

Answer

AWS AppFlow with SQL Server connector to Amazon S3

Answer

AWS Glue ETL job with JDBC connection to SQL Server and writing to Amazon S3

Answer

Amazon Kinesis Data Streams with AWS Lambda consumer writing to Amazon S3

Question 15

A data engineer is troubleshooting an AWS Glue ETL job that fails with the error: 'An error occurred while calling o137.pyWriteDynamicFrame. No such file or directory: s3://bucket/output/part-00000.parquet'. The job reads from a JDBC source and writes to S3. What is the most likely cause?

Accepted Answer

The output S3 path does not exist and the Glue job does not have permission to create it. The error 'No such file or directory: s3://bucket/output/part-00000.parquet' indicates that the Glue job is trying to write to an S3 path that does not exist. By default, AWS Glue does not automatically create the target S3 bucket or prefix; it requires the path to already exist or the IAM role to have s3:PutObject permissions that allow the S3 service to create the object. Since the error occurs at the write stage (pyWriteDynamicFrame) and not during read, the most likely cause is that the output S3 path does not exist and the Glue job lacks the necessary permissions to create it.

Answer

The schema of the source data has changed, causing a mismatch during write

Answer

The Glue job ran out of memory during the transformation phase

Answer

The IAM role attached to the Glue job lacks permissions to read from the JDBC source