DEA-C01 Data Ingestion and Transformation Practice Test 22 — 15 Questions

Question 1

A data engineering team is building a pipeline to transform CSV files uploaded to Amazon S3 into Parquet format using AWS Glue. The transformation must be serverless and handle files that arrive at irregular intervals. Which TWO actions should the team take? (Choose two.)

Accepted Answer

Use an AWS Glue ETL job to convert CSV to Parquet.. Option A: AWS Glue is serverless and can transform data. Option C: S3 event notifications can trigger a Lambda function to start the Glue job. Option B (Amazon EMR) is server-based and requires cluster management. Option D (AWS Data Pipeline) is not serverless. Option E (Amazon Redshift Spectrum) queries data in place but does not transform file formats.

Answer

Configure an Amazon EMR cluster with Apache Spark for on-demand transformation.

Answer

Use AWS Data Pipeline to schedule a periodic transformation.

Answer

Use Amazon Redshift Spectrum to convert files during query execution.

Question 2

A company ingests IoT sensor data into Amazon Kinesis Data Streams. The data must be enriched with device metadata from Amazon DynamoDB and then stored in Amazon S3 in Apache Parquet format. The solution must minimize latency and cost. Which THREE steps should a data engineer implement? (Choose three.)

Accepted Answer

Deliver the enriched data to Amazon Kinesis Data Firehose and enable Parquet conversion.. Option A: Using Kinesis Data Analytics for Flink allows stream enrichment with low latency. Option D: Kinesis Data Firehose can buffer data and convert it to Parquet before delivering to S3. Option E: Kinesis Data Analytics for Flink can look up DynamoDB data for enrichment. Option B (Lambda) has invocation limits and higher cost per record. Option C (Glue) adds latency and cost for streaming. Option F (EMR) is managed and not as seamless.

Answer

Configure an AWS Lambda function to read from the stream, enrich, and write to S3.

Answer

Use AWS Glue streaming ETL to enrich and convert data to Parquet.

Answer

Use Amazon EMR with Spark Streaming to process and store the data.

Question 3

A financial services company ingests stock trade data from multiple exchanges into an Amazon S3 bucket (trade-bucket). Each exchange sends a CSV file every 5 minutes. The data must be transformed into Parquet format and partitioned by exchange and date (trade_date) for efficient querying using Amazon Athena. The pipeline must handle late-arriving data (files up to 2 hours late) and ensure exactly-once processing to avoid duplicates. Currently, a scheduled AWS Glue ETL job runs every hour, reads new CSV files, converts them to Parquet, and writes to an output bucket. However, the team is experiencing data duplication: if the job fails midway, upon retry it reprocesses all files in the input folder, causing duplicates in the output. Additionally, the job takes too long because it scans all files each run. The engineer must redesign the pipeline to eliminate duplicates and improve efficiency. What should the engineer do?

Accepted Answer

Set up an S3 event notification to invoke an AWS Lambda function that starts a Glue job with a parameter containing the S3 object key of the new file; modify the Glue job to process only that file and use the file key to avoid duplicates.. Option D is the best approach: S3 event notifications trigger Lambda to start a Glue job passing the file key; the job processes only that file, using the file key to avoid reprocessing. Option A still processes all files and may cause duplicates if a file arrives after the job starts. Option B (Glue Workflows) does not solve the file-level tracking. Option C (EMR) is overkill and still scans the entire input folder.

Answer

Use AWS Glue Workflows to orchestrate the job and add a condition to check for duplicates before writing.

Answer

Modify the Glue job to move processed CSV files to an archive folder after successful transformation, and process only unprocessed files.

Answer

Replace Glue with Amazon EMR and use Spark Structured Streaming with checkpointing to process files incrementally.

Question 4

A retail company uses Amazon Kinesis Data Firehose to ingest clickstream data from its website into an Amazon S3 bucket. The data includes fields: user_id, event_type, timestamp, page_url. Recently, the data engineering team noticed that some records have malformed JSON (missing commas, extra brackets) causing delivery failures to S3. The Firehose delivery stream is configured to retry failed records for 300 seconds, after which the records are sent to an S3 bucket for failed records. The team wants to transform the data to correct malformed JSON before delivery to the main S3 bucket. They need a solution that does not require managing servers and can handle high throughput. What should the team do?

Accepted Answer

Configure an AWS Lambda function as a data transformation in Kinesis Data Firehose to correct malformed JSON.. Option B is correct: Lambda transformation in Firehose can process each record and fix JSON errors. Option A (Kinesis Data Analytics) is for real-time analytics, not record-level transformation. Option C (Glue streaming) adds complexity and latency. Option D (EMR) requires cluster management.

Answer

Set up an Amazon EMR cluster with Apache Spark to process the data in micro-batches and fix JSON errors.

Answer

Use an AWS Glue streaming ETL job to read from Firehose and write corrected data to S3.

Answer

Use Amazon Kinesis Data Analytics with a SQL application to parse and fix JSON.

Question 5

A marketing analytics team needs to ingest customer transaction data from an on-premises PostgreSQL database into Amazon S3 for analysis. The data volume is about 10 GB daily, and the team wants to perform full refresh daily (truncate and load) into S3 as Parquet files. The company has a Direct Connect connection to AWS. The team needs a simple, managed solution that minimizes operational overhead. What should the team use?

Accepted Answer

Use an AWS Glue ETL job with a JDBC connection to the PostgreSQL database, extract data, and write to S3 in Parquet format.. Option A is correct: AWS Glue can connect to on-premises PostgreSQL via JDBC using a connection, and perform full extract and write to S3 as Parquet. Option B (DMS) is more suited for ongoing replication, not full refresh. Option C (Data Pipeline) requires more configuration. Option D (EMR) is overkill.

Answer

Set up AWS Database Migration Service (DMS) to continuously replicate data to S3 in Parquet format.

Answer

Use Amazon EMR with a Spark job that reads from PostgreSQL and writes to S3.

Answer

Use AWS Data Pipeline with a SQLActivity to extract data and copy to S3.

Question 6

A logistics company ingests real-time GPS location data from thousands of delivery vehicles into Amazon Kinesis Data Streams. Each vehicle sends a JSON payload every 10 seconds containing vehicle_id, latitude, longitude, timestamp, and speed. The data must be stored in Amazon S3 for historical analysis, but the company wants to first aggregate the data per vehicle per minute (average speed, min/max coordinates) to reduce storage costs. The solution must be serverless and handle potential duplicate records without double-counting. What should the engineer do?

Accepted Answer

Use Amazon Kinesis Data Analytics for Apache Flink to aggregate data in a 1-minute tumbling window with deduplication logic, then output to Kinesis Data Firehose for delivery to S3.. Option D is correct: Kinesis Data Analytics for Flink can aggregate data in real-time using tumbling windows and deduplication. Option A (Firehose with Lambda) would require per-record processing and state management. Option B (Glue streaming) adds latency. Option C (EMR) is not serverless.

Answer

Use Amazon EMR with Spark Streaming to perform the aggregation and write to S3.

Answer

Use Kinesis Data Firehose with a Lambda transformation to aggregate records in a 1-minute window.

Answer

Use an AWS Glue streaming ETL job with Spark Structured Streaming to aggregate and deduplicate.

Question 7

A healthcare company is building a data pipeline to ingest electronic health records (EHR) from hospitals. The data is sent as JSON files via SFTP to an on-premises server. The company wants to move this data to AWS using AWS Transfer Family (SFTP) and then process it with AWS Glue. Data sovereignty regulations require that all data remain within the EU (Frankfurt) region. The pipeline must detect when a new file arrives and start the Glue job automatically. The engineer has set up an AWS Transfer Family server in Frankfurt, and files are uploaded to an S3 bucket in the same region. However, the Glue job is not triggering automatically. The engineer needs to implement automated triggering. What should the engineer do?

Accepted Answer

Set up an S3 event notification on the bucket to invoke an AWS Lambda function that starts the Glue job.. Option B is correct: S3 event notifications can be configured to invoke Lambda when a new object is created, and Lambda can start the Glue job. Option A (CloudWatch Events) cannot directly monitor S3 object creation. Option C (SQS) is not needed. Option D (Step Functions) adds unnecessary complexity.

Answer

Configure AWS Step Functions to poll the S3 bucket every minute and start the Glue job if new files exist.

Answer

Configure Amazon CloudWatch Events to trigger the Glue job on a schedule that checks for new files.

Answer

Use Amazon Simple Queue Service (SQS) to queue file metadata and have a Lambda function poll the queue to start the Glue job.

Question 8

A media company ingests large video files (up to 100 GB each) from content creators via Amazon S3 multipart uploads. After upload, the company needs to transcode the videos into multiple formats using AWS Elemental MediaConvert. The current pipeline uses S3 event notifications to trigger an AWS Lambda function that starts a MediaConvert job. However, for very large files, the Lambda function times out (15-minute limit) before the upload completes because the event is sent when the multipart upload is initiated, not when it completes. How should the engineer fix this issue?

Accepted Answer

Change the S3 event notification to listen for s3:ObjectCreated:Put events instead of s3:ObjectCreated:*, which fires only after the object is fully written.. Option D is correct: Use S3 event notifications for s3:ObjectCreated:Put, which triggers only after the object is completely written. Then Lambda can start MediaConvert. Option A (increase timeout) does not solve the root cause. Option B (Step Functions) adds complexity. Option C (SQS) does not address the event timing.

Answer

Increase the Lambda timeout to 30 minutes.

Answer

Use Amazon SQS to queue the event and have a Lambda function process it later.

Answer

Use AWS Step Functions to poll the S3 bucket for object existence and then start MediaConvert.

Question 9

A data engineer needs to ingest log files from multiple EC2 instances into Amazon S3. The logs are written to local disk on each instance. The engineer wants a simple agent-based solution that can collect, compress, and upload logs to S3 with minimal configuration. The solution must support incremental uploads (only new log lines) and handle log rotation. What should the engineer use?

Accepted Answer

Install and configure Amazon Kinesis Agent for Amazon CloudWatch to send logs to CloudWatch Logs, then use a subscription filter to export logs to S3.. Option C is correct: Amazon Kinesis Agent for Amazon CloudWatch can tail log files, compress, and send to CloudWatch Logs, which can then be exported to S3. Option A (AWS DataSync) is for bulk transfers, not streaming. Option B (AWS CLI) is manual. Option D (S3 sync) is not real-time.

Answer

Use AWS CLI cp command with --recursive in a cron job to copy logs to S3 every minute.

Answer

Install AWS DataSync agent on each EC2 instance to sync logs to S3 daily.

Answer

Use an S3 sync command from the AWS CLI scheduled every hour.

Question 10

A financial company needs to ingest real-time stock trade data from multiple sources and store it in Amazon S3 for compliance. The data must be delivered within 1 minute of the trade occurring. The data volume is approximately 10,000 records per second, with occasional spikes to 50,000 records per second. The engineer has set up Amazon Kinesis Data Streams with 10 shards and a Kinesis Data Firehose delivery stream that reads from the Kinesis stream and writes to S3. However, during spikes, the Firehose delivery stream falls behind, causing data to be delayed beyond the 1-minute SLA. What should the engineer do to meet the SLA without over-provisioning?

Accepted Answer

Increase the number of shards in the Kinesis data stream to handle peak throughput and enable auto-scaling.. Option A is correct: Increase shard count to handle peak throughput, and enable automatic scaling for Kinesis Data Streams. Option B (increase buffer size) would increase latency. Option C (use Lambda) may not handle the throughput. Option D (use SQS) would not reduce latency.

Answer

Increase the buffer size in Kinesis Data Firehose from 1 MB to 5 MB to batch more data per delivery.

Answer

Use Amazon SQS as a buffer between Kinesis and Firehose to absorb spikes.

Answer

Replace Firehose with an AWS Lambda function that writes directly to S3 for lower latency.

Question 11

A social media company ingests user activity data from multiple sources into Amazon S3. The data is in JSON format and includes fields: user_id, activity_type, timestamp, and metadata. The company wants to transform this data into a columnar format (Parquet) partitioned by date and activity_type for efficient querying with Amazon Athena. The pipeline must handle data that arrives up to 3 days late. Currently, a daily AWS Glue ETL job scans the entire S3 bucket for new files, transforms them, and writes to a separate output bucket. The job is taking longer as data volume grows, and the team wants to reduce processing time and cost. What should the engineer do?

Accepted Answer

Set up S3 event notifications to invoke an AWS Lambda function that triggers a Glue job for each new object, passing the object key so the job processes only that file.. Option B is correct: Using S3 event notifications to trigger a Lambda function that starts a Glue job for each new file allows incremental processing, reducing scan time. Option A (increase capacity) does not address the root cause. Option C (EMR) adds complexity. Option D (partition projection) does not help with transformation.

Answer

Increase the number of DPUs for the Glue job to process data faster.

Answer

Use AWS Glue partition projection and schema inference to reduce scan time.

Answer

Replace AWS Glue with Amazon EMR and use Spark to process data in parallel.

Question 12

A gaming company ingests player event data from mobile games into Amazon Kinesis Data Streams. Each event is a small JSON payload (<1 KB). The data must be delivered to Amazon S3 for analytics, and the company wants to minimize storage costs by aggregating events into larger files (e.g., 100 MB per file). The current setup uses Kinesis Data Firehose with a buffer size of 10 MB and a buffer interval of 60 seconds, but the resulting files are very small (average 5 MB) because the data volume is low. The engineer needs to ensure that files are at least 100 MB to reduce the number of S3 objects and lower costs. What should the engineer do?

Accepted Answer

Increase the buffer size in Kinesis Data Firehose to 100 MB and increase the buffer interval to 300 seconds.. Option D is correct: Increase the buffer size to 100 MB and the buffer interval to 300 seconds to allow more data to accumulate before writing. Option A (Lambda) would not aggregate efficiently. Option B (Glue) adds latency and cost. Option C (EMR) is overkill.

Answer

Use an AWS Glue streaming ETL job with a 100 MB file size threshold to write to S3.

Answer

Use an AWS Lambda function to buffer events in memory and write to S3 when buffer reaches 100 MB.

Answer

Use Amazon EMR with Spark Streaming to aggregate and write larger files to S3.

Question 13

A data engineering team is ingesting streaming data from IoT devices into Amazon Kinesis Data Streams. The data must be transformed in real-time and then loaded into an Amazon S3 bucket for long-term storage. Which AWS service should be used to perform the transformation and delivery to S3 with minimal operational overhead?

Accepted Answer

Amazon Kinesis Data Firehose. Option B is correct because Kinesis Data Firehose can subscribe to a Kinesis Data Stream, transform data using Lambda or its built-in transformations, and automatically deliver to S3 with no code required for delivery. Option A is wrong because Kinesis Data Analytics is for real-time analytics, not direct delivery to S3. Option C is wrong because AWS Glue is a batch ETL service, not real-time. Option D is wrong because Amazon EMR is a managed Hadoop cluster that requires significant overhead.

Answer

AWS Glue

Answer

Amazon EMR

Answer

Amazon Kinesis Data Analytics

Question 14

A company uses AWS DMS to migrate a 2 TB PostgreSQL database to Amazon Aurora PostgreSQL. The migration is taking longer than expected due to the initial load. Which AWS service can be used to accelerate the initial load by transferring the database files directly?

Accepted Answer

AWS Snowball. Option C is correct because AWS Snowball can be used to transfer large datasets physically, bypassing network constraints. Option A is wrong because S3 Transfer Acceleration speeds up uploads to S3 over the internet, but the data still goes over the network. Option B is wrong because Direct Connect provides a dedicated network connection, but the data still traverses the network. Option D is wrong because Kinesis Data Firehose is for streaming, not bulk data transfer.

Answer

Amazon S3 Transfer Acceleration

Answer

AWS Direct Connect

Answer

Amazon Kinesis Data Firehose

Question 15

A data pipeline ingests daily CSV files from an FTP server into an Amazon S3 bucket. The files must be converted to Parquet format and partitioned by date for efficient querying using Amazon Athena. Which AWS service is most suitable for this transformation?

Accepted Answer

AWS Glue. Option D is correct because AWS Glue offers a serverless Spark environment with built-in transformation libraries and can automatically partition data. Option A is wrong because Lambda has a 15-minute timeout and 10 GB memory limit, making it unsuitable for large CSV-to-Parquet conversions. Option B is wrong because EMR requires cluster management. Option C is wrong because Kinesis Data Firehose is for streaming, not batch.

Answer

Amazon Kinesis Data Firehose

Answer

Amazon EMR

Answer

AWS Lambda