DEA-C01 Data Ingestion and Transformation Practice Test 23 — 15 Questions

Question 1

A retail company uses Amazon Kinesis Data Firehose to ingest clickstream data from its website into an Amazon S3 bucket. The data includes fields: user_id, event_type, timestamp, page_url. Recently, the data engineering team noticed that some records have malformed JSON (missing commas, extra brackets) causing delivery failures to S3. The Firehose delivery stream is configured to retry failed records for 300 seconds, after which the records are sent to an S3 bucket for failed records. The team wants to transform the data to correct malformed JSON before delivery to the main S3 bucket. They need a solution that does not require managing servers and can handle high throughput. What should the team do?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Configure an AWS Lambda function as a data transformation in Kinesis Data Firehose to correct malformed JSON.. AWS Lambda can be used as a data transformation function within Kinesis Data Firehose to process each record individually, fixing malformed JSON without managing servers. The Lambda function receives records in batches, transforms them (e.g., parsing and correcting JSON syntax), and returns them to Firehose for delivery to S3. This serverless approach scales automatically with high throughput. Option B (EMR with Spark) requires managing a cluster and adds latency, making it less suitable for real-time streaming. Option C (Glue streaming ETL) is serverless but introduces more complexity and potential latency compared to Firehose's built-in Lambda transformation. Option D (Kinesis Data Analytics) is designed for real-time analytics and not for record-level transformations like JSON correction.

Answer

Set up an Amazon EMR cluster with Apache Spark to process the data in micro-batches and fix JSON errors.

Answer

Use an AWS Glue streaming ETL job to read from Firehose and write corrected data to S3.

Answer

Use Amazon Kinesis Data Analytics with a SQL application to parse and fix JSON.

Question 2

A marketing analytics team needs to ingest customer transaction data from an on-premises PostgreSQL database into Amazon S3 for analysis. The data volume is about 10 GB daily, and the team wants to perform full refresh daily (truncate and load) into S3 as Parquet files. The company has a Direct Connect connection to AWS. The team needs a simple, managed solution that minimizes operational overhead. What should the team use?

Accepted Answer

Use an AWS Glue ETL job with a JDBC connection to the PostgreSQL database, extract data, and write to S3 in Parquet format.. AWS Glue ETL job is the best choice for this scenario. It is a fully managed service that can connect to on-premises PostgreSQL via a JDBC connection (using an AWS Glue connection with the appropriate network configuration over Direct Connect). It can perform a full extract (truncate and load) each day and write the data directly to S3 in Parquet format, minimizing operational overhead. Option A (AWS DMS) is designed for ongoing change data capture (CDC) and continuous replication, not for daily full refreshes without incremental changes. Option B (Amazon EMR with Spark) requires managing clusters and is more complex than necessary for a simple daily load. Option D (AWS Data Pipeline) requires custom scripting and more configuration compared to Glue's built-in ETL capabilities. Therefore, option C is correct.

Answer

Set up AWS Database Migration Service (DMS) to continuously replicate data to S3 in Parquet format.

Answer

Use Amazon EMR with a Spark job that reads from PostgreSQL and writes to S3.

Answer

Use AWS Data Pipeline with a SQLActivity to extract data and copy to S3.

Question 3

A logistics company ingests real-time GPS location data from thousands of delivery vehicles into Amazon Kinesis Data Streams. Each vehicle sends a JSON payload every 10 seconds containing vehicle_id, latitude, longitude, timestamp, and speed. The data must be stored in Amazon S3 for historical analysis, but the company wants to first aggregate the data per vehicle per minute (average speed, min/max coordinates) to reduce storage costs. The solution must be serverless and handle potential duplicate records without double-counting. What should the engineer do?

Accepted Answer

Use Amazon Kinesis Data Analytics for Apache Flink to aggregate data in a 1-minute tumbling window with deduplication logic, then output to Kinesis Data Firehose for delivery to S3.. Amazon Kinesis Data Analytics for Apache Flink can process streaming data with tumbling windows (1-minute) to aggregate per vehicle per minute, and Flink's stateful processing allows deduplication to avoid double-counting. The output is then sent to Kinesis Data Firehose for persistent storage in Amazon S3, all serverless. Option A (EMR with Spark Streaming) is not serverless. Option C (Kinesis Data Firehose with Lambda transformation) cannot perform stateful windowed aggregation and deduplication efficiently—it processes events individually without maintaining state across records. Option D (AWS Glue streaming ETL job) is serverless but adds higher latency and complexity compared to Kinesis Data Analytics, and is not as optimized for low-latency streaming aggregations with deduplication.

Answer

Use Amazon EMR with Spark Streaming to perform the aggregation and write to S3.

Answer

Use Kinesis Data Firehose with a Lambda transformation to aggregate records in a 1-minute window.

Answer

Use an AWS Glue streaming ETL job with Spark Structured Streaming to aggregate and deduplicate.

Question 4

A healthcare company is building a data pipeline to ingest electronic health records (EHR) from hospitals. The data is sent as JSON files via SFTP to an on-premises server. The company wants to move this data to AWS using AWS Transfer Family (SFTP) and then process it with AWS Glue. Data sovereignty regulations require that all data remain within the EU (Frankfurt) region. The pipeline must detect when a new file arrives and start the Glue job automatically. The engineer has set up an AWS Transfer Family server in Frankfurt, and files are uploaded to an S3 bucket in the same region. However, the Glue job is not triggering automatically. The engineer needs to implement automated triggering. What should the engineer do?

Accepted Answer

Set up an S3 event notification on the bucket to invoke an AWS Lambda function that starts the Glue job.. S3 event notifications can be configured to invoke an AWS Lambda function when a new object is created in the bucket, and the Lambda function can then start the AWS Glue job. This is the simplest and most direct method for automated triggering. Option A (Step Functions) adds unnecessary complexity and polling overhead. Option B (CloudWatch Events) cannot directly monitor S3 object creation events; it would require additional custom logic. Option C (SQS) is an unnecessary intermediary; S3 can directly invoke Lambda without needing a queue.

Answer

Configure AWS Step Functions to poll the S3 bucket every minute and start the Glue job if new files exist.

Answer

Configure Amazon CloudWatch Events to trigger the Glue job on a schedule that checks for new files.

Answer

Use Amazon Simple Queue Service (SQS) to queue file metadata and have a Lambda function poll the queue to start the Glue job.

Question 5

A data engineer needs to ingest log files from multiple EC2 instances into Amazon S3. The logs are written to local disk on each instance. The engineer wants a simple agent-based solution that can collect, compress, and upload logs to S3 with minimal configuration. The solution must support incremental uploads (only new log lines) and handle log rotation. What should the engineer use?

Accepted Answer

Install and configure Amazon CloudWatch Agent to collect logs and send them to Amazon CloudWatch Logs, then use a subscription filter to export logs to S3.. Amazon CloudWatch Agent is a lightweight agent that can tail log files, compress them on the fly, and send them to CloudWatch Logs. From CloudWatch Logs, a subscription filter can export the logs to Amazon S3, supporting incremental uploads and log rotation. Option B (AWS CLI cp) is manual and does not handle incremental uploads efficiently. Option C (AWS DataSync) is designed for bulk data transfers, not real-time log ingestion. Option D (S3 sync) is also not real-time and lacks agent-based tailing and compression.

Answer

Use AWS CLI cp command with --recursive in a cron job to copy logs to S3 every minute.

Answer

Install AWS DataSync agent on each EC2 instance to sync logs to S3 daily.

Answer

Use an S3 sync command from the AWS CLI scheduled every hour.

Question 6

A financial company needs to ingest real-time stock trade data from multiple sources and store it in Amazon S3 for compliance. The data must be delivered within 1 minute of the trade occurring. The data volume is approximately 10,000 records per second, with occasional spikes to 50,000 records per second. The engineer has set up Amazon Kinesis Data Streams with 10 shards and a Kinesis Data Firehose delivery stream that reads from the Kinesis stream and writes to S3. However, during spikes, the Firehose delivery stream falls behind, causing data to be delayed beyond the 1-minute SLA. What should the engineer do to meet the SLA without over-provisioning?

Accepted Answer

Increase the number of shards in the Kinesis data stream to handle peak throughput and enable auto-scaling.. Increasing the number of shards in the Kinesis data stream allows it to handle the peak throughput of 50,000 records per second, and enabling auto-scaling ensures the stream adapts to varying loads without manual intervention. This reduces the backlog that causes Firehose to fall behind, meeting the 1-minute SLA. Option A is incorrect because increasing the buffer size in Firehose would increase latency, not reduce it. Option B is incorrect because adding SQS as a buffer adds another hop, increasing latency and complexity. Option C is incorrect because AWS Lambda may not scale to 50,000 records per second and adds processing latency.

Answer

Increase the buffer size in Kinesis Data Firehose from 1 MB to 5 MB to batch more data per delivery.

Answer

Use Amazon SQS as a buffer between Kinesis and Firehose to absorb spikes.

Answer

Replace Firehose with an AWS Lambda function that writes directly to S3 for lower latency.

Question 7

A social media company ingests user activity data from multiple sources into Amazon S3. The data is in JSON format and includes fields: user_id, activity_type, timestamp, and metadata. The company wants to transform this data into a columnar format (Parquet) partitioned by date and activity_type for efficient querying with Amazon Athena. The pipeline must handle data that arrives up to 3 days late. Currently, a daily AWS Glue ETL job scans the entire S3 bucket for new files, transforms them, and writes to a separate output bucket. The job is taking longer as data volume grows, and the team wants to reduce processing time and cost. What should the engineer do?

Accepted Answer

Set up S3 event notifications to invoke an AWS Lambda function that triggers a Glue job for each new object, passing the object key so the job processes only that file.. Using S3 event notifications with Lambda to trigger a Glue job for each new file allows incremental processing, reducing the time and cost of scanning the entire S3 bucket. Option A (increasing DPUs) does not address the root cause of scanning all files. Option B (partition projection) helps with query performance but not with the transformation process. Option C (replacing Glue with EMR) adds operational overhead and is not necessary for this use case.

Answer

Increase the number of DPUs for the Glue job to process data faster.

Answer

Use AWS Glue partition projection and schema inference to reduce scan time.

Answer

Replace AWS Glue with Amazon EMR and use Spark to process data in parallel.

Question 8

A gaming company ingests player event data from mobile games into Amazon Kinesis Data Streams. Each event is a small JSON payload (<1 KB). The data must be delivered to Amazon S3 for analytics, and the company wants to minimize storage costs by aggregating events into larger files (e.g., 100 MB per file). The current setup uses Kinesis Data Firehose with a buffer size of 10 MB and a buffer interval of 60 seconds, but the resulting files are very small (average 5 MB) because the data volume is low. The engineer needs to ensure that files are at least 100 MB to reduce the number of S3 objects and lower costs. What should the engineer do?

Accepted Answer

Increase the buffer size in Kinesis Data Firehose to 100 MB and increase the buffer interval to 300 seconds.. Increase the buffer size in Kinesis Data Firehose to 100 MB and increase the buffer interval to 300 seconds. This allows more data to accumulate before writing to S3, resulting in larger files (≥100 MB) and reducing the number of S3 objects, thus lowering storage costs. Option B (Lambda) would require custom buffering logic and may encounter timeouts or memory limits. Option A (Glue streaming ETL) adds latency and cost without directly controlling file size. Option D (EMR Spark Streaming) is overkill for this simple aggregation need.

Answer

Use an AWS Glue streaming ETL job with a 100 MB file size threshold to write to S3.

Answer

Use an AWS Lambda function to buffer events in memory and write to S3 when buffer reaches 100 MB.

Answer

Use Amazon EMR with Spark Streaming to aggregate and write larger files to S3.

Question 9

A data engineering team is ingesting streaming data from IoT devices into Amazon Kinesis Data Streams. The data must be transformed in real-time and then loaded into an Amazon S3 bucket for long-term storage. Which AWS service should be used to perform the transformation and delivery to S3 with minimal operational overhead?

Accepted Answer

Amazon Kinesis Data Firehose. Amazon Kinesis Data Firehose is the correct choice because it is a fully managed service designed to automatically load streaming data into Amazon S3, Redshift, Elasticsearch, and Splunk. It can invoke an AWS Lambda function for real-time data transformation before delivery, eliminating the need to manage any infrastructure or write custom code for the delivery pipeline. This directly meets the requirement of minimal operational overhead for transformation and S3 delivery.

Answer

AWS Glue

Answer

Amazon EMR

Answer

Amazon Kinesis Data Analytics

Question 10

A company uses AWS DMS to migrate a 2 TB PostgreSQL database to Amazon Aurora PostgreSQL. The migration is taking longer than expected due to the initial load. Which AWS service can be used to accelerate the initial load by transferring the database files directly?

Accepted Answer

AWS Snowball. AWS Snowball is a petabyte-scale data transport solution that uses physical storage devices to transfer large amounts of data into and out of AWS. For a 2 TB PostgreSQL database, the initial load via DMS over the network can be slow due to bandwidth constraints, especially for large datasets. By using Snowball, you can export the database files (e.g., using pg_dump or physical file copy) to the device, ship it to AWS, and have the data loaded directly into Amazon Aurora PostgreSQL, bypassing the network bottleneck and significantly accelerating the initial load.

Answer

Amazon S3 Transfer Acceleration

Answer

AWS Direct Connect

Answer

Amazon Kinesis Data Firehose

Question 11

A data pipeline ingests daily CSV files from an FTP server into an Amazon S3 bucket. The files must be converted to Parquet format and partitioned by date for efficient querying using Amazon Athena. Which AWS service is most suitable for this transformation?

Accepted Answer

AWS Glue. AWS Glue is the most suitable service because it provides a fully managed ETL (Extract, Transform, Load) capability that can natively read CSV files from S3, convert them to Parquet format, and write the output partitioned by date. Glue's built-in transform 'ConvertToParquet' and dynamic frame partitioning make this a straightforward, serverless solution without needing to manage infrastructure.

Answer

Amazon Kinesis Data Firehose

Answer

Amazon EMR

Answer

AWS Lambda

Question 12

A data engineering team is ingesting data from multiple sources into Amazon S3 using AWS Glue ETL jobs. The jobs are failing intermittently with the error: 'Task ran out of memory'. The input data size varies widely from 100 MB to 10 GB per job. Which configuration change would best mitigate this issue?

Accepted Answer

Enable job bookmarking to process only incremental data. Enabling job bookmarking allows the Glue ETL job to process only incremental (new or changed) data, which directly addresses the intermittent out-of-memory errors caused by widely varying input sizes (100 MB to 10 GB). By skipping previously processed data, the job consistently handles smaller data volumes per run, reducing memory pressure on the Spark executors.

Answer

Increase the number of workers in the Glue job

Answer

Reduce the batch size in the S3 source node

Answer

Change the job type from Spark to Python shell

Question 13

A company wants to ingest real-time clickstream data from a website into Amazon S3 with a maximum latency of 60 seconds. The data volume peaks at 500 MB/s. Which service should they use to buffer and deliver the data to S3?

Accepted Answer

Amazon Kinesis Data Firehose. Amazon Kinesis Data Firehose is the correct choice because it is designed to ingest streaming data, buffer it, and deliver it to destinations like Amazon S3 with configurable buffer intervals (e.g., 60 seconds) and buffer sizes (e.g., up to 128 MB). It can handle the peak throughput of 500 MB/s by automatically scaling, and it meets the maximum latency requirement of 60 seconds by flushing data to S3 based on time or size thresholds.

Answer

Amazon Simple Queue Service (SQS)

Answer

Amazon Kinesis Data Streams

Answer

AWS Lambda

Question 14

A data engineer is designing a pipeline that ingests JSON logs from an application into Amazon S3. The logs contain a timestamp field. The pipeline must partition the data by date in S3 (e.g., year=2024/month=10/day=01). Which approach minimizes transformation effort?

Accepted Answer

Use Amazon Kinesis Data Firehose with dynamic partitioning. Amazon Kinesis Data Firehose with dynamic partitioning can automatically partition incoming JSON data based on the timestamp field without requiring custom transformation code. It evaluates the timestamp using a JQ expression or inline parsing, then writes records directly to S3 prefixes like year=2024/month=10/day=01. This minimizes transformation effort because the partitioning logic is configured declaratively in the Firehose delivery stream, eliminating the need for Lambda functions or post-ingestion processing.

Answer

Use AWS Glue crawlers to infer schema and create partitions

Answer

Use AWS Lambda to process each object and copy to the appropriate prefix

Answer

Use Amazon Athena to create partitions on the existing data

Question 15

A company is using AWS DMS to replicate data from an on-premises Oracle database to Amazon RDS for MySQL. The replication is working, but the target table has a different schema. Which DMS feature should be used to transform the source schema to match the target?

Accepted Answer

Use DMS transformation rules. AWS DMS transformation rules allow you to modify the schema, table, or column names and data types during the migration process. This feature is specifically designed to handle schema transformations within the DMS task itself, enabling you to map the source Oracle schema to the target MySQL schema without external tools or services.

Answer

Use AWS Schema Conversion Tool (SCT)

Answer

Use AWS Glue ETL jobs

Answer

Use AWS Lambda triggers