DEA-C01 Data Ingestion and Transformation — Timed 50 Questions

Question 1

A data engineer needs to ingest streaming data from an IoT fleet into Amazon S3 for near-real-time analytics. The data volume is approximately 5 GB per hour, and each event is less than 1 KB. Which AWS service should be used as the ingestion endpoint?

Accepted Answer

AWS IoT Core. AWS IoT Core is purpose-built for ingesting data from IoT devices, supporting MQTT, HTTP, and WebSocket protocols. It can handle millions of devices and high-throughput, small-message payloads (each event <1 KB) and integrates directly with Amazon S3 via IoT Core rules, making it the ideal ingestion endpoint for near-real-time analytics on streaming IoT data.

Answer

AWS DataSync

Answer

Amazon AppFlow

Answer

Amazon Kinesis Data Streams

Question 2

A company uses AWS Glue ETL jobs to transform data from Amazon S3 to Amazon Redshift. The job reads JSON files, applies schema mapping, and writes to a Redshift table. Recently, the job started failing with memory errors. The data volume has increased tenfold. Which approach should a data engineer take to resolve this issue with minimal code changes?

Accepted Answer

Increase the number of DPUs allocated to the Glue job.. Option C is correct because increasing the number of DPUs (Data Processing Units) allocated to the AWS Glue job directly addresses the memory constraint caused by a tenfold increase in data volume. Glue ETL jobs run on Apache Spark, which distributes data processing across executors; more DPUs provide more memory and compute capacity, allowing the job to handle larger datasets without code changes.

Answer

Switch from Spark to Python Shell job type.

Answer

Implement batch processing with smaller file sizes.

Answer

Use Redshift Spectrum to query data directly from S3.

Question 3

A financial services company processes real-time stock trade data. They use Amazon Kinesis Data Streams with a shard count of 5, each shard receiving about 500 records per second. The consumer application uses the Kinesis Client Library (KCL) with DynamoDB for checkpointing. Lately, some records are being processed multiple times. What is the most likely cause?

Accepted Answer

The consumer application is crashing and restarting, causing re-processing of records.. The Kinesis Client Library (KCL) uses DynamoDB to track checkpoint progress for each shard. If the consumer application crashes and restarts, the KCL will resume processing from the last committed checkpoint, which may be behind the actual processing point. This causes records that were already processed (but not yet checkpointed) to be re-processed, leading to duplicate processing.

Answer

The Kinesis stream's iterator age is exceeding the retention period.

Answer

The DynamoDB table used for checkpointing is throttling write requests.

Answer

The record size exceeds the 1 MB API limit, causing retries.

Question 4

A data engineering team needs to transform CSV files stored in Amazon S3 into Parquet format using AWS Glue. The files are partitioned by date and are updated hourly. Which AWS Glue feature should be used to automatically detect the schema and partition structure?

Accepted Answer

AWS Glue Crawler. AWS Glue Crawler is the correct choice because it automatically scans data in S3, infers the schema (including data types), and detects the partition structure (e.g., date-based partitions like year/month/day) by examining the folder hierarchy. It then populates the AWS Glue Data Catalog with metadata, enabling ETL jobs to read the data without manual schema definition.

Answer

AWS Glue DataBrew

Answer

AWS Lake Formation

Answer

Amazon Athena

Question 5

An e-commerce company ingests clickstream data from their website into Amazon S3. The data is in JSON format, and each file is about 10 MB. They need to transform the data into a columnar format for analytics and load it into Amazon Redshift nightly. The transformation should be cost-effective and require minimal operational overhead. Which approach meets these requirements?

Accepted Answer

Use AWS Glue ETL job to convert to Parquet and load into Redshift.. AWS Glue ETL is the correct choice because it is a serverless, managed service that can efficiently convert JSON to Parquet (a columnar format optimized for Redshift) and load the data into Redshift with minimal operational overhead. The nightly batch processing of 10 MB files is well-suited for Glue's pay-per-use pricing, making it cost-effective without requiring infrastructure management.

Answer

Use Amazon Redshift COPY command to load JSON directly.

Answer

Use Amazon EMR with Spark to transform and load data.

Answer

Use AWS Lambda to transform each file and write to Redshift.

Question 6

A company uses AWS Database Migration Service (DMS) to continuously replicate data from an on-premises Oracle database to Amazon S3 in Parquet format. The replication is used for near-real-time analytics. Recently, the DMS task started failing with an error indicating insufficient memory. The source database is large (2 TB). What should a data engineer do to resolve this issue while minimizing changes to the existing architecture?

Accepted Answer

Increase the DMS replication instance size.. The error indicates the DMS replication instance is running out of memory during continuous replication of a 2 TB Oracle database to S3 in Parquet format. Increasing the replication instance size (Option D) directly addresses the memory constraint by providing more RAM and processing capacity, which is necessary for handling large volumes of Change Data Capture (CDC) data and Parquet conversion overhead. This solution requires minimal architectural changes, as it only involves modifying the instance class in the DMS task settings.

Answer

Change the target format to JSON to reduce memory usage.

Answer

Split the DMS task into multiple smaller tasks.

Answer

Use Change Data Capture (CDC) only, without full load.

Question 7

A data engineer needs to ingest data from multiple SaaS applications (Salesforce, Marketo) into Amazon S3 for a data lake. The data volumes are moderate and the sync needs to be scheduled daily. Which AWS service is most appropriate for this task?

Accepted Answer

Amazon AppFlow. Amazon AppFlow is purpose-built for securely transferring data between SaaS applications (like Salesforce and Marketo) and AWS services (like S3). It supports scheduled, incremental data syncs with built-in connectors, making it the most appropriate choice for moderate-volume daily ingestion into a data lake.

Answer

AWS Glue

Answer

AWS Database Migration Service (DMS)

Answer

Amazon Kinesis Data Firehose

Question 8

A company uses AWS Lambda to process records from an Amazon Kinesis Data Stream. Each record is about 50 KB. The Lambda function transforms the data and writes to Amazon DynamoDB. Recently, the Lambda function has been experiencing throttling and high error rates. The Kinesis stream has 10 shards. What is the most cost-effective solution to improve processing throughput?

Accepted Answer

Increase the Batch Window (MaximumBatchingWindowInSeconds) for the event source mapping.. Option D is correct because increasing the Batch Window (MaximumBatchingWindowInSeconds) allows the Lambda function to accumulate more records from the Kinesis stream before invoking the function, reducing the number of invocations and thus lowering the chance of throttling. This is the most cost-effective solution as it does not require additional shards, memory, or parallelization, and directly addresses the high error rates caused by excessive concurrent executions.

Answer

Increase the number of shards in the Kinesis stream.

Answer

Increase the Parallelization Factor for the Lambda event source mapping.

Answer

Increase the memory allocated to the Lambda function.

Question 9

A data engineer is designing a data ingestion pipeline for a social media analytics platform. The pipeline must ingest tweets in real-time, perform sentiment analysis, and store results in Amazon S3. The sentiment analysis is compute-intensive and must be done as the data arrives. The estimated throughput is 10,000 tweets per second. Which architecture is most suitable?

Accepted Answer

Amazon Kinesis Data Streams with Amazon Kinesis Data Analytics for sentiment analysis, then Kinesis Data Firehose to S3.. Option C is the most suitable because Amazon Kinesis Data Streams can ingest up to 10,000 records per second per shard (with shard-level scaling), and Kinesis Data Analytics provides built-in, low-latency stream processing for compute-intensive sentiment analysis using SQL or Apache Flink. Kinesis Data Firehose then reliably buffers and writes the processed results to Amazon S3 without custom code, ensuring near-real-time delivery.

Answer

Amazon SQS with AWS Lambda pollers to process tweets and store in S3.

Answer

Amazon EMR with Spark Streaming to process tweets and write to S3.

Answer

Amazon API Gateway with AWS Lambda to process each tweet and store in S3.

Question 10

A data engineer needs to transfer 50 TB of historical data from an on-premises HDFS cluster to Amazon S3. The network bandwidth is limited to 100 Mbps. The transfer must be completed within one week. Which service should be used?

Accepted Answer

AWS Snowball. AWS Snowball is the correct choice because transferring 50 TB over a 100 Mbps network would take approximately 46 days (50 TB * 8 bits/byte / 100 Mbps / 86400 seconds/day), far exceeding the one-week deadline. Snowball provides a physical storage device that can be shipped to the on-premises location, allowing data to be loaded locally and shipped to AWS, bypassing network bandwidth constraints entirely.

Answer

AWS DataSync

Answer

Amazon CloudFront

Answer

AWS Database Migration Service (DMS)

Question 11

A company uses Amazon Kinesis Data Firehose to deliver streaming data to Amazon S3. The data is in JSON format. The delivery stream is configured with a buffer size of 5 MB and a buffer interval of 60 seconds. However, the data engineer notices that S3 objects are being created with sizes much smaller than 5 MB. What is a likely cause?

Accepted Answer

The incoming data rate is too low, causing the buffer interval to trigger before reaching the buffer size.. Option B is correct because Kinesis Data Firehose delivers data to S3 when either the buffer size (5 MB) or the buffer interval (60 seconds) is reached, whichever occurs first. If the incoming data rate is low, the buffer interval will expire before accumulating 5 MB of data, resulting in smaller S3 objects.

Answer

The data is being compressed before delivery, reducing object size.

Answer

The data transformation lambda is splitting records into smaller ones.

Answer

The S3 bucket is configured with a lifecycle policy that splits objects.

Question 12

A data engineering team is ingesting streaming data from IoT devices into Amazon Kinesis Data Streams. The data is then consumed by an AWS Lambda function that transforms each record and writes it to Amazon S3. Recently, the Lambda function started failing with 'ProvisionedThroughputExceededException' errors when writing to S3. The team has already increased the Lambda function's memory and timeout. Which action should the team take to resolve the issue?

Accepted Answer

Implement retries with exponential backoff in the Lambda function for S3 put operations.. The 'ProvisionedThroughputExceededException' error indicates that the Lambda function is being throttled by S3 due to exceeding the bucket's request rate limits. Implementing retries with exponential backoff in the Lambda function for S3 put operations is the correct solution because it allows the function to gracefully handle transient throttling errors by waiting progressively longer between retries, which aligns with AWS's guidance for managing S3 request rate limits.

Answer

Use S3 Batch Operations to write data in batches.

Answer

Increase the number of shards in the Kinesis data stream.

Answer

Enable S3 Transfer Acceleration on the destination bucket.

Question 13

An e-commerce company uses AWS Glue to run ETL jobs that transform clickstream data from Amazon S3. The job reads Parquet files, performs aggregations, and writes the results to Amazon Redshift. The job runs successfully but takes longer than expected. The data volume is increasing. Which design change would MOST improve the job's performance?

Accepted Answer

Increase the number of Glue worker nodes (DPUs) for the job.. Increasing the number of Glue worker nodes (DPUs) directly scales the distributed processing capacity of the ETL job, allowing it to process larger volumes of Parquet data in parallel. This is the most straightforward way to reduce execution time when data volume is growing, as AWS Glue automatically partitions the workload across the additional workers.

Answer

Write the aggregated results to a single large file instead of multiple partitions.

Answer

Convert the Parquet files to CSV to simplify the schema.

Answer

Replace the Redshift target with Amazon Redshift Spectrum.

Question 14

A data engineer needs to ingest JSON data from an on-premises relational database into Amazon S3 every hour. Which AWS service should be used to set up a scheduled, incremental data transfer?

Accepted Answer

AWS Database Migration Service (DMS) with S3 as target.. AWS DMS is purpose-built for migrating databases to AWS targets, including Amazon S3. It supports ongoing replication (change data capture) and scheduled full-load tasks, making it ideal for hourly incremental transfers from an on-premises relational database to S3 without custom scripting.

Answer

Amazon S3 Transfer Acceleration with a cron job.

Answer

AWS Glue with a JDBC connection and a scheduled crawler.

Answer

Amazon Kinesis Data Firehose with a database source.

Question 15

A company is using AWS Glue to process data from Amazon S3. The Glue job reads CSV files and writes Parquet files to a different S3 bucket. The job occasionally fails with 'java.lang.OutOfMemoryError: Java heap space'. The data size varies. Which change should the engineer make to avoid this error?

Accepted Answer

Increase the number of DPUs allocated to the Glue job.. The 'java.lang.OutOfMemoryError: Java heap space' error in AWS Glue indicates that the Spark executors ran out of memory while processing the data. Increasing the number of DPUs (Data Processing Units) allocated to the Glue job increases the total memory available across the cluster, allowing larger datasets to be processed without hitting the heap limit. Each DPU provides 4 vCPUs and 16 GB of memory, so adding more DPUs scales memory linearly.

Answer

Convert the CSV files to JSON format before processing.

Answer

Decrease the Spark shuffle partitions in the job script.

Answer

Increase the job timeout setting.

Question 16

A data engineering team uses Amazon Kinesis Data Analytics for Apache Flink to process streaming data. They notice that the application's checkpointing is failing intermittently, causing data reprocessing. The application uses a large state. Which configuration change should the team make to improve checkpoint reliability?

Accepted Answer

Increase the checkpointing interval.. Increasing the checkpointing interval reduces the frequency of checkpoint operations, giving the system more time to complete each checkpoint before the next one starts. This alleviates backpressure and resource contention, which is critical when dealing with large state, as checkpointing large state is I/O and CPU intensive and can fail if intervals are too tight.

Answer

Disable checkpointing to avoid failures.

Answer

Switch the state backend from in-memory to RocksDB.

Answer

Increase the parallelism of the application.

Question 17

A data engineer is designing a serverless data ingestion pipeline that uses Amazon Kinesis Data Firehose to deliver data to Amazon S3. The data must be transformed using AWS Lambda before being written to S3. Which two steps are required to enable this transformation? (Select TWO.)

Accepted Answer

Configure a Lambda function as a data transformation source in the Firehose delivery stream.. Option B is correct because Amazon Kinesis Data Firehose can be configured to invoke a Lambda function as a data transformation source. This allows Firehose to pass incoming records to the Lambda function, which processes and returns the transformed records before they are delivered to the S3 destination. Option C is correct because the Lambda function must return data in the specific format that Firehose expects, including a record ID, result status, and base64-encoded data, otherwise the transformation will fail.

Answer

Set up an S3 event notification to trigger the Lambda function on object creation.

Answer

Subscribe the Lambda function to the CloudWatch Logs log group for the Firehose stream.

Answer

Have the Lambda function write the transformed data directly to the S3 bucket.

Question 18

A company uses AWS Glue to perform ETL on data stored in Amazon S3. The Glue job reads CSV files, converts them to Parquet, and partitions by date. The job runs daily and processes about 500 GB of data. The team wants to optimize costs and performance. Which three actions should the team take? (Select THREE.)

Accepted Answer

Use column pruning to read only necessary columns in the Glue script.. Option B is correct because column pruning in AWS Glue scripts reduces the amount of data read from Amazon S3 by specifying only the columns needed for the ETL transformation. This minimizes I/O and network overhead, directly lowering costs and improving job performance, especially when processing large CSV files.

Answer

Increase the Spark shuffle partitions to 500.

Answer

Write the output as JSON instead of Parquet to avoid compression overhead.

Question 19

A company uses AWS Glue to process streaming data from Amazon Kinesis Data Streams. The job reads JSON records and writes Parquet to Amazon S3. Recently, the job started failing with 'Out of Memory' errors. Which change is MOST likely to resolve the issue?

Accepted Answer

Increase the number of DPUs allocated to the Glue job.. The 'Out of Memory' error in AWS Glue indicates that the job's allocated resources are insufficient for the data volume or processing complexity. Increasing the number of DPUs (Data Processing Units) directly increases the available memory and compute capacity, which is the most straightforward fix for OOM errors in Glue streaming jobs. Option C is correct because it addresses the root cause—resource exhaustion—by scaling the job horizontally.

Answer

Enable compression on the Kinesis stream.

Answer

Change the output format from Parquet to ORC.

Answer

Reduce the streaming batch size in the Glue job configuration.

Question 20

A data engineer is designing a data ingestion pipeline for IoT sensor data. The data arrives as JSON via AWS IoT Core, and must be stored in Amazon S3 in partitioned Parquet format. The pipeline must handle late-arriving data (up to 1 hour) and ensure exactly-once processing. Which combination of services should the engineer use?

Accepted Answer

Amazon Kinesis Data Firehose with data transformation via AWS Lambda, delivering to Amazon S3.. Amazon Kinesis Data Firehose is the correct choice because it can directly ingest streaming data from AWS IoT Core, use a built-in AWS Lambda function to transform JSON to Parquet, and deliver the data to Amazon S3 with automatic partitioning. It also supports buffering and retry logic to handle late-arriving data (up to 1 hour) and provides exactly-once delivery to S3 when configured with the appropriate error handling and idempotent transformations.

Answer

Amazon Kinesis Data Streams with AWS Lambda for transformation and Amazon S3.

Answer

Amazon Simple Queue Service (SQS) with AWS Lambda for transformation and Amazon S3.

Answer

AWS Glue streaming jobs consuming from Amazon Kinesis Data Streams and writing to Amazon S3.