DEA-C01 Data Ingestion and Transformation — 15 Questions

Question 1

A data engineer needs to ingest streaming data from an IoT fleet into Amazon S3 for near-real-time analytics. The data volume is approximately 5 GB per hour, and each event is less than 1 KB. Which AWS service should be used as the ingestion endpoint?

Accepted Answer

AWS IoT Core. AWS IoT Core is purpose-built for ingesting data from IoT devices, supporting MQTT, HTTP, and WebSocket protocols. It can handle millions of devices and high-throughput, small-message payloads (each event <1 KB) and integrates directly with Amazon S3 via IoT Core rules, making it the ideal ingestion endpoint for near-real-time analytics on streaming IoT data.

Answer

AWS DataSync

Answer

Amazon AppFlow

Answer

Amazon Kinesis Data Streams

Question 2

A company uses AWS Glue ETL jobs to transform data from Amazon S3 to Amazon Redshift. The job reads JSON files, applies schema mapping, and writes to a Redshift table. Recently, the job started failing with memory errors. The data volume has increased tenfold. Which approach should a data engineer take to resolve this issue with minimal code changes?

Accepted Answer

Increase the number of DPUs allocated to the Glue job.. Option C is correct because increasing the number of DPUs (Data Processing Units) allocated to the AWS Glue job directly addresses the memory constraint caused by a tenfold increase in data volume. Glue ETL jobs run on Apache Spark, which distributes data processing across executors; more DPUs provide more memory and compute capacity, allowing the job to handle larger datasets without code changes.

Answer

Switch from Spark to Python Shell job type.

Answer

Implement batch processing with smaller file sizes.

Answer

Use Redshift Spectrum to query data directly from S3.

Question 3

A financial services company processes real-time stock trade data. They use Amazon Kinesis Data Streams with a shard count of 5, each shard receiving about 500 records per second. The consumer application uses the Kinesis Client Library (KCL) with DynamoDB for checkpointing. Lately, some records are being processed multiple times. What is the most likely cause?

Accepted Answer

The consumer application is crashing and restarting, causing re-processing of records.. The Kinesis Client Library (KCL) uses DynamoDB to track checkpoint progress for each shard. If the consumer application crashes and restarts, the KCL will resume processing from the last committed checkpoint, which may be behind the actual processing point. This causes records that were already processed (but not yet checkpointed) to be re-processed, leading to duplicate processing.

Answer

The Kinesis stream's iterator age is exceeding the retention period.

Answer

The DynamoDB table used for checkpointing is throttling write requests.

Answer

The record size exceeds the 1 MB API limit, causing retries.

Question 4

A data engineering team needs to transform CSV files stored in Amazon S3 into Parquet format using AWS Glue. The files are partitioned by date and are updated hourly. Which AWS Glue feature should be used to automatically detect the schema and partition structure?

Accepted Answer

AWS Glue Crawler. AWS Glue Crawler is the correct choice because it automatically scans data in S3, infers the schema (including data types), and detects the partition structure (e.g., date-based partitions like year/month/day) by examining the folder hierarchy. It then populates the AWS Glue Data Catalog with metadata, enabling ETL jobs to read the data without manual schema definition.

Answer

AWS Glue DataBrew

Answer

AWS Lake Formation

Answer

Amazon Athena

Question 5

An e-commerce company ingests clickstream data from their website into Amazon S3. The data is in JSON format, and each file is about 10 MB. They need to transform the data into a columnar format for analytics and load it into Amazon Redshift nightly. The transformation should be cost-effective and require minimal operational overhead. Which approach meets these requirements?

Accepted Answer

Use AWS Glue ETL job to convert to Parquet and load into Redshift.. AWS Glue ETL is the correct choice because it is a serverless, managed service that can efficiently convert JSON to Parquet (a columnar format optimized for Redshift) and load the data into Redshift with minimal operational overhead. The nightly batch processing of 10 MB files is well-suited for Glue's pay-per-use pricing, making it cost-effective without requiring infrastructure management.

Answer

Use Amazon Redshift COPY command to load JSON directly.

Answer

Use Amazon EMR with Spark to transform and load data.

Answer

Use AWS Lambda to transform each file and write to Redshift.

Question 6

A company uses AWS Database Migration Service (DMS) to continuously replicate data from an on-premises Oracle database to Amazon S3 in Parquet format. The replication is used for near-real-time analytics. Recently, the DMS task started failing with an error indicating insufficient memory. The source database is large (2 TB). What should a data engineer do to resolve this issue while minimizing changes to the existing architecture?

Accepted Answer

Increase the DMS replication instance size.. The error indicates the DMS replication instance is running out of memory during continuous replication of a 2 TB Oracle database to S3 in Parquet format. Increasing the replication instance size (Option D) directly addresses the memory constraint by providing more RAM and processing capacity, which is necessary for handling large volumes of Change Data Capture (CDC) data and Parquet conversion overhead. This solution requires minimal architectural changes, as it only involves modifying the instance class in the DMS task settings.

Answer

Change the target format to JSON to reduce memory usage.

Answer

Split the DMS task into multiple smaller tasks.

Answer

Use Change Data Capture (CDC) only, without full load.

Question 7

A data engineer needs to ingest data from multiple SaaS applications (Salesforce, Marketo) into Amazon S3 for a data lake. The data volumes are moderate and the sync needs to be scheduled daily. Which AWS service is most appropriate for this task?

Accepted Answer

Amazon AppFlow. Amazon AppFlow is purpose-built for securely transferring data between SaaS applications (like Salesforce and Marketo) and AWS services (like S3). It supports scheduled, incremental data syncs with built-in connectors, making it the most appropriate choice for moderate-volume daily ingestion into a data lake.

Answer

AWS Glue

Answer

AWS Database Migration Service (DMS)

Answer

Amazon Kinesis Data Firehose

Question 8

A company uses AWS Lambda to process records from an Amazon Kinesis Data Stream. Each record is about 50 KB. The Lambda function transforms the data and writes to Amazon DynamoDB. Recently, the Lambda function has been experiencing throttling and high error rates. The Kinesis stream has 10 shards. What is the most cost-effective solution to improve processing throughput?

Accepted Answer

Increase the Batch Window (MaximumBatchingWindowInSeconds) for the event source mapping.. Option D is correct because increasing the Batch Window (MaximumBatchingWindowInSeconds) allows the Lambda function to accumulate more records from the Kinesis stream before invoking the function, reducing the number of invocations and thus lowering the chance of throttling. This is the most cost-effective solution as it does not require additional shards, memory, or parallelization, and directly addresses the high error rates caused by excessive concurrent executions.

Answer

Increase the number of shards in the Kinesis stream.

Answer

Increase the Parallelization Factor for the Lambda event source mapping.

Answer

Increase the memory allocated to the Lambda function.

Question 9

A data engineer is designing a data ingestion pipeline for a social media analytics platform. The pipeline must ingest tweets in real-time, perform sentiment analysis, and store results in Amazon S3. The sentiment analysis is compute-intensive and must be done as the data arrives. The estimated throughput is 10,000 tweets per second. Which architecture is most suitable?

Accepted Answer

Amazon Kinesis Data Streams with Amazon Kinesis Data Analytics for sentiment analysis, then Kinesis Data Firehose to S3.. Option C is the most suitable because Amazon Kinesis Data Streams can ingest up to 10,000 records per second per shard (with shard-level scaling), and Kinesis Data Analytics provides built-in, low-latency stream processing for compute-intensive sentiment analysis using SQL or Apache Flink. Kinesis Data Firehose then reliably buffers and writes the processed results to Amazon S3 without custom code, ensuring near-real-time delivery.

Answer

Amazon SQS with AWS Lambda pollers to process tweets and store in S3.

Answer

Amazon EMR with Spark Streaming to process tweets and write to S3.

Answer

Amazon API Gateway with AWS Lambda to process each tweet and store in S3.

Question 10

A data engineer needs to transfer 50 TB of historical data from an on-premises HDFS cluster to Amazon S3. The network bandwidth is limited to 100 Mbps. The transfer must be completed within one week. Which service should be used?

Accepted Answer

AWS Snowball. AWS Snowball is the correct choice because transferring 50 TB over a 100 Mbps network would take approximately 46 days (50 TB * 8 bits/byte / 100 Mbps / 86400 seconds/day), far exceeding the one-week deadline. Snowball provides a physical storage device that can be shipped to the on-premises location, allowing data to be loaded locally and shipped to AWS, bypassing network bandwidth constraints entirely.

Answer

AWS DataSync

Answer

Amazon CloudFront

Answer

AWS Database Migration Service (DMS)

Question 11

A company uses Amazon Kinesis Data Firehose to deliver streaming data to Amazon S3. The data is in JSON format. The delivery stream is configured with a buffer size of 5 MB and a buffer interval of 60 seconds. However, the data engineer notices that S3 objects are being created with sizes much smaller than 5 MB. What is a likely cause?

Accepted Answer

The incoming data rate is too low, causing the buffer interval to trigger before reaching the buffer size.. Option B is correct because Kinesis Data Firehose delivers data to S3 when either the buffer size (5 MB) or the buffer interval (60 seconds) is reached, whichever occurs first. If the incoming data rate is low, the buffer interval will expire before accumulating 5 MB of data, resulting in smaller S3 objects.

Answer

The data is being compressed before delivery, reducing object size.

Answer

The data transformation lambda is splitting records into smaller ones.

Answer

The S3 bucket is configured with a lifecycle policy that splits objects.

Question 12

A data engineering team is ingesting streaming data from IoT devices into Amazon Kinesis Data Streams. The data is then consumed by an AWS Lambda function that transforms each record and writes it to Amazon S3. Recently, the Lambda function started failing with 'ProvisionedThroughputExceededException' errors when writing to S3. The team has already increased the Lambda function's memory and timeout. Which action should the team take to resolve the issue?

Accepted Answer

Implement retries with exponential backoff in the Lambda function for S3 put operations.. The 'ProvisionedThroughputExceededException' error indicates that the Lambda function is being throttled by S3 due to exceeding the bucket's request rate limits. Implementing retries with exponential backoff in the Lambda function for S3 put operations is the correct solution because it allows the function to gracefully handle transient throttling errors by waiting progressively longer between retries, which aligns with AWS's guidance for managing S3 request rate limits.

Answer

Use S3 Batch Operations to write data in batches.

Answer

Increase the number of shards in the Kinesis data stream.

Answer

Enable S3 Transfer Acceleration on the destination bucket.

Question 13

An e-commerce company uses AWS Glue to run ETL jobs that transform clickstream data from Amazon S3. The job reads Parquet files, performs aggregations, and writes the results to Amazon Redshift. The job runs successfully but takes longer than expected. The data volume is increasing. Which design change would MOST improve the job's performance?

Accepted Answer

Increase the number of Glue worker nodes (DPUs) for the job.. Increasing the number of Glue worker nodes (DPUs) directly scales the distributed processing capacity of the ETL job, allowing it to process larger volumes of Parquet data in parallel. This is the most straightforward way to reduce execution time when data volume is growing, as AWS Glue automatically partitions the workload across the additional workers.

Answer

Write the aggregated results to a single large file instead of multiple partitions.

Answer

Convert the Parquet files to CSV to simplify the schema.

Answer

Replace the Redshift target with Amazon Redshift Spectrum.

Question 14

A data engineer needs to ingest JSON data from an on-premises relational database into Amazon S3 every hour. Which AWS service should be used to set up a scheduled, incremental data transfer?

Accepted Answer

AWS Database Migration Service (DMS) with S3 as target.. AWS DMS is purpose-built for migrating databases to AWS targets, including Amazon S3. It supports ongoing replication (change data capture) and scheduled full-load tasks, making it ideal for hourly incremental transfers from an on-premises relational database to S3 without custom scripting.

Answer

Amazon S3 Transfer Acceleration with a cron job.

Answer

AWS Glue with a JDBC connection and a scheduled crawler.

Answer

Amazon Kinesis Data Firehose with a database source.

Question 15

A company is using AWS Glue to process data from Amazon S3. The Glue job reads CSV files and writes Parquet files to a different S3 bucket. The job occasionally fails with 'java.lang.OutOfMemoryError: Java heap space'. The data size varies. Which change should the engineer make to avoid this error?

Accepted Answer

Increase the number of DPUs allocated to the Glue job.. The 'java.lang.OutOfMemoryError: Java heap space' error in AWS Glue indicates that the Spark executors ran out of memory while processing the data. Increasing the number of DPUs (Data Processing Units) allocated to the Glue job increases the total memory available across the cluster, allowing larger datasets to be processed without hitting the heap limit. Each DPU provides 4 vCPUs and 16 GB of memory, so adding more DPUs scales memory linearly.

Answer

Convert the CSV files to JSON format before processing.

Answer

Decrease the Spark shuffle partitions in the job script.

Answer

Increase the job timeout setting.