DEA-C01 Data Ingestion and Transformation — 10 Questions

Question 1

A data engineer needs to ingest streaming data from an IoT fleet into Amazon S3 for near-real-time analytics. The data volume is approximately 5 GB per hour, and each event is less than 1 KB. Which AWS service should be used as the ingestion endpoint?

Accepted Answer

AWS IoT Core. AWS IoT Core is purpose-built for ingesting data from IoT devices, supporting MQTT, HTTP, and WebSocket protocols. It can handle millions of devices and high-throughput, small-message payloads (each event <1 KB) and integrates directly with Amazon S3 via IoT Core rules, making it the ideal ingestion endpoint for near-real-time analytics on streaming IoT data.

Answer

AWS DataSync

Answer

Amazon AppFlow

Answer

Amazon Kinesis Data Streams

Question 2

A company uses AWS Glue ETL jobs to transform data from Amazon S3 to Amazon Redshift. The job reads JSON files, applies schema mapping, and writes to a Redshift table. Recently, the job started failing with memory errors. The data volume has increased tenfold. Which approach should a data engineer take to resolve this issue with minimal code changes?

Accepted Answer

Increase the number of DPUs allocated to the Glue job.. Option C is correct because increasing the number of DPUs (Data Processing Units) allocated to the AWS Glue job directly addresses the memory constraint caused by a tenfold increase in data volume. Glue ETL jobs run on Apache Spark, which distributes data processing across executors; more DPUs provide more memory and compute capacity, allowing the job to handle larger datasets without code changes.

Answer

Switch from Spark to Python Shell job type.

Answer

Implement batch processing with smaller file sizes.

Answer

Use Redshift Spectrum to query data directly from S3.

Question 3

A financial services company processes real-time stock trade data. They use Amazon Kinesis Data Streams with a shard count of 5, each shard receiving about 500 records per second. The consumer application uses the Kinesis Client Library (KCL) with DynamoDB for checkpointing. Lately, some records are being processed multiple times. What is the most likely cause?

Accepted Answer

The consumer application is crashing and restarting, causing re-processing of records.. The Kinesis Client Library (KCL) uses DynamoDB to track checkpoint progress for each shard. If the consumer application crashes and restarts, the KCL will resume processing from the last committed checkpoint, which may be behind the actual processing point. This causes records that were already processed (but not yet checkpointed) to be re-processed, leading to duplicate processing.

Answer

The Kinesis stream's iterator age is exceeding the retention period.

Answer

The DynamoDB table used for checkpointing is throttling write requests.

Answer

The record size exceeds the 1 MB API limit, causing retries.

Question 4

A data engineering team needs to transform CSV files stored in Amazon S3 into Parquet format using AWS Glue. The files are partitioned by date and are updated hourly. Which AWS Glue feature should be used to automatically detect the schema and partition structure?

Accepted Answer

AWS Glue Crawler. AWS Glue Crawler is the correct choice because it automatically scans data in S3, infers the schema (including data types), and detects the partition structure (e.g., date-based partitions like year/month/day) by examining the folder hierarchy. It then populates the AWS Glue Data Catalog with metadata, enabling ETL jobs to read the data without manual schema definition.

Answer

AWS Glue DataBrew

Answer

AWS Lake Formation

Answer

Amazon Athena

Question 5

An e-commerce company ingests clickstream data from their website into Amazon S3. The data is in JSON format, and each file is about 10 MB. They need to transform the data into a columnar format for analytics and load it into Amazon Redshift nightly. The transformation should be cost-effective and require minimal operational overhead. Which approach meets these requirements?

Accepted Answer

Use AWS Glue ETL job to convert to Parquet and load into Redshift.. AWS Glue ETL is the correct choice because it is a serverless, managed service that can efficiently convert JSON to Parquet (a columnar format optimized for Redshift) and load the data into Redshift with minimal operational overhead. The nightly batch processing of 10 MB files is well-suited for Glue's pay-per-use pricing, making it cost-effective without requiring infrastructure management.

Answer

Use Amazon Redshift COPY command to load JSON directly.

Answer

Use Amazon EMR with Spark to transform and load data.

Answer

Use AWS Lambda to transform each file and write to Redshift.

Question 6

A company uses AWS Database Migration Service (DMS) to continuously replicate data from an on-premises Oracle database to Amazon S3 in Parquet format. The replication is used for near-real-time analytics. Recently, the DMS task started failing with an error indicating insufficient memory. The source database is large (2 TB). What should a data engineer do to resolve this issue while minimizing changes to the existing architecture?

Accepted Answer

Increase the DMS replication instance size.. The error indicates the DMS replication instance is running out of memory during continuous replication of a 2 TB Oracle database to S3 in Parquet format. Increasing the replication instance size (Option D) directly addresses the memory constraint by providing more RAM and processing capacity, which is necessary for handling large volumes of Change Data Capture (CDC) data and Parquet conversion overhead. This solution requires minimal architectural changes, as it only involves modifying the instance class in the DMS task settings.

Answer

Change the target format to JSON to reduce memory usage.

Answer

Split the DMS task into multiple smaller tasks.

Answer

Use Change Data Capture (CDC) only, without full load.

Question 7

A data engineer needs to ingest data from multiple SaaS applications (Salesforce, Marketo) into Amazon S3 for a data lake. The data volumes are moderate and the sync needs to be scheduled daily. Which AWS service is most appropriate for this task?

Accepted Answer

Amazon AppFlow. Amazon AppFlow is purpose-built for securely transferring data between SaaS applications (like Salesforce and Marketo) and AWS services (like S3). It supports scheduled, incremental data syncs with built-in connectors, making it the most appropriate choice for moderate-volume daily ingestion into a data lake.

Answer

AWS Glue

Answer

AWS Database Migration Service (DMS)

Answer

Amazon Kinesis Data Firehose

Question 8

A company uses AWS Lambda to process records from an Amazon Kinesis Data Stream. Each record is about 50 KB. The Lambda function transforms the data and writes to Amazon DynamoDB. Recently, the Lambda function has been experiencing throttling and high error rates. The Kinesis stream has 10 shards. What is the most cost-effective solution to improve processing throughput?

Accepted Answer

Increase the Batch Window (MaximumBatchingWindowInSeconds) for the event source mapping.. Option D is correct because increasing the Batch Window (MaximumBatchingWindowInSeconds) allows the Lambda function to accumulate more records from the Kinesis stream before invoking the function, reducing the number of invocations and thus lowering the chance of throttling. This is the most cost-effective solution as it does not require additional shards, memory, or parallelization, and directly addresses the high error rates caused by excessive concurrent executions.

Answer

Increase the number of shards in the Kinesis stream.

Answer

Increase the Parallelization Factor for the Lambda event source mapping.

Answer

Increase the memory allocated to the Lambda function.

Question 9

A data engineer is designing a data ingestion pipeline for a social media analytics platform. The pipeline must ingest tweets in real-time, perform sentiment analysis, and store results in Amazon S3. The sentiment analysis is compute-intensive and must be done as the data arrives. The estimated throughput is 10,000 tweets per second. Which architecture is most suitable?

Accepted Answer

Amazon Kinesis Data Streams with Amazon Kinesis Data Analytics for sentiment analysis, then Kinesis Data Firehose to S3.. Option C is the most suitable because Amazon Kinesis Data Streams can ingest up to 10,000 records per second per shard (with shard-level scaling), and Kinesis Data Analytics provides built-in, low-latency stream processing for compute-intensive sentiment analysis using SQL or Apache Flink. Kinesis Data Firehose then reliably buffers and writes the processed results to Amazon S3 without custom code, ensuring near-real-time delivery.

Answer

Amazon SQS with AWS Lambda pollers to process tweets and store in S3.

Answer

Amazon EMR with Spark Streaming to process tweets and write to S3.

Answer

Amazon API Gateway with AWS Lambda to process each tweet and store in S3.

Question 10

A data engineer needs to transfer 50 TB of historical data from an on-premises HDFS cluster to Amazon S3. The network bandwidth is limited to 100 Mbps. The transfer must be completed within one week. Which service should be used?

Accepted Answer

AWS Snowball. AWS Snowball is the correct choice because transferring 50 TB over a 100 Mbps network would take approximately 46 days (50 TB * 8 bits/byte / 100 Mbps / 86400 seconds/day), far exceeding the one-week deadline. Snowball provides a physical storage device that can be shipped to the on-premises location, allowing data to be loaded locally and shipped to AWS, bypassing network bandwidth constraints entirely.

Answer

AWS DataSync

Answer

Amazon CloudFront

Answer

AWS Database Migration Service (DMS)