DEA-C01 Data Ingestion and Transformation Practice Test 2 — 15 Questions

Question 1

A company wants to ingest real-time streaming data from thousands of IoT devices into AWS for immediate processing. Which service is designed for ingesting large volumes of streaming data with low latency?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Amazon Kinesis Data Streams. Amazon Kinesis Data Streams is purpose-built for real-time streaming data ingestion at scale. Option A (S3) is object storage, not designed for streaming ingestion. Option B (DMS) is for database migration, not real-time data streaming. Option C (SQS) is a message queue service that is pull-based and not optimized for high-throughput streaming with low latency. Only option D (Kinesis Data Streams) meets all requirements for ingesting large volumes of streaming data with low latency.

Answer

Amazon Simple Storage Service (S3)

Answer

AWS Database Migration Service (DMS)

Answer

Amazon Simple Queue Service (SQS)

Question 2

A data engineering team needs to ingest CSV files from an S3 bucket into a Redshift cluster on a daily basis. The files are large (up to 100 GB each). Which approach is MOST cost-effective and efficient?

Accepted Answer

Use the Redshift COPY command with an IAM role to load data directly from S3.. Using the Redshift COPY command with an IAM role is the most cost-effective and efficient approach for loading large CSV files from S3 into Redshift on a daily basis. The COPY command is optimized for bulk data loading and can handle files up to 100 GB efficiently. Option A (Kinesis Data Firehose) is designed for streaming data, not batch loading of large files. Option B (AWS Data Pipeline) adds unnecessary cost and complexity for a simple scheduled copy job. Option C (Lambda function) has execution time limits (15 minutes) and memory constraints, making it unsuitable for processing 100 GB files atomically. Therefore, the COPY command is the best choice.

Answer

Use Amazon Kinesis Data Firehose to load data into Redshift.

Answer

Use AWS Data Pipeline to copy data from S3 to Redshift.

Answer

Use a Lambda function to read each file and insert rows individually.

Question 3

A company uses AWS Database Migration Service (DMS) to migrate an on-premises PostgreSQL database to Amazon RDS for PostgreSQL. The migration is ongoing and uses change data capture (CDC). The engineer notices that the target database is falling behind the source by several hours. What is the MOST likely cause?

Accepted Answer

A large full load is being transferred, which delays the start of CDC.. A large full load can delay the start of CDC, causing the target to fall behind by hours. Option A is incorrect because disabling parallel apply on the target table can reduce replication speed, but it is not the most likely cause of a multi-hour lag. Option C is incorrect because while an under-provisioned replication instance can cause performance issues, it is less likely than the full load delaying the start of CDC. Option D is incorrect because if change data capture were actually disabled, the replication would stop completely, not just fall behind.

Answer

The target table has disabled parallel apply.

Answer

The replication instance is under-provisioned and needs to be larger.

Answer

Change data capture has been disabled on the source database.

Question 4

A company runs a nightly batch ETL job using AWS Glue to transform data from Amazon RDS for MySQL to Amazon S3. The job reads 100 tables and writes Parquet files partitioned by date. Recently, the job started failing with 'ThrottlingException' from the RDS database. The data volume has increased, and the Glue job is reading large tables without any filtering. The job uses a single Glue job with multiple Spark executors. The engineer needs to reduce the load on the RDS database while maintaining the same processing time. What should the engineer do?

Accepted Answer

Modify the Glue job to use a JDBC connection with a WHERE clause to read only the latest partition.. Using a JDBC connection with a WHERE clause to filter data by date reduces the amount of data read from RDS, reducing throttling. Option A is wrong because AWS DMS is a migration tool, not designed for regular ETL workloads, and continuous replication would not reduce the load on RDS during the Glue job. Option B is wrong because reading all data into S3 first still requires reading all data from RDS, which does not reduce the load on RDS. Option C is wrong because increasing DPUs would increase parallelism and potentially increase the load on RDS, worsening the throttling.

Answer

Use AWS DMS to continuously replicate data to S3 and then run Glue on the S3 data.

Answer

Change the job to read all data from RDS into a staging table in S3 first, then transform.

Answer

Increase the number of DPUs to process data faster.

Question 5

A data engineer needs to ingest streaming data from an IoT fleet into Amazon S3 for near-real-time analytics. The data volume is approximately 5 GB per hour, and each event is less than 1 KB. Which AWS service should be used as the ingestion endpoint?

Accepted Answer

AWS IoT Core. AWS IoT Core is purpose-built for ingesting data from IoT devices, supporting MQTT, HTTP, and WebSocket protocols. It can handle millions of devices and high-throughput, small-message payloads (each event <1 KB) and integrates directly with Amazon S3 via IoT Core rules, making it the ideal ingestion endpoint for near-real-time analytics on streaming IoT data.

Answer

AWS DataSync

Answer

Amazon AppFlow

Answer

Amazon Kinesis Data Streams

Question 6

A company uses AWS Glue ETL jobs to transform data from Amazon S3 to Amazon Redshift. The job reads JSON files, applies schema mapping, and writes to a Redshift table. Recently, the job started failing with memory errors. The data volume has increased tenfold. Which approach should a data engineer take to resolve this issue with minimal code changes?

Accepted Answer

Increase the number of DPUs allocated to the Glue job.. Increasing the number of DPUs (Data Processing Units) allocated to the AWS Glue job directly addresses the memory constraint caused by a tenfold increase in data volume. Glue ETL jobs run on Apache Spark, which distributes data processing across executors; more DPUs provide more memory and compute capacity, allowing the job to handle larger datasets without code changes.

Answer

Switch from Spark to Python Shell job type.

Answer

Implement batch processing with smaller file sizes.

Answer

Use Redshift Spectrum to query data directly from S3.

Question 7

A financial services company processes real-time stock trade data. They use Amazon Kinesis Data Streams with a shard count of 5, each shard receiving about 500 records per second. The consumer application uses the Kinesis Client Library (KCL) with DynamoDB for checkpointing. Lately, some records are being processed multiple times. What is the most likely cause?

Accepted Answer

The consumer application is crashing and restarting, causing re-processing of records.. The Kinesis Client Library (KCL) uses DynamoDB to track checkpoint progress for each shard. If the consumer application crashes and restarts, the KCL will resume processing from the last committed checkpoint, which may be behind the actual processing point. This causes records that were already processed (but not yet checkpointed) to be re-processed, leading to duplicate processing.

Answer

The Kinesis stream's iterator age is exceeding the retention period.

Answer

The DynamoDB table used for checkpointing is throttling write requests.

Answer

The record size exceeds the 1 MB API limit, causing retries.

Question 8

A data engineering team needs to transform CSV files stored in Amazon S3 into Parquet format using AWS Glue. The files are partitioned by date and are updated hourly. Which AWS Glue feature should be used to automatically detect the schema and partition structure?

Accepted Answer

AWS Glue Crawler. AWS Glue Crawler is the correct choice because it automatically scans data in S3, infers the schema (including data types), and detects the partition structure (e.g., date-based partitions like year/month/day) by examining the folder hierarchy. It then populates the AWS Glue Data Catalog with metadata, enabling ETL jobs to read the data without manual schema definition.

Answer

AWS Glue DataBrew

Answer

AWS Lake Formation

Answer

Amazon Athena

Question 9

An e-commerce company ingests clickstream data from their website into Amazon S3. The data is in JSON format, and each file is about 10 MB. They need to transform the data into a columnar format for analytics and load it into Amazon Redshift nightly. The transformation should be cost-effective and require minimal operational overhead. Which approach meets these requirements?

Accepted Answer

Use AWS Glue ETL job to convert to Parquet and load into Redshift.. AWS Glue ETL is the correct choice because it is a serverless, managed service that can efficiently convert JSON to Parquet (a columnar format optimized for Redshift) and load the data into Redshift with minimal operational overhead. The nightly batch processing of 10 MB files is well-suited for Glue's pay-per-use pricing, making it cost-effective without requiring infrastructure management.

Answer

Use Amazon Redshift COPY command to load JSON directly.

Answer

Use Amazon EMR with Spark to transform and load data.

Answer

Use AWS Lambda to transform each file and write to Redshift.

Question 10

A company uses AWS Database Migration Service (DMS) to continuously replicate data from an on-premises Oracle database to Amazon S3 in Parquet format. The replication is used for near-real-time analytics. Recently, the DMS task started failing with an error indicating insufficient memory. The source database is large (2 TB). What should a data engineer do to resolve this issue while minimizing changes to the existing architecture?

Accepted Answer

Increase the DMS replication instance size.. The error indicates the DMS replication instance is running out of memory during continuous replication of a 2 TB Oracle database to S3 in Parquet format. Increasing the replication instance size (Option D) directly addresses the memory constraint by providing more RAM and processing capacity, which is necessary for handling large volumes of Change Data Capture (CDC) data and Parquet conversion overhead. This solution requires minimal architectural changes, as it only involves modifying the instance class in the DMS task settings.

Answer

Change the target format to JSON to reduce memory usage.

Answer

Split the DMS task into multiple smaller tasks.

Answer

Use Change Data Capture (CDC) only, without full load.

Question 11

A data engineer needs to ingest data from multiple SaaS applications (Salesforce, Marketo) into Amazon S3 for a data lake. The data volumes are moderate and the sync needs to be scheduled daily. Which AWS service is most appropriate for this task?

Accepted Answer

Amazon AppFlow. Amazon AppFlow is purpose-built for securely transferring data between SaaS applications (like Salesforce and Marketo) and AWS services (like S3). It supports scheduled, incremental data syncs with built-in connectors, making it the most appropriate choice for moderate-volume daily ingestion into a data lake.

Answer

AWS Glue

Answer

AWS Database Migration Service (DMS)

Answer

Amazon Kinesis Data Firehose

Question 12

A company uses AWS Lambda to process records from an Amazon Kinesis Data Stream. Each record is about 50 KB. The Lambda function transforms the data and writes to Amazon DynamoDB. Recently, the Lambda function has been experiencing throttling and high error rates. The Kinesis stream has 10 shards. What is the most cost-effective solution to improve processing throughput?

Accepted Answer

Increase the Batch Window (MaximumBatchingWindowInSeconds) for the event source mapping.. Increasing the Batch Window (MaximumBatchingWindowInSeconds) allows the Lambda function to accumulate more records from the Kinesis stream before invoking the function, reducing the number of invocations and thus lowering the chance of throttling. This is the most cost-effective solution as it does not require additional shards, memory, or parallelization, and directly addresses the high error rates caused by excessive concurrent executions.

Answer

Increase the number of shards in the Kinesis stream.

Answer

Increase the Parallelization Factor for the Lambda event source mapping.

Answer

Increase the memory allocated to the Lambda function.

Question 13

A data engineer is designing a data ingestion pipeline for a social media analytics platform. The pipeline must ingest tweets in real-time, perform sentiment analysis, and store results in Amazon S3. The sentiment analysis is compute-intensive and must be done as the data arrives. The estimated throughput is 10,000 tweets per second. Which architecture is most suitable?

Accepted Answer

Amazon Kinesis Data Streams with Amazon Kinesis Data Analytics for sentiment analysis, then Kinesis Data Firehose to S3.. The most suitable because Amazon Kinesis Data Streams can ingest up to 10,000 records per second per shard (with shard-level scaling), and Kinesis Data Analytics provides built-in, low-latency stream processing for compute-intensive sentiment analysis using SQL or Apache Flink. Kinesis Data Firehose then reliably buffers and writes the processed results to Amazon S3 without custom code, ensuring near-real-time delivery.

Answer

Amazon SQS with AWS Lambda pollers to process tweets and store in S3.

Answer

Amazon EMR with Spark Streaming to process tweets and write to S3.

Answer

Amazon API Gateway with AWS Lambda to process each tweet and store in S3.

Question 14

A data engineer needs to transfer 50 TB of historical data from an on-premises HDFS cluster to Amazon S3. The network bandwidth is limited to 100 Mbps. The transfer must be completed within one week. Which service should be used?

Accepted Answer

AWS Snowball. AWS Snowball is the correct choice because transferring 50 TB over a 100 Mbps network would take approximately 46 days (50 TB * 8 bits/byte / 100 Mbps / 86400 seconds/day), far exceeding the one-week deadline. Snowball provides a physical storage device that can be shipped to the on-premises location, allowing data to be loaded locally and shipped to AWS, bypassing network bandwidth constraints entirely.

Answer

AWS DataSync

Answer

Amazon CloudFront

Answer

AWS Database Migration Service (DMS)

Question 15

A company uses Amazon Kinesis Data Firehose to deliver streaming data to Amazon S3. The data is in JSON format. The delivery stream is configured with a buffer size of 5 MB and a buffer interval of 60 seconds. However, the data engineer notices that S3 objects are being created with sizes much smaller than 5 MB. What is a likely cause?

Accepted Answer

The incoming data rate is too low, causing the buffer interval to trigger before reaching the buffer size.. Kinesis Data Firehose delivers data to S3 when either the buffer size (5 MB) or the buffer interval (60 seconds) is reached, whichever occurs first. If the incoming data rate is low, the buffer interval will expire before accumulating 5 MB of data, resulting in smaller S3 objects.

Answer

The data is being compressed before delivery, reducing object size.

Answer

The data transformation lambda is splitting records into smaller ones.

Answer

The S3 bucket is configured with a lifecycle policy that splits objects.