DEA-C01 Data Ingestion and Transformation Practice Test 5 — 15 Questions

Question 1

A company uses Amazon Kinesis Data Streams with enhanced fan-out consumers. The stream has 5 shards. Each consumer reads from all shards. The total incoming data rate is 25 MB/s. What is the maximum read throughput per consumer if enhanced fan-out is enabled?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

10 MB/s. With enhanced fan-out, each consumer gets a dedicated 2 MB/s read throughput per shard. Since the stream has 5 shards, the maximum read throughput per consumer is 5 shards × 2 MB/s = 10 MB/s. This is because enhanced fan-out eliminates the 2 MB/s total read limit per shard shared among all consumers, providing each consumer with its own 2 MB/s pipe per shard.

Answer

2 MB/s

Answer

1 MB/s

Answer

5 MB/s

Question 2

Which TWO AWS services can be used to ingest data from an on-premise relational database into Amazon S3 on a one-time basis?

Accepted Answer

AWS Database Migration Service (DMS). AWS Database Migration Service (DMS) (B) is correct because it can perform a one-time full-load migration from an on-premise relational database directly into Amazon S3, using S3 as the target endpoint without ongoing replication. AWS Glue (C) is correct because its ETL jobs and crawlers can connect to an on-premise relational database via JDBC and write the extracted data into Amazon S3 in a single batch run. AWS Data Pipeline (A) is not the intended answer here since it is more commonly used for orchestrating scheduled or recurring data movement rather than a straightforward one-time relational-to-S3 ingestion. Amazon SQS (D) is a message queuing service and cannot connect to a relational database or load data into S3. Amazon Kinesis Data Streams (E) is designed for real-time streaming ingestion, not one-time bulk extraction from a relational database.

Answer

AWS Data Pipeline

Answer

Amazon Simple Queue Service (SQS)

Answer

Amazon Kinesis Data Streams

Question 3

A data engineer is designing a data pipeline that ingests CSV files from an FTP server to Amazon S3. The files arrive hourly and each file is about 500 MB. The engineer wants to minimize operational overhead and cost. Which approach is best?

Accepted Answer

Use AWS Transfer for SFTP and point the endpoint to an S3 bucket. AWS Transfer for SFTP is the best approach because it provides a fully managed SFTP endpoint that integrates directly with Amazon S3, eliminating the need to manage servers or write custom code. It automatically scales, handles file transfers, and stores files directly in S3, minimizing operational overhead and cost for hourly 500 MB files.

Answer

Write a Python script in AWS Lambda using boto3 to download from FTP and upload to S3

Answer

Use AWS Snowball Edge to transfer files weekly

Answer

Deploy an Amazon EC2 instance with a cron job to run wget and aws s3 cp

Question 4

A company needs to transform JSON data from Amazon Kinesis Data Streams into Parquet format and store it in Amazon S3. The transformation includes simple field mappings and type conversions. Which approach is most cost-effective and serverless?

Accepted Answer

Use Amazon Kinesis Data Firehose with an AWS Lambda function for transformation and output to Parquet. Amazon Kinesis Data Firehose can directly deliver streaming data to Amazon S3 in Parquet format. By attaching an AWS Lambda function for simple field mappings and type conversions, you achieve a fully serverless, cost-effective solution without managing any infrastructure. This approach minimizes costs because you pay only for the data volume processed by Firehose and the Lambda invocations, avoiding the overhead of always-on compute resources.

Answer

Use Amazon EC2 instances running Apache Spark Streaming

Answer

Use Amazon SageMaker for transformation

Answer

Use an AWS Glue ETL job triggered by a Kinesis stream

Question 5

A company is ingesting CSV files into Amazon S3. Each file contains a header row. The pipeline uses AWS Glue to crawl the S3 bucket and create a table in the AWS Glue Data Catalog. However, the crawler is including the header as data. What is the most likely cause?

Accepted Answer

The crawler is configured to use a custom classifier that does not skip headers. The AWS Glue crawler uses built-in classifiers to infer schema of CSV files. The built-in CSV classifier automatically detects and skips header rows. However, if a custom classifier is configured and it does not have the correct setting to skip headers (e.g., `HasHeader` set to `false` or not defined), the crawler will treat the header row as data. Since the question describes the header being included as data, the most likely cause is that a custom classifier is in use that does not properly skip headers.

Answer

The S3 bucket has versioning enabled

Answer

The crawler is running in 'crawl all folders' mode

Answer

The CSV files are not compressed

Question 6

A company is ingesting streaming data from a fleet of weather sensors. Each sensor sends a JSON payload every second. The data is used for real-time dashboarding and also archived to S3. The pipeline should handle sudden bursts of data without data loss. Which architecture meets these requirements?

Accepted Answer

Amazon Kinesis Data Streams -> AWS Lambda for dashboard -> Amazon Kinesis Data Firehose -> S3. Kinesis Data Streams provides durable, scalable ingestion that can handle sudden bursts of data without loss, while Lambda processes records for real-time dashboarding and Kinesis Data Firehose reliably buffers and archives data to S3. This decoupled architecture ensures no data is lost even during traffic spikes, as Kinesis Data Streams retains data for up to 365 days and Firehose can buffer incoming records before writing to S3.

Answer

Amazon EC2 with Apache Kafka -> S3

Answer

Amazon Kinesis Data Firehose directly with no buffer

Answer

Amazon SQS -> AWS Lambda -> S3

Question 7

A data engineer is ingesting XML data from an external API into Amazon S3. The engineer needs to transform the XML to JSON using AWS Glue. The XML structure is deeply nested. Which Apache Spark method should be used in the Glue ETL script?

Accepted Answer

Use spark.read.format('xml') with Databricks XML library. The Databricks XML library (spark.read.format('xml')) provides native support for parsing deeply nested XML into a DataFrame, which is essential for AWS Glue ETL scripts running on Spark. AWS Glue does not have a built-in 'xml' data source, and the Databricks library handles complex nested structures, attributes, and arrays automatically, making it the standard approach for XML-to-JSON transformation in Spark.

Answer

Use the built-in AWS Glue 'xml' data source

Answer

Use Hadoop's XmlInputFormat

Answer

Use the Spark SQL function from_xml()

Question 8

A company is ingesting large volumes of sensor data into Amazon S3. The data must be encrypted at rest using an AWS KMS customer managed key. Which TWO actions are required to enable server-side encryption with AWS KMS (SSE-KMS) on the S3 bucket?

Accepted Answer

Set the default encryption on the S3 bucket to AWS-KMS and specify the KMS key. Option B is correct because configuring default bucket encryption with the SSE-KMS algorithm (aws:kms) and specifying the customer managed key is what actually enables SSE-KMS for objects uploaded to the bucket, so new objects are automatically encrypted with that KMS key. Option E is correct because the principal performing the upload must have kms:Encrypt (and typically kms:GenerateDataKey) permission on the customer managed key; without these KMS permissions, S3 cannot use the key to encrypt the object and the upload fails. Option A is not required: S3 Versioning is unrelated to enabling SSE-KMS, though it is needed for features like replication or MFA delete. Option C is not required: CloudWatch Logs is for logging/monitoring and does not enable encryption. Option D is not required: a bucket policy denying unencrypted uploads is an optional guardrail, not a prerequisite for enabling SSE-KMS.

Answer

Enable S3 Versioning on the bucket

Answer

Enable Amazon CloudWatch Logs for the bucket

Answer

Add a bucket policy that denies uploads without encryption

Question 9

A data engineer is designing a data ingestion pipeline for real-time clickstream data. The data must be available for both real-time analytics and batch processing. The engineer wants to use Amazon Kinesis Data Streams. Which THREE components should be included in the architecture?

Accepted Answer

Amazon Kinesis Data Streams. Amazon Kinesis Data Streams is the core ingestion service for real-time clickstream data, providing low-latency, durable storage of data records that can be consumed by multiple applications simultaneously. It enables both real-time analytics (via Kinesis Data Analytics or KCL applications) and batch processing (by integrating with Kinesis Data Firehose to deliver data to Amazon S3).

Answer

Amazon Kinesis Data Analytics

Answer

Amazon SQS as a buffer

Question 10

A company uses Kinesis Data Streams to ingest IoT data. The data volume varies, and occasionally the shard write throughput is exceeded, causing ProvisionedThroughputExceeded exceptions. The data engineer needs to handle these spikes without losing data. Which approach is most cost-effective and requires minimal code changes?

Accepted Answer

Use Kinesis Data Firehose as a consumer with retries and buffer settings. Kinesis Data Firehose is the most cost-effective solution because it can be configured as a consumer of the Kinesis Data Stream with built-in retry logic and buffer settings (e.g., buffer size up to 128 MB or buffer interval up to 900 seconds). This handles throughput spikes by buffering data and retrying failed writes without requiring custom code, and it scales automatically without the need to manage shard counts.

Answer

Implement custom retry logic using the Kinesis Client Library with exponential backoff

Answer

Increase the number of shards to handle peak throughput

Answer

Send data to an SQS queue first, then have a Lambda function write to Kinesis

Question 11

A company wants to migrate on-premises data to Amazon S3 using AWS DataSync. The data is 10 TB and the network bandwidth is 1 Gbps. The migration must be completed within 48 hours. What should the data engineer do to meet the deadline?

Accepted Answer

Use AWS DataSync with multiple agents and enable data compression. AWS DataSync can use multiple agents in parallel to increase throughput, and enabling data compression reduces the amount of data transferred over the network. With 10 TB at 1 Gbps, the theoretical minimum transfer time is about 22.2 hours, but real-world overhead (protocol, retransmissions) often exceeds 48 hours without parallelism and compression. Multiple agents and compression together can achieve the required throughput within the deadline.

Answer

Use S3 Transfer Acceleration to speed up the transfer

Answer

Use AWS Snowball Edge to transfer the data physically

Answer

Request a bandwidth increase from the ISP

Question 12

A data engineer is designing a data pipeline that ingests streaming data from Kinesis Data Streams, transforms it using AWS Lambda, and writes to S3. The Lambda function sometimes fails due to transient errors, and the engineer wants to ensure no data is lost. Which approach should be used?

Accepted Answer

Use Kinesis Data Firehose as the delivery stream with Lambda for transformation and configure error handling with retries and a backup S3 bucket. Kinesis Data Firehose natively integrates with Lambda for data transformation and provides built-in error handling with retries. If transformation fails after the retry limit, Firehose can automatically redirect failed records to a backup S3 bucket, ensuring no data is lost. This approach handles transient errors without requiring custom checkpointing or manual retry logic.

Answer

Use the Kinesis Client Library to process records with checkpointing

Answer

Increase the Lambda function's timeout and memory

Answer

Configure a dead-letter queue (DLQ) on Lambda to capture failed records

Question 13

A company needs to ingest data from multiple SaaS applications (e.g., Salesforce, Marketo) into Amazon S3 for centralized analytics. The data volume is several GB per day. Which AWS service is most suitable for this ingestion?

Accepted Answer

AWS Glue. AWS Glue is the most suitable service because it provides a fully managed ETL (Extract, Transform, Load) capability with built-in connectors for SaaS applications like Salesforce and Marketo. Glue can extract data from these sources, transform it as needed, and write it to Amazon S3, handling the several GB per day volume efficiently without requiring custom code for each source.

Answer

Amazon Kinesis Data Firehose

Answer

Amazon Athena

Answer

AWS Data Pipeline

Question 14

A company is ingesting log files from EC2 instances into CloudWatch Logs and then wants to deliver them to S3 for long-term storage and analysis. The data engineer needs to ensure the logs are delivered to S3 within 5 minutes of being generated. Which approach meets this requirement?

Accepted Answer

Use CloudWatch Logs subscription filter with Kinesis Data Firehose to deliver to S3. CloudWatch Logs subscription filters can stream log data in near real-time to Kinesis Data Firehose, which then delivers the data to S3 with a buffer interval configurable down to 60 seconds, easily meeting the 5-minute requirement. This approach provides the lowest latency for automated, continuous delivery without custom code.

Answer

Configure a CloudWatch Logs metric filter and invoke a Lambda function to write to S3

Answer

Use CloudWatch Logs Insights to query logs and save results to S3

Answer

Use the CloudWatch Logs export to S3 feature

Question 15

A data engineer needs to transform CSV files arriving in S3 into Parquet format and partition them by date. The transformation should be event-driven and run immediately after each file is uploaded. Which approach is most efficient?

Accepted Answer

Use S3 event notification to trigger an AWS Glue job. AWS Glue jobs can be triggered directly by S3 event notifications, enabling event-driven, serverless transformation of CSV to Parquet with partitioning by date. Glue is optimized for this batch ETL workload, handling schema inference and partitioning efficiently without managing infrastructure, making it the most efficient choice for immediate, per-file transformation.

Answer

Use an S3 event notification to invoke a Lambda function that converts the file

Answer

Use an Amazon EMR cluster running Spark to process files as they arrive

Answer

Use Amazon Athena CREATE TABLE AS SELECT (CTAS) on a schedule