DEA-C01 Data Ingestion and Transformation Practice Test 37 — 15 Questions

Question 1

A data pipeline ingests streaming data from thousands of IoT devices into Kinesis Data Streams. The data must be transformed using a simple field mapping before being stored in S3. Which service should be used to perform the transformation with minimal operational overhead?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Kinesis Data Firehose with a Lambda transformation. Kinesis Data Firehose can invoke a Lambda function to perform simple field mapping transformations before delivering data to S3, minimizing operational overhead. Option A is wrong because AWS Lambda invoked directly by the Kinesis stream requires custom logic for S3 delivery and stream management, increasing overhead. Option B is wrong because AWS Glue ETL jobs are designed for batch processing and are more complex to set up for streaming transformations. Option C is wrong because Kinesis Data Analytics is used for real-time analytics with SQL or Flink, not simple field mapping transformations.

Answer

AWS Lambda function invoked by the Kinesis stream

Answer

AWS Glue ETL job

Answer

Kinesis Data Analytics

Question 2

A company uses S3 as a data lake. They want to ingest on-premises relational database data daily with full-load snapshots. The data volume is 500 GB per day. The database is accessible over the internet. Which service should they use for this ingestion?

Accepted Answer

AWS Database Migration Service (DMS). AWS Database Migration Service (DMS) can perform full-load migrations from on-premises databases to S3, making it ideal for daily snapshots of 500 GB. Option A is wrong because Kinesis Data Firehose is designed for streaming data, not for periodic database snapshots. Option B is wrong because while AWS Glue ETL can read from JDBC, it is not optimized for large full-load snapshots and lacks built-in replication capabilities. Option D is wrong because AWS Transfer Family is for file transfers over SFTP, FTPS, or FTP, not for direct database connections.

Answer

Kinesis Data Firehose

Answer

AWS Glue ETL job reading from JDBC

Answer

AWS Transfer Family

Question 3

A company ingests streaming data from multiple sources into a single Kinesis Data Streams stream. Each source produces records with a different schema. The data must be routed to different S3 prefixes based on the source. Which approach minimizes transformation overhead?

Accepted Answer

Use a single Kinesis Data Firehose with a Lambda transformation that reads schema metadata from DynamoDB to determine the S3 prefix.. Using a single Kinesis Data Firehose delivery stream with a Lambda transformation that reads schema metadata from DynamoDB to determine the S3 prefix minimizes transformation overhead by routing data in real-time without additional storage or processing steps. Option B is wrong because ingesting all data into S3 first and then using AWS Glue ETL jobs to partition and route data to different prefixes adds latency and transformation overhead due to the need for batch processing. Option C is wrong because using separate Kinesis Data Streams for each source with separate Firehose delivery streams increases cost and complexity, and does not minimize transformation overhead as it requires managing multiple streams. Option D is wrong because using Kinesis Data Analytics to run SQL queries for routing adds unnecessary complexity and overhead, as Kinesis Data Analytics is better suited for real-time analytics, not simple routing.

Answer

Ingest all data into S3 and use AWS Glue ETL jobs to partition and route data to different prefixes.

Answer

Use separate Kinesis Data Streams for each source and configure separate Firehose delivery streams.

Answer

Use Kinesis Data Analytics to run SQL queries that route data to different Firehose streams.

Question 4

A company needs to ingest data from an external FTP server into AWS S3. The FTP server is not accessible from the internet. Which AWS service should be used to securely transfer the data?

Accepted Answer

AWS Transfer Family with SFTP endpoint in a VPC. AWS Transfer Family supports SFTP and can be configured with a VPC endpoint to securely transfer data from an FTP server in a private network to S3. Option A is wrong because Kinesis Data Firehose is designed for streaming data ingestion, not for FTP transfers. Option C is wrong because AWS DataSync requires network connectivity to the source, and the FTP server is not internet-accessible. Option D is wrong because AWS Snowball Edge is intended for large-scale offline data transfers, not regular FTP-based transfers.

Answer

Kinesis Data Firehose

Answer

AWS DataSync

Answer

AWS Snowball Edge

Question 5

A data engineer is using AWS Glue ETL to transform data from an S3 data lake. The job fails with a memory error. Which approach should be used to resolve this issue without major code changes?

Accepted Answer

Increase the number of DPUs allocated to the Glue job. Increasing the number of DPUs (Data Processing Units) allocated to the Glue job provides more memory and parallelism. Option A is wrong because rewriting in PySpark is a major code change. Option B is wrong because using a smaller file format may not address memory issues. Option D is wrong because using a different service is unnecessary.

Answer

Rewrite the ETL script in PySpark instead of Scala

Answer

Change the input file format from CSV to Parquet

Answer

Use Amazon EMR instead of AWS Glue

Question 6

A company uses Kinesis Data Firehose with a Lambda function for data transformation. The transformation is failing intermittently due to Lambda timeouts. The maximum record size is 1 MB. What is the most cost-effective way to reduce failures without losing data?

Accepted Answer

Increase the Lambda function timeout and memory allocation. The most cost-effective way to reduce failures from Lambda timeouts without losing data is to increase the Lambda function's timeout and memory allocation (Option D). Lambda timeouts occur when the transformation takes longer than the allocated timeout. Increasing memory also increases CPU, which can speed up processing and reduce timeouts. This approach is cost-effective because you only pay for the increased resources when the function runs, and it avoids data loss since all records are still transformed. Option A (Kinesis Data Analytics) adds unnecessary complexity and cost. Option B (decreasing batch size) reduces the number of records per invocation, which can help but may increase costs because more invocations are needed; also, it doesn't directly address timeouts. Option C (sending failed records to an S3 dead-letter bucket) would result in data loss for those records, as they are not transformed, and the question says 'without losing data'. Thus, D is the best choice.

Answer

Use Kinesis Data Analytics to pre-process data before Firehose

Answer

Decrease the Firehose batch size to reduce the number of records per invocation

Answer

Configure the Firehose delivery stream to send failed records to an S3 dead-letter bucket

Question 7

A company wants to ingest data from an on-premises SQL Server database into Amazon Redshift. They need to transform the data during ingestion, such as masking PII columns. Which approach meets these requirements with minimal operational overhead?

Accepted Answer

Use AWS Database Migration Service (DMS) with transformation rules. AWS DMS can perform transformations, such as data masking, during the migration process and can load data directly into Amazon Redshift, minimizing operational overhead. Option A (AWS Glue ETL) adds an extra step and overhead compared to DMS. Option B (custom application on EC2) introduces high operational overhead for management and scaling. Option C (Kinesis Data Firehose) is designed for streaming data and is not suitable for batch ingestion from an on-premises SQL Server database.

Answer

Use AWS Glue ETL jobs to extract data from SQL Server, transform it, and load into Redshift

Answer

Use a custom application on EC2 to extract, transform, and load

Answer

Use Kinesis Data Firehose to stream data from SQL Server to Redshift

Question 8

A company stores IoT sensor data in S3 as JSON files. They need to convert the data to Parquet format for efficient querying with Amazon Athena. Which AWS service can perform this transformation with minimal effort?

Accepted Answer

AWS Glue ETL job. AWS Glue ETL jobs can easily convert JSON to Parquet with built-in transforms. Option A is wrong because Kinesis Data Firehose is for streaming data ingestion, not batch transformations. Option B is wrong because Amazon Athena is a query engine, not a transformation service. Option D is wrong because AWS Lambda is for small, event-driven transformations and is not ideal for large-scale batch conversion.

Answer

Kinesis Data Firehose

Answer

Amazon Athena

Answer

AWS Lambda

Question 9

A company ingests streaming data from social media APIs into Kinesis Data Streams. Each record is approximately 5 KB. The data must be enriched with geolocation information from a DynamoDB table before being stored in S3. The enrichment process takes about 200 ms per record. Which architecture minimizes latency and cost?

Accepted Answer

Use Kinesis Data Firehose with a Lambda function that queries DynamoDB. Kinesis Data Firehose can invoke a Lambda function for per-record enrichment, providing automatic scaling and low operational overhead. Option A is wrong because an EC2 instance adds significant operational overhead and requires manual scaling, increasing complexity and cost. Option B is wrong because AWS Glue ETL jobs are optimized for batch processing, not continuous streaming with low latency per record. Option C is wrong because Kinesis Data Analytics (SQL) is designed for real-time analytics on streaming data, not for per-record enrichment involving external lookups like DynamoDB.

Answer

Use an EC2 instance running a custom application to consume from Kinesis, enrich, and write to S3

Answer

Use AWS Glue ETL jobs running continuously on the stream

Answer

Use Kinesis Data Analytics to perform enrichment with SQL

Question 10

A company is designing a data ingestion pipeline for clickstream data from a website. The data must be ingested in near real-time. Which TWO services can be used together to build this pipeline?

Accepted Answer

Amazon Kinesis Data Streams. Amazon Kinesis Data Streams can ingest clickstream data in near real-time. Option D is correct because Amazon Kinesis Data Firehose can deliver that streaming data to destinations like S3. Option B is wrong because SQS is a message queuing service, not a streaming ingestion service. Option C is wrong because S3 is a storage service, not a real-time ingestion service. Option E is wrong because DynamoDB is a NoSQL database, not a streaming ingestion service.

Answer

Amazon Simple Queue Service (SQS)

Answer

Amazon S3

Answer

Amazon DynamoDB

Question 11

A company is migrating a legacy on-premises ETL pipeline to AWS. The pipeline processes daily batch files from an FTP server. The data must be transformed using complex business logic before being loaded into Amazon Redshift. Which THREE AWS services should be used for this migration?

Accepted Answer

Amazon Redshift. (Amazon Redshift) is correct as the target data warehouse for loading transformed data. Option C (AWS Glue) is correct because it can handle complex business transformations using PySpark or Python. Option E (AWS Transfer Family) is correct for replacing the FTP server and ingesting daily batch files securely into Amazon S3, which can then be processed by Glue. Option A (Amazon Athena) is incorrect because Athena is a query service, not an ETL tool for complex transformations. Option D (Amazon Kinesis Data Streams) is incorrect because it is designed for real-time streaming data, not batch file processing from FTP.

Answer

Amazon Athena

Answer

Amazon Kinesis Data Streams

Question 12

A company uses Kinesis Data Firehose to deliver streaming data to S3. They need to transform the data by adding a timestamp and removing sensitive fields. Which TWO approaches can achieve this?

Accepted Answer

Use AWS Glue ETL to process data after delivery to S3. Options C and E are correct. E: Kinesis Firehose can invoke a Lambda function to transform records (add timestamp, remove fields) before delivery to S3. C: AWS Glue ETL can process data after it is stored in S3, performing transformations like adding timestamps and removing sensitive fields. Option A is incorrect because Kinesis Data Analytics is for real-time analytics, not for adding timestamps or removing fields in the Firehose pipeline. Option B is incorrect because S3 Select is used to retrieve subsets of data using SQL, not to transform data. Option D is incorrect because Redshift Spectrum is for querying data in S3, not for transforming it.

Answer

Use Kinesis Data Analytics to transform the stream

Answer

Use S3 Select to transform data at rest

Answer

Use Amazon Redshift Spectrum to transform data

Question 13

A company runs a data pipeline using AWS Glue ETL jobs to process daily files from an S3 bucket. The files are in CSV format and range from 1 GB to 10 GB. The Glue job runs successfully for small files but fails with an 'Out of Memory' error for files larger than 5 GB. The job uses a single G.1X DPU (16 GB memory). The company needs to process these large files without changing the existing ETL script. Which solution should the company implement?

Accepted Answer

Increase the number of DPUs and use the G.2X worker type to provide more memory per worker.. Increasing the number of DPUs and switching to the G.2X worker type allocates more memory per worker (32 GB instead of 16 GB), allowing the Glue job to process larger CSV files without modifying the ETL script. Option A is incorrect because converting to Parquet would require changing the script and may still encounter memory limits with very large files. Option B is incorrect because Optimus format is not a standard AWS Glue feature; the correct approach is to increase memory. Option C is incorrect because moving to Amazon EMR with Spark would likely require rewriting the script, which the company wants to avoid.

Answer

Convert the input files from CSV to Parquet format to reduce memory usage.

Answer

Use the Optimus format in AWS Glue to compress data.

Answer

Use Amazon EMR with Spark instead of AWS Glue.

Question 14

A company has a Kinesis Data Firehose delivery stream that receives JSON data from IoT devices. The data is delivered to an S3 bucket. The company notices that the data in S3 is delayed by up to 30 minutes. The Firehose stream is configured with a buffer size of 1 MB and a buffer interval of 60 seconds. The incoming data rate is approximately 100 KB per second. The company needs to reduce the delivery latency to under 5 minutes. Which action should the company take?

Accepted Answer

Decrease the buffer size to 256 KB.. The observed latency of up to 30 minutes is likely due to the buffer size being too large relative to the data rate, causing long waits to fill the buffer. With a data rate of 100 KB/s and a buffer size of 1 MB, the buffer fills in approximately 10 seconds, but the buffer interval of 60 seconds already limits delivery to at most 60 seconds. However, the 30-minute delay suggests additional issues such as backlog or configuration errors. Decreasing the buffer size to 256 KB will cause more frequent deliveries (every ~2.5 seconds), reducing latency. Option A (Lambda transformation) adds processing time and increases latency. Option B (increase buffer interval to 300 seconds) would increase latency. Option C (change compression) does not affect delivery frequency. Therefore, option D is correct.

Answer

Enable Lambda transformation to process data faster.

Answer

Increase the buffer interval to 300 seconds.

Answer

Change the compression format from GZIP to Snappy.

Question 15

A company needs to ingest data from an on-premises MySQL database into Amazon S3 for analytics. The database is 2 TB in size. The company has a low-bandwidth internet connection (10 Mbps). They need to perform an initial full load and then incremental updates every hour. Which approach should they use?

Accepted Answer

Use AWS Database Migration Service (DMS) to perform the full load and ongoing replication.. AWS Database Migration Service (DMS) supports full load and ongoing replication, and can be used with limited bandwidth. Option A is wrong because Kinesis Data Firehose is for streaming data, not database replication. Option C is wrong because Glue ETL is not optimized for continuous replication. Option D is wrong because Snowball Edge is for offline transfer, not ongoing replication.

Answer

Use Kinesis Data Firehose to stream data from MySQL to S3.

Answer

Use AWS Glue ETL jobs to extract data and load into S3.

Answer

Use AWS Snowball Edge to transfer the initial full load, then use AWS DataSync for incremental updates.