DEA-C01 Data Ingestion and Transformation Practice Test 39 — 15 Questions

Question 1

A company uses AWS Glue to process CSV files stored in Amazon S3. The data pipeline runs daily, but recently some jobs have failed with a 'MemoryError'. The data volume has grown from 1 GB to 10 GB per day. What is the MOST cost-effective solution to resolve this issue?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Increase the number of DPUs (Data Processing Units) allocated to the Glue job.. Increasing the number of DPUs allocates more memory and processing capacity to the Glue job, directly addressing the MemoryError caused by data growth from 1 GB to 10 GB. Option A is wrong because changing the worker type to G.2X provides more memory per worker, but it is a more expensive option compared to simply increasing DPUs, which scales horizontally. Option C is wrong because converting CSV to Parquet improves performance and reduces storage but does not add memory to the Glue job; the job still runs with the same DPU allocation. Option D is wrong because migrating to Amazon EMR introduces additional operational complexity and cost; increasing DPUs in Glue is simpler and more cost-effective for scaling an existing job.

Answer

Change the Glue worker type from Standard to G.2X.

Answer

Convert the CSV files to Parquet format using an S3 batch operation.

Answer

Migrate the job to Amazon EMR with a larger cluster.

Question 2

A data engineer is troubleshooting a Kinesis Data Firehose delivery stream that sends data to an Amazon S3 bucket. The delivery stream has a buffer size of 5 MB and a buffer interval of 60 seconds. The data ingestion rate is 2 MB per second. The engineer notices that S3 objects are created every 60 seconds but each object is only about 2 MB. What should the engineer do to reduce the number of small S3 objects?

Accepted Answer

Increase the buffer size to 10 MB.. Increasing the buffer size to 10 MB will allow the stream to buffer more data before writing to S3, resulting in larger objects. Option B is wrong because decreasing the buffer interval would create objects more frequently, making the problem worse. Option C is wrong because reducing the buffer size would create even smaller objects. Option D is wrong because switching to Kinesis Data Streams does not solve the buffering issue.

Answer

Decrease the buffer interval to 30 seconds.

Answer

Reduce the buffer size to 2 MB.

Answer

Switch to Kinesis Data Streams and use a Lambda function to write to S3.

Question 3

A company uses Amazon Kinesis Data Analytics for Apache Flink to process streaming data. The application reads from a Kinesis data stream and writes results to an Amazon S3 bucket. Recently, the application has been failing with 'ResourceNotFoundException' for the S3 bucket. What is the MOST likely cause?

Accepted Answer

The S3 bucket ARN is incorrectly specified in the application configuration.. The 'ResourceNotFoundException' for the S3 bucket indicates that the application cannot find the bucket. This is most likely due to an incorrectly specified bucket name or ARN in the application configuration. If the bucket name is misspelled or the ARN is malformed, the Kinesis Data Analytics application cannot resolve the bucket resource. Option A is incorrect because missing 's3:PutObject' permission would cause an 'AccessDenied' error, not 'ResourceNotFoundException'. Option C is incorrect because the AWS Region for S3 is determined by the bucket's location, and specifying a wrong region in Flink code would typically result in a different error (e.g., 'IllegalArgumentException' or cross-region access issues) but not 'ResourceNotFoundException'. Option D is incorrect because versioning is unrelated to bucket existence; disabling versioning does not cause a resource not found error.

Answer

The IAM role used by the application does not have s3:PutObject permission.

Answer

The Flink application code specifies the wrong AWS Region for the S3 bucket.

Answer

The S3 bucket has versioning disabled.

Question 4

A data engineer is designing a data ingestion pipeline for real-time clickstream data from a website. The data must be stored in Amazon S3 in near-real time, and also be available for real-time analytics using Amazon Athena. The pipeline must handle occasional spikes of up to 10x the normal throughput. Which combination of services should the engineer use?

Accepted Answer

Amazon Kinesis Data Streams with AWS Lambda to write to Amazon S3, and Amazon Athena for queries.. Amazon Kinesis Data Streams can handle high throughput spikes (10x normal) with its scalable stream capacity. AWS Lambda can process records in near-real time and write them to Amazon S3. Amazon Athena can query the data directly from S3 using its schema-on-read approach. Option A is wrong because Amazon SQS is a pull-based queue; Lambda would need to poll SQS, adding latency and complexity for near-real-time ingestion. Option B is wrong because AWS DMS is designed for database migration and continuous replication, not for ingesting real-time clickstream data from a website. Option D is wrong because Amazon Kinesis Data Firehose is a simpler option for streaming data to S3, but the question requires near-real-time availability for Athena; Firehose has a minimum buffer interval of 60 seconds, which may not meet near-real-time requirements, and AWS Glue is typically used for batch ETL rather than low-latency streaming transformations.

Answer

Amazon Simple Queue Service (SQS) with AWS Lambda to write to Amazon S3, and Amazon Athena for queries.

Answer

AWS Database Migration Service (DMS) to stream data to Amazon S3, and Amazon Athena for queries.

Answer

Amazon Kinesis Data Firehose with AWS Glue for transformation, and Amazon Athena for queries.

Question 5

A company uses AWS Glue to run ETL jobs that process data from Amazon RDS to Amazon S3. The job runs successfully but the data in S3 is missing some records that exist in the source. The engineer notices that the job uses a JDBC connection and the query extracts data based on a timestamp column. What is the MOST likely cause of the missing records?

Accepted Answer

The timestamp column includes time portion and the job is using an exclusive upper bound.. The job extracts data based on a timestamp column and uses an exclusive upper bound (e.g., WHERE timestamp < some_value). If the timestamp includes a time portion, records with a timestamp exactly equal to the upper bound are excluded, causing them to be missing from S3. Option B is incorrect because S3 bucket permissions would cause the job to fail, not simply miss records. Option C is incorrect because connection pooling does not cause records to be dropped. Option D is incorrect because the Glue job can be configured to read from multiple tables; this is unrelated to the missing records.

Answer

The S3 bucket lacks write permissions.

Answer

The JDBC connection uses connection pooling, causing some records to be dropped.

Answer

The Glue job is configured to only read from one table.

Question 6

A data engineer is ingesting data from an on-premises database to Amazon S3 using AWS DataSync. The data transfer is scheduled to run daily at midnight. The engineer notices that the transfer takes longer than expected and sometimes does not complete before the next scheduled task. What should the engineer do to ensure the transfer completes within the window?

Accepted Answer

Increase the bandwidth limit in the DataSync task settings.. Increase the bandwidth limit in the DataSync task settings. Increasing the bandwidth limit allows DataSync to use more available network capacity, thereby speeding up the data transfer and ensuring it completes within the scheduled window. Option B is incorrect because decreasing the bandwidth limit would slow the transfer further. Option C is incorrect because increasing the schedule frequency does not address the underlying bandwidth issue; it would merely start more transfers that would still be slow. Option D is incorrect because S3 Transfer Acceleration is designed for long-distance transfers over public internet and may not help if the bottleneck is on-premises bandwidth; additionally, DataSync is the recommended service for this scenario.

Answer

Decrease the bandwidth limit to reduce network congestion.

Answer

Increase the schedule frequency to every 12 hours.

Answer

Use S3 Transfer Acceleration instead of DataSync.

Question 7

A company uses Amazon Kinesis Data Firehose to deliver log data to Amazon S3. The data is transformed by a Lambda function that adds a timestamp field. Recently, the Firehose delivery stream has been failing with 'Lambda invocation failed' errors. The Lambda function's CloudWatch Logs show that the function is timing out. What is the MOST likely cause?

Accepted Answer

The Firehose buffer size is too large, causing the Lambda function to receive too many records.. Kinesis Data Firehose sends batches of records to the Lambda function for transformation. If the buffer size (or batch size) is too large, the Lambda function receives too many records and exceeds its configured timeout, causing 'Lambda invocation failed' errors. The timeout errors in CloudWatch Logs confirm this. Option A is incorrect because the function is being invoked, so Lambda permissions are working. Option C is incorrect because a 1-minute timeout may be insufficient for large batches. Option D is incorrect because the logs indicate a timeout, not an out-of-memory error.

Answer

The Lambda function lacks permission to write to CloudWatch Logs.

Answer

The Lambda function timeout is set to 1 minute, which is adequate.

Answer

The Lambda function is running out of memory.

Question 8

A data engineer needs to transform JSON data from Amazon S3 into Parquet format using AWS Glue. The data contains nested fields. Which Glue feature should the engineer use to define the schema and handle the nested structure?

Accepted Answer

Use the 'Relationalize' transform in a Glue ETL script.. The 'Relationalize' transform in AWS Glue is specifically designed to convert nested JSON data into flat tables by extracting and flattening nested fields into separate relational tables. This allows handling complex nested structures effectively. Option A is incorrect because 'FindMatches' is used for deduplication, not for handling nested data. Option B is incorrect because 'DropFields' is used to remove fields from a dataset, not to transform nested structures. Option D is incorrect because 'Spigot' is used for writing sample data for testing or debugging, not for schema definition or handling nesting.

Answer

Use the 'FindMatches' transform to identify duplicates.

Answer

Use the 'DropFields' transform to remove nested fields.

Answer

Use the 'Spigot' transform to write sample data.

Question 9

A company uses AWS Glue to process data from Amazon S3. The Glue job fails with a 'SchemaDetectionException'. The data engineer wants to ensure the schema is correctly inferred. Which TWO actions should the engineer take? (Choose two.)

Accepted Answer

Use the Glue Data Catalog as the source for schema definition.. The correct answers are A and E. Option A uses the Glue Data Catalog as the source for schema definition, providing a consistent schema and avoiding schema detection failures. Option E sets the 'groupFiles' option to 'inPartition', which helps Glue combine small files within a partition for schema inference, reducing 'SchemaDetectionException' errors. Option B is incorrect because adding a column with a default value does not affect schema detection. Option C is incorrect because increasing DPUs only improves processing speed, not schema inference. Option D is incorrect because converting to Parquet may change the schema but does not directly address schema detection issues.

Answer

Add a column with a default value to the data.

Answer

Increase the number of Glue DPUs to speed up processing.

Answer

Convert all input files to Parquet format.

Question 10

A data engineer is designing a data ingestion pipeline that uses AWS DMS to migrate data from an on-premises Oracle database to Amazon S3 in Parquet format. The engineer needs to ensure that data is continuously replicated with minimal latency. Which THREE steps should the engineer take? (Choose three.)

Accepted Answer

Configure a DMS task with a transformation rule to convert to Parquet.. Options A, B, and D are correct. To continuously replicate data with minimal latency from Oracle to Amazon S3 in Parquet format, the engineer should: (A) configure a DMS task with a transformation rule to convert to Parquet, (B) specify an S3 bucket as the target endpoint with data format set to Parquet, and (D) enable change data capture (CDC) on the source database. Option C (using AWS SCT) is incorrect because SCT is for schema conversion, not for DMS replication. Option E (full load only) is incorrect because it does not provide continuous replication.

Answer

Use AWS Schema Conversion Tool (SCT) to convert the schema.

Answer

Perform a full load only, without CDC.

Question 11

A company uses Amazon Kinesis Data Streams to ingest clickstream data. The data must be transformed and stored in Amazon S3 for batch analytics. The engineer wants to use AWS Lambda for transformation. Which TWO configurations are required? (Choose two.)

Accepted Answer

Create an event source mapping from the Kinesis stream to the Lambda function.. The correct answers are D and E. D: An event source mapping is required for Lambda to poll records from Kinesis. E: Lambda needs an IAM role with permissions to read from Kinesis and write to S3. Option A is incorrect because Lambda does not write to S3 via Firehose; Firehose is a separate service that could directly write to S3, but in this scenario, Lambda is the transformation actor and writes directly to S3. Option B is incorrect because a Kinesis stream cannot send records directly to S3; it needs a consumer or Firehose. Option C is incorrect because Lambda errors can be sent to SQS but it is not required for this setup; the question asks for required configurations.

Answer

Configure the Lambda function to write to S3 via Kinesis Data Firehose.

Answer

Configure the Kinesis stream to send records directly to S3.

Answer

Set up an SQS queue as a destination for Lambda errors.

Question 12

A data engineer is building a real-time data pipeline to ingest sensor data from IoT devices. The data is sent to AWS IoT Core, which publishes messages to a Kinesis Data Stream. Each message is about 1 KB in size. The data must be transformed (add a device location field) and then stored in Amazon S3 for long-term analytics. The engineer has set up a Lambda function to transform the records and write to S3. However, the engineer notices that the Lambda function is invoked thousands of times per second, causing high costs and occasional throttling. The Lambda function processes only one record at a time. The engineer wants to reduce the number of Lambda invocations and improve throughput. What should the engineer do?

Accepted Answer

Configure the event source mapping to use a larger batch size and set a batch window.. Configuring the event source mapping with a larger batch size and a batch window allows Lambda to process multiple records per invocation, reducing the number of invocations and costs. This improves throughput and reduces throttling. Option A is incorrect because reducing shards reduces the stream capacity, causing backpressure and potential data loss. Option B is incorrect because increasing memory does not reduce the number of invocations; it only speeds up processing per invocation, but still processes one record at a time. Option C is incorrect because Kinesis Data Firehose can batch records, but it still uses per-record Lambda transformation if you use a Lambda function, or it can use built-in transformations but not the flexible logic described. The most direct solution is to batch records in the existing Lambda function via event source mapping parameters.

Answer

Reduce the number of shards in the Kinesis stream to limit concurrency.

Answer

Increase the Lambda function's memory allocation to improve performance.

Answer

Replace the Lambda function with Amazon Kinesis Data Firehose and use its built-in transformation.

Question 13

A data engineer is setting up a data ingestion pipeline using Amazon Kinesis Data Firehose to deliver web server logs to Amazon S3. The logs are in JSON format and the engineer wants to convert them to Parquet format. The engineer has configured a Glue table for the schema. However, when testing, the Firehose delivery stream fails with 'Error converting to Parquet'. The engineer checks the Glue table schema and notices that it includes a column 'timestamp' of type 'string' in the format 'yyyy-MM-dd HH:mm:ss'. The logs have a 'timestamp' field in the same format. What is the MOST likely cause of the failure?

Accepted Answer

The Glue table schema does not match the data schema exactly.. The failure is most likely due to a schema mismatch between the Glue table and the actual data. Firehose uses the Glue table schema to convert JSON to Parquet, and if the schema does not exactly match (e.g., column order, data types, or extra columns), the conversion fails. Option A is incorrect because Firehose does support Parquet conversion when a Glue table is provided. Option C is incorrect because S3 permissions would cause a different error (e.g., AccessDenied). Option D is incorrect because a string type for timestamp is acceptable as long as it matches; the issue is not the type but the overall schema mismatch.

Answer

Firehose does not support Parquet conversion.

Answer

The S3 bucket lacks write permissions for Firehose.

Answer

The 'timestamp' column must be of type 'timestamp' instead of 'string'.

Question 14

A company is ingesting streaming data from IoT devices into Amazon Kinesis Data Streams. The data is then processed by an AWS Lambda function that transforms the records and writes them to an Amazon S3 bucket. Recently, the Lambda function has been timing out and the S3 bucket is not receiving all expected data. The Kinesis stream is not throttling and has sufficient shards. Which step should the company take to resolve this issue?

Accepted Answer

Increase the Lambda function's timeout and memory allocation.. The Lambda function is timing out, indicating that it cannot process the records within the allotted time. Increasing the Lambda function's timeout and memory allocation (Option B) provides more execution time and CPU resources, which can help process larger or more frequent records before timing out. Option A is incorrect because reserved concurrency limits the maximum number of concurrent instances, not the execution duration of a single instance. Option C is incorrect because increasing shards is unnecessary; the stream is not throttling, and more shards do not resolve Lambda timeouts. Option D is incorrect because enhanced fan-out reduces latency for multiple consumers but does not prevent a single Lambda function from timing out due to processing time.

Answer

Increase the Lambda function's reserved concurrency.

Answer

Increase the number of shards in the Kinesis stream.

Answer

Enable enhanced fan-out on the Kinesis stream to reduce latency.

Question 15

A data engineer needs to ingest data from an on-premises Oracle database into Amazon S3 for analytics. The data changes frequently and the engineer wants to capture both initial load and incremental changes with minimal latency. Which TWO AWS services should be used together? (Choose TWO.)

Accepted Answer

AWS Database Migration Service (DMS). AWS DMS (Option A) is correct because it can perform an initial load of data from an on-premises Oracle database to Amazon S3 and then continuously replicate incremental changes using change data capture (CDC) with minimal latency. Amazon Kinesis Data Streams (Option E) complements DMS by enabling near-real-time streaming of data changes, which can be further processed or stored in S3. Option B (Lambda) is not a direct replacement for database replication; it can process events but does not natively capture database changes. Option C (Glue) is an ETL service, not designed for real-time replication. Option D (Transfer Family) is for file transfers, not database replication.

Answer

AWS Lambda

Answer

AWS Glue

Answer

AWS Transfer Family