DEA-C01 Data Ingestion and Transformation Practice Test 46 — 15 Questions

Question 1

A company uses Amazon Kinesis Data Analytics for Apache Flink to process streaming data. The application reads from a Kinesis data stream, performs a 1-minute tumbling window aggregation, and writes results to an S3 bucket. Recently, the application started experiencing checkpoint failures and increasing processing delay. Which action should the engineer take FIRST to diagnose the issue?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Monitor CPU and memory utilization of the Flink application using Amazon CloudWatch metrics.. When a Flink application shows checkpoint failures and growing processing delay, the first diagnostic step is to check resource utilization via CloudWatch metrics (CPU, memory, heap, checkpoint duration). Checkpoint failures often stem from backpressure or resource exhaustion, so understanding whether the application is CPU- or memory-bound guides the next action. This is the least invasive, most informative first step.

Answer

Increase the parallelism of the Flink application.

Answer

Switch to the Kinesis Client Library (KCL) for checkpointing.

Answer

Increase the checkpoint interval to reduce checkpoint frequency.

Question 2

A company needs to transform JSON data from an Amazon S3 bucket into Parquet format and load it into an Amazon Redshift cluster. The transformation includes joining with a reference table stored in Amazon RDS. Which AWS service is BEST suited for this task?

Accepted Answer

AWS Glue ETL job. (AWS Glue ETL job) is the best choice because it natively integrates with S3, RDS, and Redshift. Glue can read JSON from S3, connect to RDS via JDBC to join with the reference table, transform the data to Parquet using its built-in converter, and write directly to Redshift. Option A (AWS Data Pipeline) is older and less integrated for this purpose. Option C (Amazon Athena) can query S3 and convert to Parquet but cannot natively join with RDS without additional services. Option D (Amazon EMR with Spark) is possible but requires more setup and maintenance.

Answer

AWS Data Pipeline

Answer

Amazon Athena

Answer

Amazon EMR with Spark

Question 3

A company uses AWS Glue to run ETL jobs that process data from Amazon S3 and load into Amazon Redshift. The jobs have recently started failing with 'Out of Memory' errors. The data volume has increased 3x in the past month. Which is the MOST effective solution to resolve this issue without redesigning the job?

Accepted Answer

Increase the number of Glue workers (DPUs) for the job.. Glue ETL jobs run on a Spark cluster sized by the number of DPUs (workers). Out-of-memory errors during a 3x data volume increase indicate the existing worker count cannot hold the partitions and shuffle data in memory. Increasing the number of Glue workers adds executor memory and parallelism, which is the most direct fix without redesigning the job.

Answer

Use Amazon Athena instead of Glue for the transformation.

Answer

Rewrite the job to use Spark SQL instead of PySpark.

Answer

Increase the number of partitions in the input S3 data.

Question 4

A data engineer needs to ingest streaming data from thousands of devices sending JSON messages via HTTP POST. The data should be stored in Amazon S3 with minimal latency and also be available for real-time analytics. Which combination of services is MOST appropriate?

Accepted Answer

Amazon API Gateway, Amazon Kinesis Data Streams, and Kinesis Data Firehose.. Amazon API Gateway provides a scalable, managed HTTP endpoint that can receive JSON POST requests from thousands of devices and proxy them directly into Amazon Kinesis Data Streams. Kinesis Data Streams ingests the data with low latency and makes it available for real-time analytics via Kinesis Client Library (KCL) applications, AWS Lambda, or Kinesis Data Analytics. Kinesis Data Firehose then delivers the stream to Amazon S3 with minimal buffering (as low as 60 seconds), satisfying both the real-time and storage requirements.

Answer

Amazon DynamoDB with DynamoDB Streams and Lambda.

Answer

Amazon SQS and AWS Lambda to write to S3.

Answer

AWS Lambda directly writing to S3 via API Gateway.

Question 5

A company has CSV files in an S3 bucket that need to be converted to Parquet and loaded into a Redshift table daily. The transformation is a simple schema mapping without joins. Which AWS Glue feature is BEST suited for this task?

Accepted Answer

AWS Glue ETL job. AWS Glue ETL jobs are purpose-built for extracting data from sources like S3, applying transformations (including schema mapping and format conversion), and loading the results into targets like Redshift. For a simple CSV-to-Parquet conversion with schema mapping and no joins, a Glue ETL job using the built-in transforms or a Spark script is the correct tool. It handles the conversion and load in a single managed job.

Answer

AWS Glue DataBrew

Answer

AWS Glue Workflow

Answer

AWS Glue Crawler

Question 6

A data engineer is designing a data ingestion pipeline to load clickstream data from an Amazon S3 bucket into an Amazon Redshift cluster. The data arrives in 5-minute batches. Which TWO actions should the engineer take to ensure data consistency and avoid duplicates? (Select TWO.)

Accepted Answer

Use the STL_LOAD_ERRORS system table to monitor and resolve load errors.. Option E is correct because loading each 5-minute batch into a staging table and then running a MERGE (or DELETE+INSERT) against the target table lets you match on a business key and update or insert rows idempotently, which prevents duplicate records when batches are retried or overlap. Option C is correct because monitoring STL_LOAD_ERRORS (and STL_LOAD_ERRORS is populated per load with err_code, err_reason, and raw_field_value) lets the engineer detect and resolve failed or partially loaded rows so the pipeline maintains consistent data. Option A is not correct because a SORTKEY only affects physical ordering and query performance; it does not deduplicate rows. Option B is not correct because disabling WLM removes query queue management and does not address duplicates or consistency. Option D is not correct because Redshift distributes data across slices automatically, and forcing data into a single slice would hurt performance without providing deduplication.

Answer

Define a SORTKEY on the target table to improve deduplication.

Answer

Disable workload management (WLM) to maximize resources.

Answer

Load data into a single slice to maintain order.

Question 7

A company is ingesting real-time financial transactions into Amazon Kinesis Data Streams. The data is then consumed by a Kinesis Data Analytics for Apache Flink application that calculates running totals. The application is experiencing high latency and checkpoint failures. Which TWO steps should the engineer take to improve performance and reliability? (Select TWO.)

Accepted Answer

Increase the number of shards in the Kinesis data stream.. Option C is correct because increasing the number of shards in the Kinesis data stream raises the stream's read throughput capacity (each shard supports up to 2 MB/sec read and 1 MB/sec write, or 1,000 records/sec), which relieves the ingestion bottleneck that causes high latency in the Flink application. Option D is correct because increasing KPUs adds parallel processing resources (each KPU provides 1 vCPU and 4 GB memory) to the Kinesis Data Analytics for Apache Flink application, allowing it to consume and process records faster and complete checkpoints more reliably. Option A is not correct because enhanced fan-out is a Kinesis Data Streams consumer feature that gives dedicated 2 MB/sec read throughput per consumer via HTTP/2, but it does not address the Flink application's internal processing or checkpointing bottlenecks. Option B is not correct because reducing the batch size per checkpoint does not improve throughput and can actually increase checkpoint overhead and latency. Option E is not correct because decreasing the checkpoint interval makes checkpoints more frequent, increasing overhead and state I/O rather than reducing state size or improving reliability.

Answer

Enable enhanced fan-out for the Flink application.

Answer

Reduce the batch size of records processed per checkpoint.

Answer

Decrease the checkpoint interval to reduce state size.

Question 8

Refer to the exhibit. A data engineer runs a Glue job manually and receives a ThrottlingException. The engineer checks the job run history and sees a previous failure with the same error. What is the MOST likely cause of the throttling, and which solution is MOST appropriate?

Accepted Answer

Implement retry logic with exponential backoff in the script that calls start-job-run.. A ThrottlingException from AWS Glue typically occurs when the StartJobRun API is called too frequently, exceeding the service's request rate limits. Implementing retry logic with exponential backoff in the calling script is the standard AWS-recommended solution to handle transient throttling gracefully. This allows the job to start successfully after a short delay without manual intervention.

Answer

Increase the number of DPUs for the job to reduce runtime.

Answer

Use AWS Glue reserved capacity to guarantee API throughput.

Answer

Delete old job runs to reduce the number of entries in the job run history.

Question 9

A company runs a data ingestion pipeline that uses AWS Glue to read 500 GB of JSON files from an S3 bucket (s3://raw-data/) every hour. The Glue ETL job transforms the data and writes Parquet files to another S3 bucket (s3://processed-data/). The job is triggered by a time-based CloudWatch Events rule. Recently, the job has started taking over 2 hours to complete, causing delays in downstream processes. The data volume has been consistent, and no changes have been made to the job code or infrastructure. The S3 bucket 's3://raw-data/' receives new files continuously, but the Glue job reads all files in the bucket each run (no incremental processing). The engineer suspects that the job is reprocessing old data. Which action should the engineer take FIRST to reduce the job duration?

Accepted Answer

Enable Glue job bookmarking and configure the job to process only new data.. The Glue job is reprocessing all 500 GB of data every hour because it lacks job bookmarking, which tracks previously processed data. Enabling job bookmarking allows Glue to persist state about which files have already been processed, so subsequent runs only read new files. This directly reduces the input volume and thus the job duration without changing the job logic or infrastructure. Since the data volume is consistent and no code changes were made, the issue is purely due to reprocessing old data, making bookmarking the correct first step.

Answer

Increase the parallelism of the Spark job by repartitioning the data.

Answer

Add partition pruning by modifying the S3 path to include date-based partitions.

Answer

Increase the number of DPUs for the Glue job to 100.

Question 10

A company uses Amazon Kinesis Data Analytics for Apache Flink to process streaming data. The application reads from a Kinesis data stream, performs a 1-minute tumbling window aggregation, and writes results to an S3 bucket. Recently, the application started experiencing checkpoint failures and increasing processing delay. Which action should the engineer take FIRST to diagnose the issue?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

The S3 bucket ARN is incorrectly specified in the application configuration.. The 'ResourceNotFoundException' for the S3 bucket indicates that the application cannot find the bucket. This is most likely due to an incorrectly specified bucket name or ARN in the application configuration. If the bucket name is misspelled or the ARN is malformed, the Kinesis Data Analytics application cannot resolve the bucket resource. Option A is incorrect because missing 's3:PutObject' permission would cause an 'AccessDenied' error, not 'ResourceNotFoundException'. Option C is incorrect because the AWS Region for S3 is determined by the bucket's location, and specifying a wrong region in Flink code would typically result in a different error (e.g., 'IllegalArgumentException' or cross-region access issues) but not 'ResourceNotFoundException'. Option D is incorrect because versioning is unrelated to bucket existence; disabling versioning does not cause a resource not found error.

Answer

The IAM role used by the application does not have s3:PutObject permission.

Answer

The Flink application code specifies the wrong AWS Region for the S3 bucket.

Answer

The S3 bucket has versioning disabled.

Question 11

A data engineer needs to transform JSON data from Amazon S3 into Parquet format using AWS Glue. The data contains nested fields. Which Glue feature should the engineer use to define the schema and handle the nested structure?

Accepted Answer

Use the 'Relationalize' transform in a Glue ETL script.. The Relationalize transform in AWS Glue is specifically designed to convert nested JSON data into a relational format. It flattens nested structures by creating separate tables for arrays and nested objects, which can then be written to Parquet. This is the appropriate feature to handle nested fields when transforming JSON to Parquet.

Answer

Use the 'FindMatches' transform to identify duplicates.

Answer

Use the 'DropFields' transform to remove nested fields.

Answer

Use the 'Spigot' transform to write sample data.

Question 12

A company uses Amazon Kinesis Data Streams to ingest clickstream data. The data must be transformed and stored in Amazon S3 for batch analytics. The engineer wants to use AWS Lambda for transformation. Which TWO configurations are required? (Choose two.)

Accepted Answer

Create an event source mapping from the Kinesis stream to the Lambda function.. Option D is correct because AWS Lambda consumes records from Kinesis Data Streams through an event source mapping, which polls the stream using the enhanced fan-out or standard iterator and invokes the function with batches of records. Option E is correct because the Lambda execution role must include IAM permissions for the required actions, such as kinesis:GetRecords, kinesis:GetShardIterator, kinesis:DescribeStream, and kinesis:ListShards to read the stream, plus s3:PutObject to store the transformed data in Amazon S3. Option A is not required because Kinesis Data Firehose is a separate delivery service; the Lambda function can write directly to S3 using the AWS SDK, and Firehose is not a mandatory intermediary. Option B is incorrect because a Kinesis data stream cannot send records directly to Amazon S3; it requires a consumer such as Lambda or Firehose. Option C is incorrect because an SQS dead-letter queue for Lambda errors is an optional reliability configuration, not a required configuration for transforming and storing the data.

Answer

Configure the Lambda function to write to S3 via Kinesis Data Firehose.

Answer

Configure the Kinesis stream to send records directly to S3.

Answer

Set up an SQS queue as a destination for Lambda errors.

Question 13

A data engineer is building a real-time data pipeline to ingest sensor data from IoT devices. The data is sent to AWS IoT Core, which publishes messages to a Kinesis Data Stream. Each message is about 1 KB in size. The data must be transformed (add a device location field) and then stored in Amazon S3 for long-term analytics. The engineer has set up a Lambda function to transform the records and write to S3. However, the engineer notices that the Lambda function is invoked thousands of times per second, causing high costs and occasional throttling. The Lambda function processes only one record at a time. The engineer wants to reduce the number of Lambda invocations and improve throughput. What should the engineer do?

Accepted Answer

Configure the event source mapping to use a larger batch size and set a batch window.. Configuring the event source mapping with a larger batch size and a batch window allows Lambda to process multiple records per invocation, reducing the number of invocations and costs. This improves throughput and reduces throttling. Option A is incorrect because reducing shards reduces the stream capacity, causing backpressure and potential data loss. Option B is incorrect because increasing memory does not reduce the number of invocations; it only speeds up processing per invocation, but still processes one record at a time. Option C is incorrect because Kinesis Data Firehose can batch records, but it still uses per-record Lambda transformation if you use a Lambda function, or it can use built-in transformations but not the flexible logic described. The most direct solution is to batch records in the existing Lambda function via event source mapping parameters.

Answer

Reduce the number of shards in the Kinesis stream to limit concurrency.

Answer

Increase the Lambda function's memory allocation to improve performance.

Answer

Replace the Lambda function with Amazon Kinesis Data Firehose and use its built-in transformation.

Question 14

A company runs a nightly batch ETL job using AWS Glue to transform data from Amazon RDS for MySQL to Amazon S3. The job reads 100 tables and writes Parquet files partitioned by date. Recently, the job started failing with 'ThrottlingException' from the RDS database. The data volume has increased, and the Glue job is reading large tables without any filtering. The job uses a single Glue job with multiple Spark executors. The engineer needs to reduce the load on the RDS database while maintaining the same processing time. What should the engineer do?

Accepted Answer

Modify the Glue job to use a JDBC connection with a WHERE clause to read only the latest partition.. The ThrottlingException originates from RDS because the Glue job issues full-table JDBC reads across 100 tables with no predicate pushdown. Adding a WHERE clause to read only the latest partition (D) reduces the number of rows scanned and transferred per run, directly lowering the query load on RDS while keeping the same nightly processing window. This is the least disruptive change that preserves the existing architecture and processing time.

Answer

Use AWS DMS to continuously replicate data to S3 and then run Glue on the S3 data.

Answer

Change the job to read all data from RDS into a staging table in S3 first, then transform.

Answer

Increase the number of DPUs to process data faster.

Question 15

A company is ingesting streaming data from IoT devices into Amazon Kinesis Data Streams. The data is then processed by an AWS Lambda function that transforms the records and writes them to an Amazon S3 bucket. Recently, the Lambda function has been timing out and the S3 bucket is not receiving all expected data. The Kinesis stream is not throttling and has sufficient shards. Which step should the company take to resolve this issue?

Accepted Answer

Increase the Lambda function's timeout and memory allocation.. The Lambda function is timing out, indicating that it cannot process the records within the allotted time. Increasing the Lambda function's timeout and memory allocation (Option B) provides more execution time and CPU resources, which can help process larger or more frequent records before timing out. Option A is incorrect because reserved concurrency limits the maximum number of concurrent instances, not the execution duration of a single instance. Option C is incorrect because increasing shards is unnecessary; the stream is not throttling, and more shards do not resolve Lambda timeouts. Option D is incorrect because enhanced fan-out reduces latency for multiple consumers but does not prevent a single Lambda function from timing out due to processing time.

Answer

Increase the Lambda function's reserved concurrency.

Answer

Increase the number of shards in the Kinesis stream.

Answer

Enable enhanced fan-out on the Kinesis stream to reduce latency.