DEA-C01 Data Ingestion and Transformation Practice Test 38 — 15 Questions

Question 1

A company uses Amazon Kinesis Data Analytics for Apache Flink to process streaming data. The application reads from a Kinesis data stream, performs a 1-minute tumbling window aggregation, and writes results to an S3 bucket. Recently, the application started experiencing checkpoint failures and increasing processing delay. Which action should the engineer take FIRST to diagnose the issue?

Accepted Answer

Monitor CPU and memory utilization of the Flink application using Amazon CloudWatch metrics.. Option B is correct because checkpoint failures are often due to insufficient resources (CPU/memory) for the Flink job. Monitoring CPU and memory utilization via CloudWatch metrics directly helps identify resource bottlenecks. Option A (checkpoint interval) might help but is not diagnostic. Option C (parallelism) is a tuning step. Option D (KCL) is not relevant for Flink. The first step is to check resource utilization.

Answer

Increase the parallelism of the Flink application.

Answer

Switch to the Kinesis Client Library (KCL) for checkpointing.

Answer

Increase the checkpoint interval to reduce checkpoint frequency.

Question 2

A company needs to transform JSON data from an Amazon S3 bucket into Parquet format and load it into an Amazon Redshift cluster. The transformation includes joining with a reference table stored in Amazon RDS. Which AWS service is BEST suited for this task?

Accepted Answer

AWS Glue ETL job. Option D is correct because AWS Glue ETL jobs can read from S3, connect to RDS via JDBC, perform joins and transformations, and write to Redshift. Option A (Athena) can query S3 but cannot join with RDS natively. Option B (EMR) is possible but more complex to set up. Option C (Data Pipeline) is older and less integrated.

Answer

AWS Data Pipeline

Answer

Amazon Athena

Answer

Amazon EMR with Spark

Question 3

A data engineer is tasked with ingesting on-premises database snapshots (full load) into Amazon S3 on a daily basis. The database is PostgreSQL and the snapshot size is 50 GB. The network link is 1 Gbps. Which approach is the MOST time-efficient and cost-effective?

Accepted Answer

Use AWS Database Migration Service (DMS) with S3 as target.. Option B is correct because AWS DMS can perform a full load from on-premises PostgreSQL to S3 efficiently, and it's a managed service. Option A (CLI) would be manual and slow. Option C (Snowball) is for larger datasets or slow networks; here 50 GB at 1 Gbps takes ~7 minutes, so Snowball is overkill. Option D (Lambda + pg_dump) adds custom code and complexity.

Answer

Use AWS Snowball Edge to transfer the snapshot.

Answer

Use AWS CLI to copy the snapshot file directly to S3.

Answer

Write a Lambda function to run pg_dump and upload to S3.

Question 4

A company uses AWS Glue to run ETL jobs that process data from Amazon S3 and load into Amazon Redshift. The jobs have recently started failing with 'Out of Memory' errors. The data volume has increased 3x in the past month. Which is the MOST effective solution to resolve this issue without redesigning the job?

Accepted Answer

Increase the number of Glue workers (DPUs) for the job.. Option A is correct because increasing the number of workers (DPUs) provides more memory and processing capacity. Option B (increasing S3 partitions) may help with parallelism but not directly with memory. Option C (using Spark SQL) is not a direct fix. Option D (switching to Athena) changes the architecture.

Answer

Use Amazon Athena instead of Glue for the transformation.

Answer

Rewrite the job to use Spark SQL instead of PySpark.

Answer

Increase the number of partitions in the input S3 data.

Question 5

A data engineer needs to ingest streaming data from thousands of devices sending JSON messages via HTTP POST. The data should be stored in Amazon S3 with minimal latency and also be available for real-time analytics. Which combination of services is MOST appropriate?

Accepted Answer

Amazon API Gateway, Amazon Kinesis Data Streams, and Kinesis Data Firehose.. Option C is correct because API Gateway ingests HTTP POST, sends to Kinesis Data Streams for real-time consumption, and Firehose delivers to S3. Option A (SQS) is not ideal for real-time streaming. Option B (DynamoDB Streams) is for database changes. Option D (Lambda + S3) lacks real-time analytics capability.

Answer

Amazon DynamoDB with DynamoDB Streams and Lambda.

Answer

Amazon SQS and AWS Lambda to write to S3.

Answer

AWS Lambda directly writing to S3 via API Gateway.

Question 6

A company has CSV files in an S3 bucket that need to be converted to Parquet and loaded into a Redshift table daily. The transformation is a simple schema mapping without joins. Which AWS Glue feature is BEST suited for this task?

Accepted Answer

AWS Glue ETL job. Option B is correct because Glue ETL job is the standard way to transform and load data. Option A (Crawler) only catalogs data. Option C (DataBrew) is a visual tool, but the question asks for Glue feature for automated daily jobs. Option D (Workflows) orchestrates jobs but does not perform transformation.

Answer

AWS Glue DataBrew

Answer

AWS Glue Workflow

Answer

AWS Glue Crawler

Question 7

A company is using Amazon Kinesis Data Firehose to deliver streaming data to an S3 bucket. The data includes a timestamp field. They want to partition the S3 objects by hour dynamically. The Firehose delivery stream is configured with a prefix like 'year=!{timestamp:yyyy}/month=!{timestamp:MM}/day=!{timestamp:dd}/hour=!{timestamp:HH}/'. However, the objects are not being partitioned as expected; all files end up in a single partition. What is the MOST likely cause?

Accepted Answer

Dynamic partitioning is not enabled in the Firehose delivery stream configuration.. Option D is correct because dynamic partitioning requires a separate configuration (enabling dynamic partitioning and specifying the partitioning keys). The prefix with timestamp expressions is used for S3 prefix customization but not for dynamic partitioning. Option A (time zone) could cause wrong hour but not single partition. Option B (buffer size) affects file size. Option C (IAM) would cause errors, not mispartitioning. The key is that dynamic partitioning must be explicitly enabled.

Answer

The buffer size is set too large, delaying file delivery.

Answer

The timestamp is in UTC but the prefix uses local time.

Answer

The IAM role for Firehose lacks permissions to write to S3 with dynamic prefixes.

Question 8

A data engineer is designing a data ingestion pipeline to load clickstream data from an Amazon S3 bucket into an Amazon Redshift cluster. The data arrives in 5-minute batches. Which TWO actions should the engineer take to ensure data consistency and avoid duplicates? (Select TWO.)

Accepted Answer

Use the STL_LOAD_ERRORS system table to monitor and resolve load errors.. Options B and D are correct. Using the STL_LOAD_ERRORS table helps identify errors, and using a staging table with a MERGE/upsert pattern ensures idempotent loads. Option A (disabling WLM) is unrelated. Option C (loading into a single slice) reduces performance. Option E (using SORTKEY) improves query performance but not consistency.

Answer

Define a SORTKEY on the target table to improve deduplication.

Answer

Disable workload management (WLM) to maximize resources.

Answer

Load data into a single slice to maintain order.

Question 9

A company is ingesting real-time financial transactions into Amazon Kinesis Data Streams. The data is then consumed by a Kinesis Data Analytics for Apache Flink application that calculates running totals. The application is experiencing high latency and checkpoint failures. Which TWO steps should the engineer take to improve performance and reliability? (Select TWO.)

Accepted Answer

Increase the number of shards in the Kinesis data stream.. Options B and D are correct. Increasing the number of shards increases throughput and parallelism, helping reduce latency. Increasing the number of KPUs (Kinesis Processing Units) for the Flink application provides more compute resources, addressing checkpoint failures. Option A (decreasing checkpoint interval) may increase checkpoint overhead. Option C (using Fan-Out) is for multiple consumers, not for a single Flink job. Option E (reducing batch size) may not help with overall throughput.

Answer

Enable enhanced fan-out for the Flink application.

Answer

Reduce the batch size of records processed per checkpoint.

Answer

Decrease the checkpoint interval to reduce state size.

Question 10

A company uses AWS Glue to catalog and transform data in Amazon S3. The Glue ETL jobs are failing intermittently with 'ThrottlingException' errors. Which THREE actions can help mitigate this issue? (Select THREE.)

Accepted Answer

Implement exponential backoff and retry in the Glue job code.. Options A, C, and D are correct. Implementing retry logic with exponential backoff handles transient throttling. Increasing the service quota for Glue API calls if the limit is reached. Using Glue job bookmarking reduces reprocessing, thus lowering API calls. Option B (increasing DPUs) addresses resource issues but not API throttling. Option E (changing to Spark SQL) does not affect API calls.

Answer

Increase the number of DPUs for the Glue job.

Answer

Switch from PySpark to Spark SQL.

Question 11

Refer to the exhibit. A data engineer runs a Glue job manually and receives a ThrottlingException. The engineer checks the job run history and sees a previous failure with the same error. What is the MOST likely cause of the throttling, and which solution is MOST appropriate?

Accepted Answer

Implement retry logic with exponential backoff in the script that calls start-job-run.. Option B is correct because the error 'Rate exceeded' indicates the Glue API rate limit has been reached. The most likely cause is multiple concurrent job runs or too many API calls. Implementing retry logic with exponential backoff in the job or script that triggers the job will handle transient throttling. Option A (increasing DPUs) does not affect API rate. Option C (deleting old job runs) might free up quota but is not the best practice. Option D (using reserved capacity) is not applicable for Glue API calls.

Answer

Increase the number of DPUs for the job to reduce runtime.

Answer

Use AWS Glue reserved capacity to guarantee API throughput.

Answer

Delete old job runs to reduce the number of entries in the job run history.

Question 12

A company runs a data ingestion pipeline that uses AWS Glue to read 500 GB of JSON files from an S3 bucket (s3://raw-data/) every hour. The Glue ETL job transforms the data and writes Parquet files to another S3 bucket (s3://processed-data/). The job is triggered by a time-based CloudWatch Events rule. Recently, the job has started taking over 2 hours to complete, causing delays in downstream processes. The data volume has been consistent, and no changes have been made to the job code or infrastructure. The S3 bucket 's3://raw-data/' receives new files continuously, but the Glue job reads all files in the bucket each run (no incremental processing). The engineer suspects that the job is reprocessing old data. Which action should the engineer take FIRST to reduce the job duration?

Accepted Answer

Enable Glue job bookmarking and configure the job to process only new data.. Option C is correct because enabling job bookmarking in Glue allows the job to process only new files since the last run, dramatically reducing processing time. Option A (increasing DPUs) would help with resource constraints but does not address the root cause of reprocessing old data. Option B (increasing parallelism) may help but not as much as eliminating reprocessing. Option D (using partition pruning) assumes the data is partitioned, but the stem says all files are read; partitioning might not help if files are not organized by time. The most impactful first step is to enable bookmarking.

Answer

Increase the parallelism of the Spark job by repartitioning the data.

Answer

Add partition pruning by modifying the S3 path to include date-based partitions.

Answer

Increase the number of DPUs for the Glue job to 100.

Question 13

A company is using Amazon Kinesis Data Firehose to ingest clickstream data from a website into an S3 bucket. The data is then analyzed using Amazon Athena. Recently, the company noticed that Athena queries are returning incomplete results for the last 30 minutes of data. The Firehose delivery stream is configured to buffer data for 60 seconds or 5 MB before delivering to S3. The S3 bucket has a lifecycle policy that transitions objects to Amazon S3 Glacier after 30 days. The IAM role for Firehose has permissions to write to S3 and access a CloudWatch Logs group. The engineer checks the Firehose monitoring and sees that the delivery rate is healthy, but the 'S3.Bytes' metric shows a spike in the last hour. The 'BackupToS3.Bytes' metric is zero. What is the MOST likely cause of the missing data?

Accepted Answer

The data is still being buffered in Firehose and has not yet been delivered to S3.. Option A is correct. The buffer settings (60 seconds or 5 MB) mean that data may be held in the buffer for up to 60 seconds before being written to S3. For the last 30 minutes, some data may still be in the buffer and not yet delivered. Athena queries only see data that has been delivered. Option B (lifecycle policy) would not affect recent data. Option C (IAM permissions) would cause errors, not missing data. Option D (backup) is unrelated.

Answer

The lifecycle policy is transitioning data before Athena can query it.

Answer

The backup is enabled and data is being sent to the backup bucket instead.

Answer

The IAM role for Firehose does not have permissions to write to the S3 bucket.

Question 14

A company uses AWS Glue to process CSV files stored in Amazon S3. The data pipeline runs daily, but recently some jobs have failed with a 'MemoryError'. The data volume has grown from 1 GB to 10 GB per day. What is the MOST cost-effective solution to resolve this issue?

Accepted Answer

Increase the number of DPUs (Data Processing Units) allocated to the Glue job.. Option C is correct because increasing the number of DPUs allows Glue to handle larger data volumes without changing the job logic. Option A is wrong because switching to a different file format does not address memory limits. Option B is wrong because moving to Amazon EMR is more complex and costly for a simple ETL job. Option D is wrong because additional memory is provisioned via DPUs, not by changing the worker type to 'G.2X' which is for machine learning.

Answer

Change the Glue worker type from Standard to G.2X.

Answer

Convert the CSV files to Parquet format using an S3 batch operation.

Answer

Migrate the job to Amazon EMR with a larger cluster.

Question 15

A data engineer is troubleshooting a Kinesis Data Firehose delivery stream that sends data to an Amazon S3 bucket. The delivery stream has a buffer size of 5 MB and a buffer interval of 60 seconds. The data ingestion rate is 2 MB per second. The engineer notices that S3 objects are created every 60 seconds but each object is only about 2 MB. What should the engineer do to reduce the number of small S3 objects?

Accepted Answer

Increase the buffer size to 10 MB.. Option A is correct because increasing the buffer size to 10 MB will allow the stream to buffer more data before writing to S3, resulting in larger objects. Option B is wrong because decreasing the buffer interval would create objects more frequently, making the problem worse. Option C is wrong because switching to Kinesis Data Streams does not solve the buffering issue. Option D is wrong because reducing the buffer size would create even smaller objects.

Answer

Decrease the buffer interval to 30 seconds.

Answer

Reduce the buffer size to 2 MB.

Answer

Switch to Kinesis Data Streams and use a Lambda function to write to S3.