DEA-C01 Data Ingestion and Transformation Practice Test 38 — 15 Questions

Question 1

A company is streaming IoT sensor data to Amazon Kinesis Data Streams. The data is JSON with a schema that changes occasionally. They want to load the data into Amazon S3 in Parquet format partitioned by date and sensor_id. Which approach is MOST cost-effective and operationally efficient?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Use Amazon Kinesis Data Firehose with a schema from AWS Glue Data Catalog to convert to Parquet and enable dynamic partitioning by date and sensor_id.. Amazon Kinesis Data Firehose can directly convert incoming JSON data to Parquet using a schema from AWS Glue Data Catalog, and it supports dynamic partitioning by date and sensor_id without requiring custom code. This is the most cost-effective and operationally efficient approach as it is a fully managed service that handles buffering, conversion, and partitioning automatically. Option A (EMR) is overkill for this use case and adds operational complexity. Option B (Lambda) would require additional transformation logic and is less efficient for high-throughput streaming. Option C (custom KCL application on EC2) requires ongoing management and is not as simple as using Firehose.

Answer

Use Amazon EMR to read from Kinesis Data Streams and write to S3 in Parquet format.

Answer

Use a Lambda function to transform records to Parquet and write to S3.

Answer

Use a custom Kinesis Client Library application on EC2 to buffer and write Parquet files to S3.

Question 2

A data engineer needs to ingest daily CSV files from an external FTP server into Amazon S3. The files are 5 GB each. Which service is MOST suitable to automate this ingestion?

Accepted Answer

AWS DataSync. AWS DataSync is the most suitable service for automated, scheduled transfers of large files from an external FTP server to Amazon S3. It supports both one-time and recurring transfers and can handle high throughput for files up to 5 GB. S3 Transfer Acceleration only speeds up uploads to S3, not transfers from FTP servers. AWS Glue is an ETL service, not a file transfer tool. AWS AppSync is for real-time APIs, not batch file ingestion.

Answer

AWS AppSync

Answer

Amazon S3 Transfer Acceleration

Answer

AWS Glue

Question 3

A company uses Amazon Kinesis Data Analytics for Apache Flink to process streaming data. The application reads from a Kinesis data stream, performs a 1-minute tumbling window aggregation, and writes results to an S3 bucket. Recently, the application started experiencing checkpoint failures and increasing processing delay. Which action should the engineer take FIRST to diagnose the issue?

Accepted Answer

Monitor CPU and memory utilization of the Flink application using Amazon CloudWatch metrics.. Checkpoint failures are often due to insufficient resources (CPU/memory) for the Flink job. Monitoring CPU and memory utilization via CloudWatch metrics directly helps identify resource bottlenecks. Option A (increasing parallelism) is a tuning step that might help but is not diagnostic. Option C (switching to KCL) is not relevant for Flink checkpointing. Option D (increasing checkpoint interval) might reduce checkpoint frequency but does not address the root cause of checkpoint failures. The first step is to check resource utilization.

Answer

Increase the parallelism of the Flink application.

Answer

Switch to the Kinesis Client Library (KCL) for checkpointing.

Answer

Increase the checkpoint interval to reduce checkpoint frequency.

Question 4

A company needs to transform JSON data from an Amazon S3 bucket into Parquet format and load it into an Amazon Redshift cluster. The transformation includes joining with a reference table stored in Amazon RDS. Which AWS service is BEST suited for this task?

Accepted Answer

AWS Glue ETL job. (AWS Glue ETL job) is the best choice because it natively integrates with S3, RDS, and Redshift. Glue can read JSON from S3, connect to RDS via JDBC to join with the reference table, transform the data to Parquet using its built-in converter, and write directly to Redshift. Option A (AWS Data Pipeline) is older and less integrated for this purpose. Option C (Amazon Athena) can query S3 and convert to Parquet but cannot natively join with RDS without additional services. Option D (Amazon EMR with Spark) is possible but requires more setup and maintenance.

Answer

AWS Data Pipeline

Answer

Amazon Athena

Answer

Amazon EMR with Spark

Question 5

A data engineer is tasked with ingesting on-premises database snapshots (full load) into Amazon S3 on a daily basis. The database is PostgreSQL and the snapshot size is 50 GB. The network link is 1 Gbps. Which approach is the MOST time-efficient and cost-effective?

Accepted Answer

Use AWS Database Migration Service (DMS) with S3 as target.. AWS Database Migration Service (DMS) is a managed service that efficiently performs full loads from on-premises PostgreSQL to Amazon S3, handling schema conversion and data transfer automatically. For a 50 GB snapshot over a 1 Gbps link, the transfer takes approximately 7 minutes, making Snowball (Option B) overkill and more expensive. Option C (AWS CLI) would require manual steps and is less efficient for database snapshots. Option D (Lambda + pg_dump) adds unnecessary complexity and potential performance issues. Therefore, DMS is the most time-efficient and cost-effective approach.

Answer

Use AWS Snowball Edge to transfer the snapshot.

Answer

Use AWS CLI to copy the snapshot file directly to S3.

Answer

Write a Lambda function to run pg_dump and upload to S3.

Question 6

A company uses AWS Glue to run ETL jobs that process data from Amazon S3 and load into Amazon Redshift. The jobs have recently started failing with 'Out of Memory' errors. The data volume has increased 3x in the past month. Which is the MOST effective solution to resolve this issue without redesigning the job?

Accepted Answer

Increase the number of Glue workers (DPUs) for the job.. To increase the number of Glue workers (DPUs). This provides more memory and processing capacity to handle the increased data volume, directly resolving the 'Out of Memory' errors. Increasing S3 partitions (option D) may improve parallelism but does not directly increase memory for the Glue job. Using Spark SQL (option C) instead of PySpark does not necessarily address memory issues. Switching to Athena (option A) would change the architecture and is not a fix for the existing Glue job.

Answer

Use Amazon Athena instead of Glue for the transformation.

Answer

Rewrite the job to use Spark SQL instead of PySpark.

Answer

Increase the number of partitions in the input S3 data.

Question 7

A data engineer needs to ingest streaming data from thousands of devices sending JSON messages via HTTP POST. The data should be stored in Amazon S3 with minimal latency and also be available for real-time analytics. Which combination of services is MOST appropriate?

Accepted Answer

Amazon API Gateway, Amazon Kinesis Data Streams, and Kinesis Data Firehose.. It combines Amazon API Gateway to ingest HTTP POST messages, Amazon Kinesis Data Streams for real-time analytics, and Kinesis Data Firehose to deliver the data to Amazon S3 with minimal latency. Option A (DynamoDB with Streams) is meant for database change tracking, not direct HTTP streaming. Option B (SQS + Lambda) introduces latency and lacks native streaming analytics. Option C (Lambda directly via API Gateway) does not provide the buffering and streaming analytics capabilities needed.

Answer

Amazon DynamoDB with DynamoDB Streams and Lambda.

Answer

Amazon SQS and AWS Lambda to write to S3.

Answer

AWS Lambda directly writing to S3 via API Gateway.

Question 8

A company has CSV files in an S3 bucket that need to be converted to Parquet and loaded into a Redshift table daily. The transformation is a simple schema mapping without joins. Which AWS Glue feature is BEST suited for this task?

Accepted Answer

AWS Glue ETL job. (AWS Glue ETL job) is the best suited because it can convert CSV to Parquet and load into Redshift daily. Option B (DataBrew) is a visual data preparation tool, not ideal for automated daily jobs. Option C (Workflow) orchestrates multiple jobs but does not perform transformation. Option D (Crawler) only discovers schema and catalogs data, not transform.

Answer

AWS Glue DataBrew

Answer

AWS Glue Workflow

Answer

AWS Glue Crawler

Question 9

A company is using Amazon Kinesis Data Firehose to deliver streaming data to an S3 bucket. The data includes a timestamp field. They want to partition the S3 objects by hour dynamically. The Firehose delivery stream is configured with a prefix like 'year=!{timestamp:yyyy}/month=!{timestamp:MM}/day=!{timestamp:dd}/hour=!{timestamp:HH}/'. However, the objects are not being partitioned as expected; all files end up in a single partition. What is the MOST likely cause?

Accepted Answer

Dynamic partitioning is not enabled in the Firehose delivery stream configuration.. Amazon Kinesis Data Firehose requires dynamic partitioning to be explicitly enabled in the delivery stream configuration in order to use custom partitioning keys and create partition prefixes dynamically. Without enabling dynamic partitioning, the timestamp expressions in the prefix are treated as static strings, resulting in a single partition. Option B is incorrect because buffer size affects delivery frequency, not partitioning behavior; even with large buffers, partitioning would still occur if dynamic partitioning were enabled. Option C is incorrect because time zone differences would cause incorrect hours, not a single partition; all files would still be distributed across hours. Option D is incorrect because IAM permission issues would cause write failures, not mispartitioning; the objects would simply not be delivered.

Answer

The buffer size is set too large, delaying file delivery.

Answer

The timestamp is in UTC but the prefix uses local time.

Answer

The IAM role for Firehose lacks permissions to write to S3 with dynamic prefixes.

Question 10

A data engineer is designing a data ingestion pipeline to load clickstream data from an Amazon S3 bucket into an Amazon Redshift cluster. The data arrives in 5-minute batches. Which TWO actions should the engineer take to ensure data consistency and avoid duplicates? (Select TWO.)

Accepted Answer

Use the STL_LOAD_ERRORS system table to monitor and resolve load errors.. Options C and E are correct. Using the STL_LOAD_ERRORS system table (option C) allows the engineer to monitor and resolve load errors, ensuring data consistency. Using a staging table with a MERGE operation (option E) helps avoid duplicates by performing an upsert. Option A (defining a SORTKEY) improves query performance but does not affect deduplication. Option B (disabling WLM) would reduce resource management and is not recommended. Option D (loading into a single slice) reduces performance and does not maintain order for consistency.

Answer

Define a SORTKEY on the target table to improve deduplication.

Answer

Disable workload management (WLM) to maximize resources.

Answer

Load data into a single slice to maintain order.

Question 11

A company is ingesting real-time financial transactions into Amazon Kinesis Data Streams. The data is then consumed by a Kinesis Data Analytics for Apache Flink application that calculates running totals. The application is experiencing high latency and checkpoint failures. Which TWO steps should the engineer take to improve performance and reliability? (Select TWO.)

Accepted Answer

Increase the number of shards in the Kinesis data stream.. Options C and D are correct. Increasing the number of shards (C) increases the throughput and parallelism of the stream, reducing latency. Increasing KPUs (D) provides more compute resources for the Flink application, helping to prevent checkpoint failures. Option A (enhanced fan-out) is designed for multiple consumers, not for a single Flink job. Option B (reducing batch size) may not improve overall throughput. Option E (decreasing checkpoint interval) could lead to more frequent checkpointing, increasing overhead and potentially causing more failures.

Answer

Enable enhanced fan-out for the Flink application.

Answer

Reduce the batch size of records processed per checkpoint.

Answer

Decrease the checkpoint interval to reduce state size.

Question 12

A company uses AWS Glue to catalog and transform data in Amazon S3. The Glue ETL jobs are failing intermittently with 'ThrottlingException' errors. Which THREE actions can help mitigate this issue? (Select THREE.)

Accepted Answer

Implement exponential backoff and retry in the Glue job code.. Options A, C, and D are correct. Implementing exponential backoff and retry (A) handles transient ThrottlingException errors by pacing retries. Requesting a service quota increase for the Glue API (C) raises the maximum allowed requests per second. Enabling job bookmarking (D) reduces the amount of data reprocessed, thereby lowering the number of API calls made to Glue. Option B (increasing DPUs) improves job performance but does not directly address API throttling. Option E (switching to Spark SQL) does not change the underlying API call frequency.

Answer

Increase the number of DPUs for the Glue job.

Answer

Switch from PySpark to Spark SQL.

Question 13

Refer to the exhibit. A data engineer runs a Glue job manually and receives a ThrottlingException. The engineer checks the job run history and sees a previous failure with the same error. What is the MOST likely cause of the throttling, and which solution is MOST appropriate?

Accepted Answer

Implement retry logic with exponential backoff in the script that calls start-job-run.. The error 'ThrottlingException' indicates the Glue API rate limit has been exceeded. The most likely cause is making too many API calls, such as multiple concurrent job starts. Implementing retry logic with exponential backoff in the script that calls StartJobRun will handle transient throttling. Option A (increasing DPUs) does not affect API rate limits. Option C (using reserved capacity) reserves compute capacity, not API throughput, so it does not prevent throttling. Option D (deleting old job runs) does not reduce API calls; it only affects job history visibility.

Answer

Increase the number of DPUs for the job to reduce runtime.

Answer

Use AWS Glue reserved capacity to guarantee API throughput.

Answer

Delete old job runs to reduce the number of entries in the job run history.

Question 14

A company runs a data ingestion pipeline that uses AWS Glue to read 500 GB of JSON files from an S3 bucket (s3://raw-data/) every hour. The Glue ETL job transforms the data and writes Parquet files to another S3 bucket (s3://processed-data/). The job is triggered by a time-based CloudWatch Events rule. Recently, the job has started taking over 2 hours to complete, causing delays in downstream processes. The data volume has been consistent, and no changes have been made to the job code or infrastructure. The S3 bucket 's3://raw-data/' receives new files continuously, but the Glue job reads all files in the bucket each run (no incremental processing). The engineer suspects that the job is reprocessing old data. Which action should the engineer take FIRST to reduce the job duration?

Accepted Answer

Enable Glue job bookmarking and configure the job to process only new data.. Enabling job bookmarking in Glue allows the job to process only new files since the last run, dramatically reducing processing time. Option D (increasing DPUs) would help with resource constraints but does not address the root cause of reprocessing old data. Option B (increasing parallelism) may help but not as much as eliminating reprocessing. Option C (using partition pruning) assumes the data is partitioned, but the stem says all files are read; partitioning might not help if files are not organized by time. The most impactful first step is to enable bookmarking.

Answer

Increase the parallelism of the Spark job by repartitioning the data.

Answer

Add partition pruning by modifying the S3 path to include date-based partitions.

Answer

Increase the number of DPUs for the Glue job to 100.

Question 15

A company is using Amazon Kinesis Data Firehose to ingest clickstream data from a website into an S3 bucket. The data is then analyzed using Amazon Athena. Recently, the company noticed that Athena queries are returning incomplete results for the last 30 minutes of data. The Firehose delivery stream is configured to buffer data for 60 seconds or 5 MB before delivering to S3. The S3 bucket has a lifecycle policy that transitions objects to Amazon S3 Glacier after 30 days. The IAM role for Firehose has permissions to write to S3 and access a CloudWatch Logs group. The engineer checks the Firehose monitoring and sees that the delivery rate is healthy, but the 'S3.Bytes' metric shows a spike in the last hour. The 'BackupToS3.Bytes' metric is zero. What is the MOST likely cause of the missing data?

Accepted Answer

The data is still being buffered in Firehose and has not yet been delivered to S3.. The data is still being buffered in Firehose and has not yet been delivered to S3. The buffer settings (60 seconds or 5 MB) mean data can be held in the buffer for up to 60 seconds before being written to S3. For the last 30 minutes, some data may still be in the buffer and not yet delivered. Athena queries only see data that has been delivered to S3. Option A (lifecycle policy) would not affect recent data. Option B (backup) is unrelated and the 'BackupToS3.Bytes' metric is zero, indicating backup is not active. Option D (IAM permissions) would cause errors, not missing data.

Answer

The lifecycle policy is transitioning data before Athena can query it.

Answer

The backup is enabled and data is being sent to the backup bucket instead.

Answer

The IAM role for Firehose does not have permissions to write to the S3 bucket.