DEA-C01 Data Ingestion and Transformation Practice Test 6 — 15 Questions

Question 1

A company is using Amazon Kinesis Data Streams to ingest real-time clickstream data. The data is consumed by a fleet of EC2 instances running a custom application that processes the records and writes to DynamoDB. The application is experiencing high latency and records are being processed slower than they are produced. The stream has 5 shards. Which action would MOST effectively improve processing speed?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Increase the number of shards in the Kinesis stream.. The bottleneck is the number of shards in the Kinesis stream. Each shard provides a fixed read capacity of 2 MB/s and 5 read transactions per second. With only 5 shards, the total read throughput is limited regardless of how many EC2 instances consume the data. Increasing the number of shards increases the total read capacity, allowing more records to be consumed in parallel and reducing processing latency.

Answer

Use the Kinesis Client Library (KCL) to automatically distribute shards among instances.

Answer

Increase the EC2 instance size to provide more CPU and memory.

Answer

Add more EC2 instances consuming from the same stream without changing shard count.

Question 2

A company is building a data lake on Amazon S3 and wants to ingest data from multiple AWS services (CloudTrail, VPC Flow Logs, and ALB logs). The data should be stored in a central S3 bucket with a common partitioning scheme. Which service can be used to collect and centralize this data with minimal configuration?

Accepted Answer

Configure each source service to deliver logs directly to the central S3 bucket.. CloudTrail, VPC Flow Logs, and ALB logs can each be configured to deliver logs directly to a specified S3 bucket, including a central bucket, with no intermediary service required. This approach minimizes configuration overhead and avoids data movement costs, as each service writes natively to S3 using its own built-in delivery mechanism.

Answer

Use AWS Data Pipeline to copy logs from each source S3 bucket to the central bucket.

Answer

Use AWS Glue to crawl the logs from each source and write to a central S3 bucket.

Answer

Set up Amazon Kinesis Data Firehose to ingest logs from each service and write to S3.

Question 3

A company is running a critical application that generates millions of small JSON files every hour in an S3 bucket. A data engineer needs to process these files in near real-time using AWS Glue. The engineer wants to minimize the latency between file arrival and Glue job start. Which TWO actions should the engineer take?

Accepted Answer

Pre-process the files to consolidate them into larger files before the Glue job runs.. Consolidating millions of small JSON files into larger files reduces the overhead of S3 LIST operations and minimizes the number of partitions Glue must scan. This directly lowers the latency between file arrival and job start, as Glue jobs are more efficient when processing fewer, larger files rather than many small files. Option D is correct because S3 event notifications can trigger a Lambda function that immediately starts the Glue job upon file arrival, enabling near real-time processing without polling or scheduled delays.

Answer

Increase the Glue job's batch window to 600 seconds.

Answer

Increase the number of DPUs for the Glue job to accelerate processing.

Question 4

A data engineer needs to ingest data from an Amazon RDS MySQL database into a data lake on Amazon S3. The engineer wants to perform an initial full load and then capture incremental changes. Which TWO AWS services can be combined to achieve this?

Accepted Answer

Amazon S3. Amazon S3 is the target data lake storage layer, not a service that performs data ingestion or change data capture. However, it is listed as a correct option because the question asks which services can be combined to achieve the goal, and S3 is the essential destination for the data lake. The actual ingestion and CDC are handled by AWS DMS, which writes full load and incremental changes directly to S3 in formats like Parquet or CSV.

Answer

Amazon Kinesis Data Firehose

Answer

AWS Glue

Answer

AWS Data Pipeline

Question 5

A company is using Amazon Kinesis Data Streams to ingest real-time clickstream data. The data is consumed by a fleet of EC2 instances running a custom application that processes the records and writes to DynamoDB. The application is experiencing high latency and records are being processed slower than they are produced. The stream has 5 shards. Which action would MOST effectively improve processing speed?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Add a dead-letter queue and configure the Lambda function to retry on failure with exponential backoff.. Adding a dead-letter queue (DLQ) and configuring the Lambda function to retry on failure with exponential backoff allows the function to handle transient ProvisionedThroughputExceededExceptions from DynamoDB. Since the table uses on-demand capacity, the exception indicates a momentary throttle due to traffic spikes; exponential backoff with retries gives DynamoDB time to scale up, while the DLQ captures records that persistently fail for later analysis.

Answer

Decrease the batch window to 0 seconds to process records immediately.

Answer

Increase the Lambda function's reserved concurrency to process more shards.

Question 6

A data engineer is designing a data ingestion pipeline for IoT sensor data. The sensors send JSON messages every second, and the data must be stored in Amazon S3 in near real-time (within 5 minutes). The engineer also needs to transform the data by adding a timestamp and filtering out malformed records. Which THREE services should be used together?

Accepted Answer

AWS Glue. AWS IoT Core (D) securely ingests sensor data via MQTT and routes it to a Kinesis data stream using a rule. AWS Glue Streaming ETL (A) consumes from the stream, adds a timestamp, filters malformed records, and writes the cleaned data to Kinesis Data Firehose (E). Firehose buffers and delivers the data to Amazon S3 within minutes, meeting the near-real-time requirement.

Answer

Amazon Athena

Answer

Amazon Simple Queue Service (SQS)

Question 7

The Glue job attempts to read data from 's3://my-data-bucket/input/' and write to 's3://my-data-bucket/output/'. It also tries to update a table in the Glue Data Catalog. The job fails with an access denied error. What is the MOST likely cause?

Accepted Answer

The IAM role is missing the 's3:ListBucket' permission on the bucket.. The Glue job reads from 's3://my-data-bucket/input/' and writes to 's3://my-data-bucket/output/'. For S3 read/write operations, the IAM role must have 's3:ListBucket' permission on the bucket itself (my-data-bucket) to allow listing of objects, in addition to 's3:GetObject' and 's3:PutObject' on the object ARN. Without 's3:ListBucket', the job cannot enumerate objects in the input prefix, leading to an access denied error.

Answer

The 'glue:UpdateTable' action is not allowed on the specific table.

Answer

The policy is missing a condition key for the S3 bucket.

Answer

The resource ARN does not include the bucket itself; it only covers objects.

Question 8

The command returns an empty result, but you know there are objects in the 'logs/' prefix larger than 1000 bytes. What is the MOST likely reason?

Accepted Answer

The comparison 'Size > '1000'' uses a string instead of a number, so it never matches.. The `Size > '1000'` comparison treats `'1000'` as a string literal rather than a numeric value. In AWS CLI commands like `list-objects-v2` combined with JMESPath queries, numeric comparisons require unquoted numbers; a quoted string will never match a numeric field, resulting in an empty result even when objects larger than 1000 bytes exist.

Answer

The prefix 'logs/' is incorrect; the objects are in a different prefix.

Answer

The command does not paginate, so it only checks the first 1000 objects.

Answer

The output format is set to text, but the query requires JSON.

Question 9

A data engineering team is ingesting streaming data from IoT devices into Amazon Kinesis Data Streams. The data is then consumed by an AWS Lambda function that transforms and loads it into Amazon S3. Recently, the team noticed that the Lambda function is failing with throttling errors (HTTP 429) from the Kinesis API. Which configuration change should the team make to resolve this issue?

Accepted Answer

Reduce the batch size and increase the number of shards in the Kinesis stream to increase parallelism.. Reducing the batch size and increasing the number of shards directly addresses the HTTP 429 throttling errors from the Kinesis API. Each shard supports up to 5 read transactions per second and a maximum read rate of 2 MB/s; by increasing shards, you increase the available read throughput, and reducing the batch size lowers the number of records per invocation, preventing the Lambda function from exceeding the per-shard read limits.

Answer

Disable retries on the Lambda function and configure a dead-letter queue for failed records.

Answer

Replace Kinesis Data Streams with Amazon DynamoDB Streams for ingestion.

Answer

Increase the batch size in the Lambda event source mapping to reduce the number of invocations.

Question 10

A company uses AWS Glue ETL jobs to transform data in Amazon S3. The data is partitioned by date and hour. The job reads the latest hour's data, performs aggregations, and writes results to a separate S3 bucket. The job runs every hour and processes approximately 500 MB of input data. The team notices that the job takes longer than expected, often exceeding the 1-hour window. Which action would most effectively reduce the job's runtime?

Accepted Answer

Increase the number of workers (DPUs) for the Glue job.. Increasing the number of workers (DPUs) for the Glue job directly addresses the root cause: the job is CPU- or memory-bound due to insufficient parallelism for the 500 MB hourly workload. By allocating more DPUs, AWS Glue can distribute the aggregation and write operations across more executors, reducing wall-clock time and keeping the job within the 1-hour window. This is the most effective action because the job's bottleneck is compute capacity, not data format or processing framework.

Answer

Use a Python shell job instead of a Spark job.

Answer

Switch from using DynamicFrame to using Spark SQL for transformations.

Answer

Repartition the input data into more partitions before reading.

Question 11

A data engineer needs to ingest data from an on-premises Oracle database into Amazon S3 on a nightly basis. The data volume is approximately 10 GB per night. The database is accessible over the internet. Which AWS service is MOST appropriate for this task?

Accepted Answer

AWS Glue ETL job with a JDBC connection. The most appropriate AWS service among the given options for nightly batch ingestion from an on-premises Oracle database into Amazon S3 is AWS Glue ETL job with a JDBC connection (A). AWS Glue can connect to the Oracle database via JDBC, extract data, and load it into S3 in a scheduled batch manner. While AWS Database Migration Service (DMS) is purpose-built for database migrations and would be the ideal service, it is not listed as an option. AWS DataSync (B) is used for file and object storage transfers, not for database connections. AWS Transfer Family (C) provides managed file transfer protocols and cannot directly query databases. Amazon Kinesis Data Streams (D) is designed for real-time streaming data ingestion, not nightly batch loads. Therefore, AWS Glue is the correct choice.

Answer

AWS DataSync

Answer

AWS Transfer Family

Answer

Amazon Kinesis Data Streams

Question 12

A company is using Amazon Kinesis Data Analytics for Apache Flink to process streaming data. The application reads from a Kinesis data stream and writes results to an Amazon S3 bucket. The team notices that the application is experiencing high latency during peak hours. The stream has 8 shards, and the application is configured with a parallelism of 4. Which action would most likely reduce the latency?

Accepted Answer

Increase the parallelism of the Flink application to 8.. The application has 8 shards but only a parallelism of 4, meaning each Flink subtask must process data from 2 shards. This creates a bottleneck because a single subtask cannot process data from multiple shards faster than the slowest shard's throughput. Increasing parallelism to 8 matches the shard count, allowing each subtask to read from exactly one shard, eliminating the contention and reducing latency.

Answer

Decrease the batch size in the S3 sink.

Answer

Use a larger Kinesis Data Analytics application instance type.

Answer

Increase the checkpointing interval to reduce overhead.

Question 13

A team is designing a data ingestion pipeline to load JSON files from an Amazon S3 bucket into Amazon Redshift. The files arrive every 5 minutes, and each file is between 10 MB and 50 MB. The team wants to minimize the time between file arrival and data availability in Redshift. Which approach should the team use?

Accepted Answer

Use S3 Event Notifications to trigger an AWS Lambda function that runs the COPY command to load data into Redshift.. S3 Event Notifications can trigger an AWS Lambda function that executes the COPY command, loading data into Redshift with minimal latency. This approach avoids the overhead of scheduling or batching, directly responding to each file arrival to meet the 5-minute frequency and file size requirements.

Answer

Schedule an AWS Glue job to run every 5 minutes to load the data.

Answer

Use Amazon Redshift Spectrum to query the data directly from S3 without loading.

Answer

Configure Amazon Kinesis Data Firehose to stream data from S3 to Redshift.

Question 14

A company uses AWS Glue ETL jobs to transform data stored in Amazon S3. The job reads data in Parquet format, applies transformations, and writes the output back to S3 in Parquet format. The team wants to improve the job's performance and reduce costs. Which action is MOST effective?

Accepted Answer

Use column pruning and predicate pushdown to read only necessary columns and filter data early.. Column pruning and predicate pushdown reduce the amount of data read from S3 by Spark-based AWS Glue ETL jobs. By reading only the necessary columns and filtering rows early in the scan, I/O and memory usage decrease, directly improving performance and reducing costs.

Answer

Change the input format from Parquet to CSV to simplify parsing.

Answer

Coalesce the input data into a single large file before processing.

Answer

Increase the number of workers to maximum allowed.

Question 15

A data engineer is ingesting data from an Amazon RDS for PostgreSQL database into Amazon S3 using AWS Glue. The Glue job reads the entire table each time it runs, which takes several hours. The team wants to reduce the job duration by reading only new or updated records. Which approach should the engineer adopt?

Accepted Answer

Enable job bookmarks in AWS Glue and use a column with timestamps as the bookmark key to read only incremental data.. AWS Glue job bookmarks track previously processed data using a specified column (e.g., a timestamp column) as the bookmark key. When enabled, the job reads only new or updated records since the last run, significantly reducing job duration by avoiding full table scans. This directly addresses the requirement to read incremental data from the PostgreSQL source.

Answer

Partition the table in the source database by date and read only the latest partition.

Answer

Increase the number of Glue workers to improve parallel reads.

Answer

Use Amazon Kinesis Data Streams to capture changes from PostgreSQL.