DEA-C01 Data Ingestion and Transformation Practice Test 3 — 15 Questions

Question 1

The command returns an empty result, but you know there are objects in the 'logs/' prefix larger than 1000 bytes. What is the MOST likely reason?

Accepted Answer

The comparison 'Size > '1000'' uses a string instead of a number, so it never matches.. The `Size > '1000'` comparison treats `'1000'` as a string literal rather than a numeric value. In AWS CLI commands like `list-objects-v2` combined with JMESPath queries, numeric comparisons require unquoted numbers; a quoted string will never match a numeric field, resulting in an empty result even when objects larger than 1000 bytes exist.

Answer

The prefix 'logs/' is incorrect; the objects are in a different prefix.

Answer

The command does not paginate, so it only checks the first 1000 objects.

Answer

The output format is set to text, but the query requires JSON.

Question 2

A data engineering team is ingesting streaming data from IoT devices into Amazon Kinesis Data Streams. The data is then consumed by an AWS Lambda function that transforms and loads it into Amazon S3. Recently, the team noticed that the Lambda function is failing with throttling errors (HTTP 429) from the Kinesis API. Which configuration change should the team make to resolve this issue?

Accepted Answer

Reduce the batch size and increase the number of shards in the Kinesis stream to increase parallelism.. Reducing the batch size and increasing the number of shards directly addresses the HTTP 429 throttling errors from the Kinesis API. Each shard supports up to 5 read transactions per second and a maximum read rate of 2 MB/s; by increasing shards, you increase the available read throughput, and reducing the batch size lowers the number of records per invocation, preventing the Lambda function from exceeding the per-shard read limits.

Answer

Disable retries on the Lambda function and configure a dead-letter queue for failed records.

Answer

Replace Kinesis Data Streams with Amazon DynamoDB Streams for ingestion.

Answer

Increase the batch size in the Lambda event source mapping to reduce the number of invocations.

Question 3

A company uses AWS Glue ETL jobs to transform data in Amazon S3. The data is partitioned by date and hour. The job reads the latest hour's data, performs aggregations, and writes results to a separate S3 bucket. The job runs every hour and processes approximately 500 MB of input data. The team notices that the job takes longer than expected, often exceeding the 1-hour window. Which action would most effectively reduce the job's runtime?

Accepted Answer

Increase the number of workers (DPUs) for the Glue job.. Increasing the number of workers (DPUs) for the Glue job directly addresses the root cause: the job is CPU- or memory-bound due to insufficient parallelism for the 500 MB hourly workload. By allocating more DPUs, AWS Glue can distribute the aggregation and write operations across more executors, reducing wall-clock time and keeping the job within the 1-hour window. This is the most effective action because the job's bottleneck is compute capacity, not data format or processing framework.

Answer

Use a Python shell job instead of a Spark job.

Answer

Switch from using DynamicFrame to using Spark SQL for transformations.

Answer

Repartition the input data into more partitions before reading.

Question 4

A data engineer needs to ingest data from an on-premises Oracle database into Amazon S3 on a nightly basis. The data volume is approximately 10 GB per night. The database is accessible over the internet. Which AWS service is MOST appropriate for this task?

Accepted Answer

AWS Glue ETL job with a JDBC connection. AWS Glue ETL with a JDBC connection is the most appropriate service because it can connect directly to an on-premises Oracle database over the internet, extract the data on a nightly schedule, and write it to Amazon S3. Glue handles schema discovery, transformation, and scheduling natively, making it ideal for this recurring batch ingestion pattern.

Answer

AWS DataSync

Answer

AWS Transfer Family

Answer

Amazon Kinesis Data Streams

Question 5

A company is using Amazon Kinesis Data Analytics for Apache Flink to process streaming data. The application reads from a Kinesis data stream and writes results to an Amazon S3 bucket. The team notices that the application is experiencing high latency during peak hours. The stream has 8 shards, and the application is configured with a parallelism of 4. Which action would most likely reduce the latency?

Accepted Answer

Increase the parallelism of the Flink application to 8.. The application has 8 shards but only a parallelism of 4, meaning each Flink subtask must process data from 2 shards. This creates a bottleneck because a single subtask cannot process data from multiple shards faster than the slowest shard's throughput. Increasing parallelism to 8 matches the shard count, allowing each subtask to read from exactly one shard, eliminating the contention and reducing latency.

Answer

Decrease the batch size in the S3 sink.

Answer

Use a larger Kinesis Data Analytics application instance type.

Answer

Increase the checkpointing interval to reduce overhead.

Question 6

A team is designing a data ingestion pipeline to load JSON files from an Amazon S3 bucket into Amazon Redshift. The files arrive every 5 minutes, and each file is between 10 MB and 50 MB. The team wants to minimize the time between file arrival and data availability in Redshift. Which approach should the team use?

Accepted Answer

Use S3 Event Notifications to trigger an AWS Lambda function that runs the COPY command to load data into Redshift.. S3 Event Notifications can trigger an AWS Lambda function that executes the COPY command, loading data into Redshift with minimal latency. This approach avoids the overhead of scheduling or batching, directly responding to each file arrival to meet the 5-minute frequency and file size requirements.

Answer

Schedule an AWS Glue job to run every 5 minutes to load the data.

Answer

Use Amazon Redshift Spectrum to query the data directly from S3 without loading.

Answer

Configure Amazon Kinesis Data Firehose to stream data from S3 to Redshift.

Question 7

A data engineer is ingesting data from an Amazon RDS for PostgreSQL database into Amazon S3 using AWS Glue. The Glue job reads the entire table each time it runs, which takes several hours. The team wants to reduce the job duration by reading only new or updated records. Which approach should the engineer adopt?

Accepted Answer

Enable job bookmarks in AWS Glue and use a column with timestamps as the bookmark key to read only incremental data.. AWS Glue job bookmarks track previously processed data using a specified column (e.g., a timestamp column) as the bookmark key. When enabled, the job reads only new or updated records since the last run, significantly reducing job duration by avoiding full table scans. This directly addresses the requirement to read incremental data from the PostgreSQL source.

Answer

Partition the table in the source database by date and read only the latest partition.

Answer

Increase the number of Glue workers to improve parallel reads.

Answer

Use Amazon Kinesis Data Streams to capture changes from PostgreSQL.

Question 8

A company uses Amazon Kinesis Data Firehose to deliver streaming data to Amazon S3. The data is in JSON format and each record is about 2 KB. The delivery stream is configured to buffer data for 60 seconds or 5 MB, whichever comes first. The team notices that the S3 objects are very small (around 1 MB) and numerous, causing high costs due to S3 PUT requests. Which configuration change should the team make to reduce the number of S3 objects?

Accepted Answer

Increase the buffer size to 50 MB and the buffer interval to 300 seconds.. Increasing the buffer size to 50 MB and the buffer interval to 300 seconds allows Kinesis Data Firehose to accumulate more data before writing to S3, resulting in fewer, larger objects. The current configuration triggers a write every 60 seconds or when 5 MB is buffered, but since each record is only 2 KB, the 5 MB threshold is rarely met, causing frequent small writes. By raising both thresholds, the delivery stream will buffer more records and write larger objects, reducing the number of S3 PUT requests and associated costs.

Answer

Enable compression (GZIP) on the delivery stream.

Answer

Reduce the buffer interval to 30 seconds and keep buffer size at 5 MB.

Answer

Switch from Kinesis Data Firehose to Amazon Kinesis Data Streams and use a Lambda function to write to S3.

Question 9

A data engineer needs to transfer 50 TB of historical data from an on-premises Hadoop cluster to Amazon S3. The company has a slow internet connection (100 Mbps). The data must be transferred within 2 weeks. Which service should the engineer recommend?

Accepted Answer

AWS Snowball Edge. B is correct because transferring 50 TB over a 100 Mbps connection would take approximately 48 days (50 TB * 1024 GB/TB * 8 bits/byte / (100 Mbps * 86400 seconds/day)), far exceeding the 2-week deadline. AWS Snowball Edge is a physical appliance that can be loaded with data on-premises and shipped to AWS, bypassing the slow network entirely and completing the transfer within days.

Answer

AWS DataSync

Answer

Amazon Kinesis Data Firehose

Answer

AWS Glue ETL job with JDBC connection to Hadoop

Question 10

A company is using Amazon Kinesis Data Streams to ingest clickstream data from a website. The data is consumed by an AWS Lambda function that enriches records and writes to Amazon S3. The Lambda function is experiencing high error rates due to records exceeding the 256 KB payload limit. Which TWO actions should the team take to resolve this issue?

Accepted Answer

Enable compression on the producer side before sending records to Kinesis.. Enabling compression on the producer side reduces the size of each record before it is sent to Kinesis Data Streams, directly addressing the 256 KB payload limit. Option C is correct because the Kinesis Producer Library (KPL) aggregates multiple small records into a single larger record, which is then stored as one Kinesis record, reducing the number of records that exceed the limit and improving throughput.

Answer

Increase the Lambda function timeout.

Answer

Switch from Kinesis Data Streams to Kinesis Data Firehose.

Answer

Increase the number of shards in the Kinesis stream.

Question 11

A data engineer is ingesting streaming data from an IoT fleet into Amazon S3 using Amazon Kinesis Data Firehose. The data arrives as JSON, but the downstream analytics require Parquet format. Which Firehose transformation should the engineer configure?

Accepted Answer

Configure a Lambda function as a data transformation in Firehose to convert JSON to Parquet.. Amazon Kinesis Data Firehose can invoke an AWS Lambda function as a data transformation step to convert incoming JSON records to Parquet format before delivery to S3. This is the native, serverless way to perform record-level format conversion within the Firehose delivery stream, ensuring downstream analytics tools can directly query the Parquet data without additional processing.

Answer

Use an S3 lifecycle policy to convert JSON to Parquet.

Answer

Use S3 Batch Operations to convert existing JSON objects to Parquet.

Answer

Use Kinesis Data Analytics to convert the stream to Parquet before writing to S3.

Question 12

A data engineer is designing a real-time analytics pipeline for clickstream data. The source is Amazon Kinesis Data Streams, and the data must be stored in Amazon S3 in partitioned Parquet format with near-real-time latency. The engineer must also handle late-arriving data (up to 1 hour). Which combination of services meets these requirements?

Accepted Answer

Use Kinesis Data Firehose with a Lambda transformation to write to S3, and a separate Lambda consumer to reprocess late data from the stream.. AWS Glue streaming ETL, while capable of handling late data via Spark Structured Streaming's watermarking, is not ideal for near-real-time Parquet partitioning to S3 at high throughput; it is more suited for complex transformations with micro-batches. Option B is incorrect: Kinesis Data Analytics tumbling windows process data in memory and output to destinations like S3, but they lack built-in late data handling and Parquet conversion. Option C is correct: Kinesis Data Firehose delivers streaming data to S3 with near-real-time latency, and a Lambda transformation converts it to Parquet. To handle late-arriving data (up to 1 hour), a separate Lambda consumer reads from the Kinesis Data Streams shard iterator to reprocess records that arrive after the Firehose delivery window, ensuring no data loss. Option D is incorrect: S3 Batch Operations are designed for batch processing of existing objects, not for near-real-time handling of late-arriving streaming data.

Answer

Use AWS Glue streaming ETL to write to S3 in real-time.

Answer

Use Kinesis Data Analytics with a tumbling window to write to S3.

Answer

Use Kinesis Data Firehose to deliver to S3, and use S3 Batch Operations to process late data.

Question 13

A data engineer is building a data ingestion pipeline that reads JSON files from Amazon S3 and loads them into an Amazon Redshift table using COPY commands. The files are gzip compressed and contain nested JSON. The engineer wants to minimize transformation steps. Which approach should the engineer use?

Accepted Answer

Use the COPY command with the 'auto' option to ingest JSON directly.. The COPY command with the 'auto' option can directly ingest gzip-compressed JSON files from S3 into Redshift, automatically inferring the schema and handling nested structures without requiring intermediate transformation steps. This minimizes transformation steps by leveraging Redshift's native JSON parsing capability, which supports both 'auto' and 'jsonpaths' options for nested data.

Answer

Use Amazon Athena to query the JSON and INSERT INTO Redshift.

Answer

Use Kinesis Data Firehose to transform and load into Redshift.

Answer

Use AWS Glue ETL to flatten the JSON and write to S3 as CSV, then COPY from CSV.

Question 14

A company is migrating its on-premises data warehouse to Amazon Redshift. The daily batch load from the source database takes 6 hours using a single-node Redshift cluster. The engineer needs to reduce load time to under 2 hours without increasing cost significantly. Which strategy should the engineer adopt?

Accepted Answer

Increase the number of nodes in the Redshift cluster and use parallel COPY from multiple files.. Increasing the number of nodes in the Redshift cluster provides more compute and I/O capacity, and using parallel COPY from multiple files allows Redshift to automatically split the load across the node slices, dramatically reducing load time. This approach scales performance linearly with the number of nodes, enabling the engineer to meet the sub-2-hour target without significantly increasing cost if the cluster is sized appropriately.

Answer

Use COPY with compression (gzip) to reduce data volume.

Answer

Use a VPC endpoint to improve network throughput to S3.

Answer

Change the table distribution style to EVEN to distribute data evenly.

Question 15

A data engineer needs to ingest data from an external HTTP API into Amazon S3. The API returns JSON data for a list of users, updated hourly. The engineer wants to use a serverless solution with minimal operational overhead. Which AWS service should the engineer use?

Accepted Answer

Amazon AppFlow with an HTTP connector on a scheduled flow.. Amazon AppFlow with an HTTP connector on a scheduled flow is the correct choice because it provides a fully managed, serverless integration that directly connects to external HTTP APIs, retrieves JSON data, and writes it to Amazon S3 on a scheduled basis (e.g., hourly) without requiring any custom code or infrastructure management. This minimizes operational overhead while meeting the ingestion requirements.

Answer

Amazon Kinesis Data Firehose with a custom HTTP endpoint.

Answer

AWS Lambda function triggered by CloudWatch Events.

Answer

AWS Glue ETL job triggered by EventBridge.