DEA-C01 Data Ingestion and Transformation Practice Test 27 — 15 Questions

Question 1

A data engineer is designing a pipeline that ingests streaming events from an Amazon Kinesis Data Stream. Downstream consumers must process each record exactly once, and the pipeline must tolerate consumer restarts without reprocessing events. The engineer configures an AWS Lambda function as the consumer. Which action should the engineer take to meet these requirements?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Use the Kinesis Client Library (KCL) with DynamoDB for lease and checkpoint management, and make the consumer idempotent.. Exactly-once processing in Kinesis requires durable checkpointing so a restarted consumer resumes from the last processed position rather than reprocessing events. The Kinesis Client Library persists shard leases and checkpoints in DynamoDB, and pairing that with idempotent consumer logic avoids duplicate side effects. Configuration flags on the Lambda event source mapping alone do not provide this guarantee.

Answer

Set the Lambda function's event source mapping to use a tumbling window and set the maximum retry attempts to zero.

Answer

Configure the Lambda function's event source mapping with StartingPosition set to LATEST and enable parallelization factor of 10.

Answer

Enable the event source mapping's checkpointing by setting BisectBatchOnFunctionError to true and configuring a dead-letter queue.

Question 2

A data engineer is using AWS Glue to transform data from an Amazon S3 bucket that contains nested JSON files. The engineer needs to flatten the nested structure and write the output to Amazon S3 in Parquet format, partitioned by a date column. The Glue job must handle schema evolution and minimize manual coding. Which approach should the engineer take?

Accepted Answer

Use AWS Glue DynamicFrame with relationalize() to flatten nested data, then write to Parquet with partition keys.. The requirement is to flatten nested JSON, handle schema evolution, and minimize coding. AWS Glue DynamicFrame's relationalize() method is designed to unnest nested structures automatically and integrates with the Glue Data Catalog for schema management. Writing to Parquet with partitioning is straightforward. Other options require more manual coding or do not handle schema evolution as seamlessly.

Answer

Use AWS Glue Studio visual editor to create a job that uses the Unnest transform, then write to Parquet with partitioning.

Answer

Use AWS Glue crawler to infer schema, then use AWS Glue ETL job with a custom Scala script to flatten and write to Parquet.

Answer

Use AWS Glue with Apache Spark DataFrame and explode() function to flatten arrays, then write to Parquet with partitioning.

Question 3

A data engineer is building a streaming ingestion pipeline using Amazon Kinesis Data Streams. The producer is a fleet of application servers that emit events using the Kinesis Producer Library (KPL). The engineer needs the consumer application to receive records with minimal latency and to handle the case where the same shard is processed by multiple worker threads. Which two configurations should the engineer apply? (Choose two.)

Accepted Answer

Configure the consumer to use enhanced fan-out with the SubscribeToShard API and a dedicated throughput of 2 MB per second per consumer.. Enhanced fan-out with SubscribeToShard gives the consumer a dedicated 2 MB per second per shard push-based read path, lowering latency compared to shared-throughput polling. The Kinesis Client Library handles shard leases and checkpoints across multiple worker threads so that no two workers process the same shard simultaneously, which is required for safe parallel consumption.

Answer

Set the KPL aggregation setting to reduce per-record overhead and increase throughput.

Answer

Disable server-side encryption on the stream to reduce per-record decryption latency for the consumer.

Answer

Increase the number of shards in the stream so each worker thread maps to a distinct shard.

Question 4

A data engineer must load a 500 GB compressed CSV file from Amazon S3 into Amazon Redshift using the COPY command. The cluster is a multi-node RA3 cluster, and the engineer wants the load to complete as quickly as possible while minimizing the impact on existing queries. Which approach is MOST appropriate?

Accepted Answer

Split the CSV file into multiple smaller files and use the COPY command with `COMPUPDATE OFF` and `STATUPDATE OFF` to load in parallel.. The fastest and least disruptive approach is to split the large CSV into multiple files so Redshift can parallelize the read across slices, and to disable automatic compression and statistics updates during the load. These options reduce load time and resource contention. Single-file loads and Glue-based writes do not achieve the same throughput or efficiency for a bulk load.

Answer

Use the COPY command with `GZIP` and load the single compressed file directly without splitting.

Answer

Use AWS Glue to read the CSV from S3 and write to Redshift using the Redshift connector with the `overwrite` write mode.

Answer

Use the COPY command with the `COMPUPDATE OFF` and `STATUPDATE OFF` options and load from a single large file.

Question 5

A data engineer runs an AWS Glue crawler against an Amazon S3 prefix that contains JSON files with evolving schemas. New fields appear frequently, and the crawler keeps updating the Data Catalog table schema, which breaks downstream Athena queries. The engineer wants the crawler to detect new columns without changing the existing column order or removing previously cataloged columns. Which crawler configuration should the engineer use?

Accepted Answer

Set the crawler configuration to 'MergeNewColumns' so new columns are appended and existing columns are retained.. The MergeNewColumns crawler configuration is designed for additive schema evolution: it appends newly detected columns to the existing Data Catalog table while retaining previously cataloged columns and their order. This keeps downstream Athena queries stable while still surfacing new JSON fields, which is exactly what the engineer needs.

Answer

Set the crawler's schema change policy to 'Log' and configure an Amazon CloudWatch Events rule to trigger a Lambda function that edits the table.

Answer

Enable the crawler's 'Create a single schema for each S3 path' option and set the table's SerDe to ignore unknown fields.

Answer

Set the crawler's schema change policy to 'Update the table definition in the Data Catalog' and set the configuration option to 'Ignore the change and don't update the table in the Data Catalog'.

Question 6

A company ingests data into Amazon S3 from hundreds of sources. A data engineer must ensure that any records failing validation during AWS Glue ETL are captured for later analysis without failing the entire job. The engineer wants to separate good records from bad ones and write both to Amazon S3. Which approach should the engineer use?

Accepted Answer

Use the EvaluateDataQuality transform with a rule set and configure the job to write records that fail rules to a separate S3 location.. The EvaluateDataQuality transform in AWS Glue applies a rule set to a DynamicFrame and can emit a separate frame containing records that fail those rules. Writing that frame to a distinct Amazon S3 location isolates invalid records without stopping the job. This gives the engineer both cleaned output and a quarantine area for later investigation, matching the requirement.

Answer

Configure the Glue job's error handling to retry failed tasks and enable job metrics, then write all records to a single S3 path.

Answer

Apply a Filter transformation to keep valid records, and write the discarded records to a separate S3 path using a second write.

Answer

Use a DynamicFrame with the ResolveChoice transformation to split records into separate columns, then write both outputs to S3.

Question 7

A data engineer is building an AWS Glue job that reads from an Amazon DynamoDB table and writes to Amazon S3 in Parquet format. The table has millions of items and the engineer wants the read to use the table's provisioned capacity efficiently while avoiding a full table scan that would consume too much read capacity. The job only needs items where the status attribute equals 'ACTIVE'. Which approach should the engineer use?

Accepted Answer

Use the Glue DynamoDB connector with a query that targets an index or key condition on status, or export the table to S3 and read the filtered subset.. To avoid a full scan, the read must use a query against a key condition or a secondary index, or the data must be exported and filtered outside the table. The Glue DynamoDB connector supports filter expressions, but those apply after the read and do not save capacity. Querying an index or exporting to S3 both reduce or eliminate table read consumption.

Answer

Increase the read capacity units on the DynamoDB table to the maximum before running the Glue job and reduce them afterward.

Answer

Query the table through the DynamoDB Accelerator (DAX) cluster from within the Glue job to reduce read capacity consumption.

Answer

Use the Glue DynamoDB connector with a filter expression on the status attribute and enable the job bookmark.

Question 8

A data engineer has an AWS Glue ETL job that processes data from an S3 bucket and writes to another S3 bucket. The job runs hourly and must avoid reprocessing the same source files. Some source files are added late, after the hour in which they were expected. The engineer enables job bookmarks but notices that late-arriving files are sometimes skipped. Which action should the engineer take to ensure late files are processed while still avoiding duplicate processing?

Accepted Answer

Disable job bookmarks and instead maintain a separate DynamoDB table that records processed file keys, checking the table before processing each file.. Glue job bookmarks track processed data based on job runs and can miss files that arrive after the run in which they were expected. A custom state store such as DynamoDB lets the job check each file key before processing, so late files are handled while duplicates are still prevented. Disabling bookmarks and managing state explicitly gives the needed control.

Answer

Increase the bookmark's time window by setting the job's --job-bookmark-option to job-bookmark-enable and adding a longer --TempDir prefix.

Answer

Change the job trigger to run every 15 minutes instead of hourly so late files are picked up in a subsequent run.

Answer

Set the job's --job-bookmark-option to job-bookmark-reset before each run so the bookmark starts fresh every hour.

Question 9

A data engineer must ingest data from an Amazon DynamoDB table into an Amazon S3 data lake. The table has a high write rate and the ingestion must capture item-level changes without impacting the table's throughput. The engineer needs a solution that automatically streams changes to S3 with minimal operational overhead. Which solution meets these requirements?

Accepted Answer

Enable DynamoDB Streams and use Amazon Kinesis Data Firehose to deliver the stream to S3.. Amazon Kinesis Data Firehose natively integrates with DynamoDB Streams, allowing you to deliver change records to Amazon S3 with minimal configuration and operational overhead. It automatically scales, buffers, and retries, ensuring reliable delivery. This approach captures item-level changes in near real-time without affecting the table's provisioned throughput, making it the ideal solution for the given requirements.

Answer

Use AWS Database Migration Service (AWS DMS) with change data capture (CDC) to replicate changes to S3.

Answer

Use AWS Glue to periodically scan the entire DynamoDB table and write changed items to S3.

Answer

Enable DynamoDB Streams on the table and use AWS Lambda to read the stream and write to S3.

Question 10

A data engineer needs to ingest streaming data from an Amazon Kinesis Data Stream into an Amazon S3 bucket. The data is in JSON format and must be stored in Parquet for efficient querying. The engineer wants a fully managed solution that requires minimal coding. Which service should be used?

Accepted Answer

Amazon Kinesis Data Firehose with AWS Lambda for transformation. Amazon Kinesis Data Firehose is a fully managed service that can ingest streaming data from Kinesis Data Streams and deliver it to Amazon S3. It supports Lambda-based transformation, allowing you to convert JSON to Parquet with minimal code. Firehose handles buffering, scaling, and retries, making it the ideal choice for a low-code, managed solution.

Answer

AWS Glue streaming ETL job

Answer

AWS Database Migration Service (AWS DMS) with Kinesis as source

Answer

Amazon Managed Service for Apache Flink

Question 11

A data engineer needs to ingest streaming data from an IoT fleet into Amazon S3 for near-real-time analytics. The data volume is approximately 5 GB per hour, and each event is less than 1 KB. Which AWS service should be used as the ingestion endpoint?

Accepted Answer

AWS IoT Core. AWS IoT Core is purpose-built for ingesting data from IoT devices, supporting MQTT, HTTP, and WebSocket protocols. It can handle millions of devices and high-throughput, small-message payloads (each event <1 KB) and integrates directly with Amazon S3 via IoT Core rules, making it the ideal ingestion endpoint for near-real-time analytics on streaming IoT data.

Answer

AWS DataSync

Answer

Amazon AppFlow

Answer

Amazon Kinesis Data Streams

Question 12

A company uses AWS Glue ETL jobs to transform data from Amazon S3 to Amazon Redshift. The job reads JSON files, applies schema mapping, and writes to a Redshift table. Recently, the job started failing with memory errors. The data volume has increased tenfold. Which approach should a data engineer take to resolve this issue with minimal code changes?

Accepted Answer

Increase the number of DPUs allocated to the Glue job.. Increasing the number of DPUs (Data Processing Units) allocated to the AWS Glue job directly addresses the memory constraint caused by a tenfold increase in data volume. Glue ETL jobs run on Apache Spark, which distributes data processing across executors; more DPUs provide more memory and compute capacity, allowing the job to handle larger datasets without code changes.

Answer

Switch from Spark to Python Shell job type.

Answer

Implement batch processing with smaller file sizes.

Answer

Use Redshift Spectrum to query data directly from S3.

Question 13

A financial services company processes real-time stock trade data. They use Amazon Kinesis Data Streams with a shard count of 5, each shard receiving about 500 records per second. The consumer application uses the Kinesis Client Library (KCL) with DynamoDB for checkpointing. Lately, some records are being processed multiple times. What is the most likely cause?

Accepted Answer

The consumer application is crashing and restarting, causing re-processing of records.. The Kinesis Client Library (KCL) uses DynamoDB to track checkpoint progress for each shard. If the consumer application crashes and restarts, the KCL will resume processing from the last committed checkpoint, which may be behind the actual processing point. This causes records that were already processed (but not yet checkpointed) to be re-processed, leading to duplicate processing.

Answer

The Kinesis stream's iterator age is exceeding the retention period.

Answer

The DynamoDB table used for checkpointing is throttling write requests.

Answer

The record size exceeds the 1 MB API limit, causing retries.

Question 14

A data engineering team needs to transform CSV files stored in Amazon S3 into Parquet format using AWS Glue. The files are partitioned by date and are updated hourly. Which AWS Glue feature should be used to automatically detect the schema and partition structure?

Accepted Answer

AWS Glue Crawler. AWS Glue Crawler is the correct choice because it automatically scans data in S3, infers the schema (including data types), and detects the partition structure (e.g., date-based partitions like year/month/day) by examining the folder hierarchy. It then populates the AWS Glue Data Catalog with metadata, enabling ETL jobs to read the data without manual schema definition.

Answer

AWS Glue DataBrew

Answer

AWS Lake Formation

Answer

Amazon Athena

Question 15

An e-commerce company ingests clickstream data from their website into Amazon S3. The data is in JSON format, and each file is about 10 MB. They need to transform the data into a columnar format for analytics and load it into Amazon Redshift nightly. The transformation should be cost-effective and require minimal operational overhead. Which approach meets these requirements?

Accepted Answer

Use AWS Glue ETL job to convert to Parquet and load into Redshift.. AWS Glue ETL is the correct choice because it is a serverless, managed service that can efficiently convert JSON to Parquet (a columnar format optimized for Redshift) and load the data into Redshift with minimal operational overhead. The nightly batch processing of 10 MB files is well-suited for Glue's pay-per-use pricing, making it cost-effective without requiring infrastructure management.

Answer

Use Amazon Redshift COPY command to load JSON directly.

Answer

Use Amazon EMR with Spark to transform and load data.

Answer

Use AWS Lambda to transform each file and write to Redshift.