DEA-C01 Data Ingestion and Transformation Practice Test 23 — 15 Questions

Question 1

A data engineer needs to ingest data from an Amazon DynamoDB table into an Amazon S3 bucket for analytics. The table has a steady write rate of 500 write capacity units and contains 20 GB of data. The ingestion must capture incremental changes and run continuously with minimal operational overhead. Which AWS service should the engineer use?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

AWS Database Migration Service (AWS DMS) with change data capture (CDC). AWS DMS with change data capture is the appropriate service for continuously replicating incremental changes from DynamoDB to Amazon S3. It reads from DynamoDB Streams and writes change records to S3, handling checkpointing and scaling automatically. Other options either lack native CDC support for DynamoDB or are not designed for database replication, making them unsuitable for this continuous ingestion requirement.

Answer

Amazon Kinesis Data Firehose with a DynamoDB stream as the source

Answer

AWS DataSync with a DynamoDB location as the source

Answer

AWS Glue ETL job with a DynamoDB connection and bookmarks

Question 2

A data engineer is configuring an AWS Glue ETL job that reads from an Amazon S3 bucket containing nested JSON files. The engineer needs to flatten the nested structures and write the output to Amazon S3 in Parquet format. The job must run daily and handle schema changes automatically. Which approach should the engineer take to meet these requirements?

Accepted Answer

Use the AWS Glue DynamicFrame with relationalize() to flatten nested data, and enable job bookmarks to track processed data.. The relationalize() transform in AWS Glue is specifically designed to flatten nested data by creating separate relational tables for nested structures, which can then be written as Parquet. Combined with job bookmarks, it enables incremental processing for daily runs and handles schema evolution by dynamically creating new tables when new nested fields appear. Other options either lack the flattening capability or do not automatically handle schema changes.

Answer

Use AWS Glue crawlers to catalog the nested JSON, then use Amazon Athena to flatten the data and write Parquet to S3.

Answer

Use AWS Glue with a custom Spark script that uses explode() on nested arrays and writes Parquet, with job bookmarks enabled.

Answer

Use AWS Glue Studio to create a visual ETL job with a Flatten transform, and configure the job to run on a schedule with schema inference enabled.

Question 3

A data engineer is building an AWS Glue ETL job that reads JSON files from Amazon S3, flattens nested arrays, and writes Parquet to another S3 prefix. The job must minimize the number of small files written and must avoid reprocessing files that were already transformed in previous runs. Which combination of AWS Glue features should the engineer use?

Accepted Answer

Enable job bookmarks and call repartition or coalesce on the DynamicFrame before writing, then write with a format that supports file grouping such as Parquet with a specified compression.. To avoid reprocessing, AWS Glue job bookmarks must be enabled so the job tracks previously processed S3 objects. To minimize small files, the DynamicFrame should be repartitioned or coalesced before the write, which controls the number of output partitions and therefore the number of Parquet files. Writing compressed Parquet completes the efficient columnar output. Options that rely on nonexistent parameters or on S3 versioning do not satisfy either requirement.

Answer

Disable job bookmarks and rely on S3 object versioning to skip already transformed files, then use the Glue 'groupFiles' option to merge output.

Answer

Enable job bookmarks and set the Glue job parameter '--enable-metrics' with '--output-file-count' to limit files, then write CSV instead of Parquet.

Answer

Enable job bookmarks and set the write option to coalesce partitions by using repartition or coalesce before writing, then configure the S3 target with a larger group size.

Question 4

A data engineer is configuring an AWS Glue crawler to catalog data stored in Amazon S3. The data is partitioned by year/month/day, and the crawler must recognize these partitions and add them to the AWS Glue Data Catalog. The engineer also wants to ensure that the crawler does not reprocess all partitions on every run. Which configuration should the engineer use?

Accepted Answer

Enable the crawler's 'Incremental crawl' option and specify a schedule that runs frequently.. AWS Glue crawlers support an incremental crawl mode that only processes new partitions or folders added since the last crawl. This is the correct way to avoid reprocessing all partitions on every run. The other options either do not exist or enable full recrawling, which is inefficient for partitioned datasets.

Answer

Use a custom classifier to parse the partition structure and set the crawler to 'Ignore the change' for existing partitions.

Answer

Set the crawler's 'Update all new and existing partitions with metadata from the table' option and enable 'Recrawl every time'.

Answer

Configure the crawler to use 'Crawl new folders only' and set the 'Maximum number of partitions' to a high value.

Question 5

A data engineer is building a real-time streaming pipeline using Amazon Kinesis Data Streams. The data is ingested by multiple producers and must be processed by an AWS Lambda function. The engineer needs to ensure that records are processed in order and that each record is processed exactly once. Which configuration should the engineer use?

Accepted Answer

Use a Kinesis Data Streams stream with a single shard, and configure the Lambda function with a batch size of 1 and a starting position of LATEST, and implement idempotent processing in the Lambda function.. To achieve ordered and exactly-once processing with Kinesis and Lambda, use a single shard to guarantee order, a batch size of 1 to process sequentially, and implement idempotent processing in the Lambda function to handle duplicates from at-least-once delivery. Other options either do not ensure ordering or do not address exactly-once.

Answer

Use a Kinesis Data Streams stream with multiple shards, and configure the Lambda function with a batch size of 1 and a starting position of TRIM_HORIZON, and enable function-level error handling with a dead-letter queue.

Answer

Use a Kinesis Data Streams stream with multiple shards, and configure the Lambda function with a batch size of 1 and a starting position of LATEST.

Answer

Use a Kinesis Data Streams stream with a single shard, and configure the Lambda function with a batch size greater than 1 and a starting position of TRIM_HORIZON.

Question 6

A data engineer needs to run a daily AWS Glue extract, transform, and load (ETL) job that reads JSON files from Amazon S3, applies transformations, and writes the results to Amazon Redshift. The job must be triggered automatically each day at 02:00 UTC. Which approach should the engineer use to schedule the job with the LEAST effort?

Accepted Answer

Create an Amazon EventBridge scheduled rule that starts an AWS Glue workflow on a cron expression. An Amazon EventBridge scheduled rule can start an AWS Glue workflow on a cron schedule with minimal configuration. Glue workflows support triggers and dependencies, so the daily ETL can be orchestrated without custom code. This is the least-effort managed option and integrates natively with Glue, unlike Lambda-based scheduling or Redshift stored procedures.

Answer

Write an AWS Lambda function with a CloudWatch Events timer that calls the Glue StartJobRun API

Answer

Use an AWS Glue trigger of type SCHEDULED attached directly to the job with a cron expression

Answer

Configure an Amazon Redshift scheduled query that invokes the Glue job through a stored procedure

Question 7

A data engineer is building an AWS Glue ETL job that reads from an Amazon RDS for PostgreSQL database and writes to Amazon S3 in Parquet format. The job must handle large tables efficiently without causing excessive load on the source database. The engineer needs to enable parallel reads across multiple partitions. Which AWS Glue connection property should be configured to achieve this?

Accepted Answer

In the Glue job script, specify 'hashfield' and 'hashexpression' in the connection options to partition the data.. To enable parallel reads from a JDBC source in AWS Glue, the job script must specify 'hashfield' and 'hashexpression' in the connection options. These parameters allow Glue to partition the data based on a numeric column, creating multiple concurrent connections. This approach improves performance and minimizes load on the source database by distributing the read operations.

Answer

Configure the Glue connection with 'jdbcParallelism' set to a high value and enable 'parallelRead'.

Answer

Set the connection's 'UseConnectionProperties' to true and specify 'hashfield' and 'hashexpression' in the job script.

Answer

Enable 'parallelRead' in the Glue job's connection parameters and set 'numPartitions' to the desired level.

Question 8

A data engineer is using AWS Glue Studio to build a job that reads from an Amazon Kinesis Data Stream and writes to Amazon S3 in Parquet format. The job must handle late-arriving data and produce exactly-once semantics for downstream consumers. The engineer configures the job with a 1-minute window and enables job bookmarks. However, after several runs, duplicate records appear in S3. Which configuration change should the engineer make to resolve the duplicates?

Accepted Answer

Configure the Glue job to use the 'Kinesis' source with job bookmarks enabled and set the 'startingPosition' to 'TRIM_HORIZON' or 'LATEST' as appropriate, and ensure the job's output uses a deterministic file naming or idempotent write pattern.. Duplicates in S3 from a Glue streaming job typically stem from two sources: reprocessing Kinesis records because bookmarks are not correctly tracking sequence numbers, and non-idempotent writes to S3 when the job retries. Enabling bookmarks for the Kinesis source ensures the job resumes from the last processed record, while a deterministic output pattern (such as writing to a partition that can be overwritten) ensures that even if a retry occurs, the same data is not appended twice. Together, these provide exactly-once semantics for downstream consumers.

Answer

Disable job bookmarks and instead use a custom checkpointing mechanism in the Glue script.

Answer

Enable the 'Kinesis source' bookmark option and ensure the job uses the 'from the latest' starting position with checkpointing enabled.

Answer

Increase the window size to 5 minutes to allow late data to arrive before processing.

Question 9

A data engineer is designing an ingestion pipeline for semi-structured event data that arrives as JSON files in an Amazon S3 bucket. The downstream AWS Glue job must handle schema drift, where new fields appear over time and some existing fields change type. The engineer needs the Glue job to process the data without failing on these changes. Which two approaches should the engineer use? (Choose two.)

Accepted Answer

Use the 'ResolveChoice' transform with the 'cast' action to coerce conflicting types into a single type such as string, preventing job failure on type changes.. Schema drift in JSON requires both handling type conflicts within the job and keeping the catalog current. ResolveChoice with the cast action coerces conflicting types to a single consistent type so the job does not fail, while writing with the catalog update option adds newly discovered fields to the Data Catalog table. Together these allow the pipeline to process evolving JSON and expose the updated schema to downstream query engines.

Answer

Use a Glue DynamicFrame with the 'ResolveChoice' transform set to 'make_cols' so conflicting types are preserved as separate columns.

Answer

Configure the Glue job to read with 'ignoreErrors' so any record with an unexpected type is silently dropped from the output.

Answer

Set the Glue job parameter '--enable-auto-scaling' to allow the job to add workers when new fields are encountered.

Question 10

A data engineer is using AWS Glue Studio to build an ETL job that reads from an Amazon S3 source and writes to Amazon Redshift. The engineer needs to ensure the job handles data errors gracefully and logs detailed information for troubleshooting. Which two actions should the engineer take? (Choose two.)

Accepted Answer

Configure the job to write error records to a separate Amazon S3 path using the 'Error' output option in the Glue Studio visual editor.. To handle data errors gracefully, the engineer should use the error output path in Glue Studio to redirect failed records, and use the Evaluate Data Quality transform to apply rules and route bad records. These two actions ensure that errors are captured and isolated, allowing the job to continue processing valid data.

Answer

Increase the number of workers to ensure the job can handle errors without failing.

Answer

Enable job metrics in the AWS Glue job configuration to capture detailed execution statistics.

Answer

Set the job's log retention to a longer period in Amazon CloudWatch Logs.

Question 11

A data engineer is building a data pipeline that ingests data from an Amazon Kinesis Data Stream into an Amazon S3 bucket. The data is in JSON format and must be transformed into Parquet before being stored in S3. The pipeline must handle data continuously and minimize latency. Which solution should the engineer use?

Accepted Answer

Use Amazon Kinesis Data Firehose with the record format conversion feature to convert JSON to Parquet before delivering to S3.. Kinesis Data Firehose provides a native record format conversion feature that automatically converts JSON to Parquet using a schema from the AWS Glue Data Catalog. This is a fully managed, low-latency solution that requires no custom code and efficiently delivers transformed data to S3. Other options involve custom code, higher latency, or unnecessary complexity for a simple format conversion.

Answer

Use Amazon Kinesis Data Firehose to read from the Kinesis Data Stream, apply an AWS Lambda function to transform JSON to Parquet, and deliver to S3.

Answer

Use AWS Glue streaming ETL to read from the Kinesis Data Stream, transform to Parquet, and write to S3.

Answer

Use Amazon Kinesis Data Analytics to read from the Kinesis Data Stream, transform to Parquet, and write to S3.

Question 12

A data engineer is designing a data ingestion pipeline that uses Amazon Kinesis Data Streams to capture real-time events from thousands of IoT devices. The events are JSON-formatted and must be stored in Amazon S3 in Parquet format for long-term analytics. The engineer needs to ensure the pipeline can handle sudden spikes in traffic and automatically scale. Which two actions should the engineer take to meet these requirements? (Choose two.)

Accepted Answer

Configure the Kinesis data stream to use on-demand capacity mode to automatically scale with traffic.. Using Kinesis Data Firehose to consume from the stream, convert JSON to Parquet via AWS Glue, and deliver to S3 provides automatic scaling and managed format conversion. Configuring the stream for on-demand capacity ensures it can handle traffic spikes without manual shard management. Together, these actions meet the requirements for scalability and Parquet storage with minimal operational overhead.

Answer

Attach an IAM role to the Kinesis data stream that allows it to write directly to Amazon S3.

Answer

Use an AWS Lambda function to read from the Kinesis data stream, convert JSON to Parquet, and write to Amazon S3.

Answer

Enable enhanced fan-out on the Kinesis data stream to provide dedicated throughput for each consumer.

Question 13

A data engineer is configuring an AWS Glue crawler to catalog a large Amazon S3 data lake. The crawler must handle data where the schema evolves over time, and the engineer wants the catalog to reflect new columns without breaking existing queries. The engineer also wants to minimize the number of partitions created. Which two crawler configuration choices BEST meet these requirements? (Choose two.)

Accepted Answer

Configure the crawler to update the table definition in the Data Catalog and add new columns to the existing schema.. Updating the table definition to add new columns preserves existing queries while exposing newly introduced fields, which is exactly how Glue supports schema evolution. A partition index keeps partition metadata lookups efficient as the table grows, reducing the overhead of managing many partitions. Together these choices let the catalog evolve safely and scale without fragmenting tables or freezing the schema.

Answer

Set the crawler to create a new table for each schema change it detects in the S3 prefix.

Answer

Enable the crawler option to ignore changes and only catalog the schema captured during the first run.

Answer

Set the crawler to sample only the first 100 objects in each S3 prefix to reduce runtime.

Question 14

A data engineer needs to ingest data from an Amazon DynamoDB table into an Amazon S3 bucket for analytics. The table has a steady write rate of 500 write capacity units and contains 20 GB of data. The ingestion must capture incremental changes and run continuously with minimal operational overhead. Which AWS service should the engineer use?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Enable DynamoDB Streams on the table and use Amazon Kinesis Data Firehose to deliver the stream records to Amazon S3.. DynamoDB Streams provides a time-ordered sequence of item-level changes with minimal performance impact. Kinesis Data Firehose can consume from DynamoDB Streams and deliver to S3 without custom code or server management, automatically batching and compressing data. This combination is fully managed, scales with the write rate, and requires the least operational effort compared to custom Lambda code, DMS, or periodic scans.

Answer

Enable DynamoDB Streams on the table and use AWS Lambda to write each change record to Amazon S3.

Answer

Use AWS Database Migration Service (AWS DMS) with ongoing replication to replicate the table to Amazon S3.

Answer

Schedule an AWS Glue job to periodically scan the entire DynamoDB table and write the data to Amazon S3.

Question 15

A data engineer needs to copy 500 GB of existing data from an on-premises NFS file server into Amazon S3 once. The engineer has a 100 Mbps internet connection and wants to complete the transfer within a few days without provisioning long-lived infrastructure. Which AWS service should the engineer use?

Accepted Answer

AWS DataSync with an agent deployed on-premises, configured with an S3 bucket as the destination.. For a one-time copy of hundreds of gigabytes over a usable network link, AWS DataSync with an on-premises agent is the right fit. It is a managed migration service that reads from NFS and writes to S3 with acceleration and integrity verification. Snowball is overkill for this volume, Transfer Family targets ongoing file exchange, and File Gateway provides ongoing access rather than a bulk copy.

Answer

AWS Snowball Edge Storage Optimized device shipped to the data center for offline transfer.

Answer

AWS Transfer Family with an SFTP endpoint backed by Amazon S3.

Answer

AWS Storage Gateway in File Gateway mode mounted as an NFS share on the source server.