DEA-C01 Data Ingestion and Transformation Practice Test 25 — 15 Questions

Question 1

A data engineer is using AWS Glue to extract data from an Amazon RDS for PostgreSQL database. The database contains a table with millions of rows, and the engineer needs to perform an incremental extract of only the rows that have changed since the last run. The table has a column named last_updated that stores a timestamp for each row. Which approach should the engineer use to efficiently extract only the changed rows?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Create an AWS Glue job that reads the entire table and filters rows using a WHERE clause with the last_updated timestamp.. Filtering with a WHERE clause on the last_updated column allows the Glue job to read only the rows that have changed since the last run. By parameterizing the job with the previous extraction timestamp, the engineer achieves an efficient incremental extract without reading the entire table. This approach is simple, integrates directly with Glue's JDBC capabilities, and leverages the existing timestamp column for change tracking.

Answer

Enable change data capture (CDC) on the RDS instance and use AWS Glue to read from the CDC stream.

Answer

Use AWS Database Migration Service (AWS DMS) with ongoing replication to capture changes and write to Amazon S3.

Answer

Use AWS Glue job bookmarks with a JDBC connection to automatically track changes.

Question 2

A data engineer is building an AWS Glue ETL job that reads semi-structured JSON event logs from Amazon S3. Many records contain nested arrays, and the engineer needs to flatten these arrays into separate rows so each array element becomes its own row in the output. The job uses a DynamicFrame. Which transformation should be used to achieve this?

Accepted Answer

Use the ExplodeFields transform on the DynamicFrame specifying the array column.. ExplodeFields is the AWS Glue DynamicFrame transform that expands array-type columns so that each element becomes a separate row. Relationalize produces multiple related tables, Unnest promotes struct fields to columns, and ApplyMapping only renames or casts fields. For turning nested arrays into rows, ExplodeFields is the correct choice.

Answer

Use the Relationalize transform on the DynamicFrame before writing to the target.

Answer

Use the Unnest transform on the DynamicFrame with the array column specified.

Answer

Use the ApplyMapping transform to map nested fields to top-level columns.

Question 3

A data engineer is building an AWS Glue ETL job that reads JSON files from Amazon S3 and writes to Amazon Redshift. The job occasionally fails with 'Communication link failure' errors during the write phase. The Redshift cluster is healthy and the network connectivity is stable. The engineer wants to make the job more resilient to transient connection issues without modifying the Redshift cluster. Which solution should the engineer implement?

Accepted Answer

Set the Glue job's '--write_retry_count' parameter to a value greater than 0 and configure the job to use the 'redshift' write option with 'retry' enabled.. The '--write_retry_count' job parameter in AWS Glue allows the job to automatically retry write operations that fail due to transient issues such as communication link failures. This is a built-in feature that can be set without modifying the target Redshift cluster. It is the correct approach for making the job resilient to temporary network glitches during writes.

Answer

Use the Glue DynamicFrame write with the 'redshift' connection type and set the 'aws_iam_role' parameter, then enable the 'useConnectionProperties' option.

Answer

Configure the Glue connection to use a JDBC URL with the 'retry' parameter set to 5 and increase the connection timeout.

Answer

Enable the Glue job's automatic retry with a maximum retry count of 3 and configure a job bookmark.

Question 4

A data engineer must ingest data from an Amazon DynamoDB table into an Amazon S3 bucket continuously with minimal operational overhead. The table uses on-demand capacity mode and has a stream enabled. The S3 bucket must receive only items that were newly added or modified, with no duplicate processing of unchanged items. Which solution BEST meets these requirements?

Accepted Answer

Enable DynamoDB Streams with NEW_AND_OLD_IMAGES, then use Amazon Kinesis Data Streams as the stream destination and a Lambda consumer to write changed items to Amazon S3.. DynamoDB Streams provide an ordered, near-real-time log of item-level changes, and delivering that stream through Kinesis Data Streams to a Lambda consumer allows only newly added or modified items to be written to Amazon S3. This avoids full-table rescans, prevents duplicate processing of unchanged items, and keeps operational overhead low because the components are managed and serverless.

Answer

Configure an AWS Glue ETL job with a DynamoDB connection that runs on a schedule every five minutes and reads the entire table each time.

Answer

Use AWS Database Migration Service with change data capture on the DynamoDB table to replicate changes into an S3 bucket continuously.

Answer

Export the DynamoDB table to Amazon S3 using the native export to S3 feature on a recurring schedule and process the exported files.

Question 5

A data engineer is using AWS Glue to extract data from an Amazon RDS for PostgreSQL database. The database contains a table with millions of rows, and the engineer needs to perform an incremental extract of only the rows that have changed since the last run. The table has a column named last_updated that stores a timestamp for each row. Which approach should the engineer use to efficiently extract only the changed rows?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Set the Glue job's 'dynamodb.throughput.read.percent' parameter to 0.5 and enable auto-scaling for the DynamoDB table.. To minimize impact on DynamoDB, the Glue job should limit its read throughput. The 'dynamodb.throughput.read.percent' parameter controls the fraction of read capacity the job consumes; setting it to 0.5 limits it to 50%. Enabling DynamoDB auto-scaling ensures the table can handle other workloads. Increasing workers or read percent would increase impact, and DAX or Streams are not suitable for full table extraction in Glue.

Answer

Increase the Glue job's number of workers to 20 and set the read percent to 1.0.

Answer

Enable DynamoDB Streams and have the Glue job read from the stream instead of the table.

Answer

Use DynamoDB Accelerator (DAX) to cache reads and set the Glue job to read from DAX.

Question 6

A data engineer is using AWS Glue Studio to build a visual ETL job that reads from an Amazon S3 bucket and writes to another S3 bucket. The job must process only files that were added in the last 24 hours. The engineer has already configured the S3 source node with the correct bucket and path. Which feature should the engineer use to filter files based on their creation timestamp?

Accepted Answer

Enable job bookmarking on the S3 source node.. Job bookmarks in AWS Glue maintain state across job runs, enabling incremental processing. For an S3 source, bookmarks compare the last job run time with the modification timestamps of objects, so only newly added or updated files are processed. This exactly matches the need to process files from the last 24 hours without writing custom code.

Answer

Add a Filter transform after the S3 source to compare the file path against a date string.

Answer

Configure the S3 source node to use a custom classifier that recognizes recent files.

Answer

Set the job's timeout property to 24 hours so it only processes recent data.

Question 7

A data engineer is using AWS Glue to process data from an Amazon S3 bucket and write to Amazon Redshift. The job must handle data that includes nested JSON structures and also needs to optimize the write performance to Redshift. Which two actions should the engineer take to achieve these goals? (Choose two.)

Accepted Answer

Use the Relationalize transform to flatten nested JSON structures.. To handle nested JSON, the Relationalize transform flattens the structure into relational tables suitable for Redshift. To optimize write performance, configuring the Redshift write to use parallel COPY from S3 improves throughput by leveraging Redshift's parallel loading. Together, these actions address both requirements.

Answer

Use the ApplyMapping transform to flatten nested JSON structures.

Answer

Enable job bookmarks to track processed data and avoid reprocessing.

Answer

Use the 'redshift' connection with the 'preactions' and 'postactions' options to manage table staging.

Question 8

A data engineer is designing an AWS Glue ETL job that reads from Amazon S3 and writes to Amazon Redshift. The job must efficiently handle large volumes of data and minimize the load time. The engineer wants to use AWS Glue's native capabilities to optimize the write to Redshift. (Choose two.)

Accepted Answer

Specify a temporary staging directory in Amazon S3 for the Redshift load operation.. To write from AWS Glue to Amazon Redshift efficiently, use the write_dynamic_frame.from_jdbc_conf method and provide a temporary S3 staging directory. This method writes data to S3 and then uses Redshift's COPY command for fast parallel loading. The from_options and from_catalog methods are not used for Redshift writes, and there is no explicit 'Use COPY command' parameter.

Answer

Enable the 'Use COPY command' option in the job's connection parameters to load data directly from S3.

Answer

Use the write_dynamic_frame.from_options method with connection_type 'redshift' and provide the necessary connection options.

Answer

Use the write_dynamic_frame.from_catalog method with the Redshift table name.

Question 9

A data engineer is using AWS Glue Studio to build a visual ETL job that reads from Amazon S3, applies a filter, and writes to Amazon Redshift. The job must run on a schedule and send an Amazon SNS notification if it fails. Which combination of steps should the engineer take to meet these requirements with the least operational overhead?

Accepted Answer

Create the Glue job in Glue Studio, add a trigger with a schedule, and configure the job's retry and SNS notification settings in the job details.. Glue Studio supports visual job authoring and, in the job details, allows configuring a schedule trigger and SNS notifications on failure. This native capability satisfies both scheduling and alerting with the least operational overhead, avoiding additional orchestration or monitoring services.

Answer

Use AWS Step Functions to orchestrate the Glue job, adding a Schedule state and an SNS task for failure handling.

Answer

Configure an Amazon EventBridge rule to trigger the Glue job on a schedule and another rule to invoke an SNS topic on Glue job state change.

Answer

Use AWS Glue workflows with a schedule trigger and add an Amazon CloudWatch alarm on the job's failure metric that publishes to SNS.

Question 10

A data engineer is using AWS Glue Studio to build a job that reads JSON files from Amazon S3, applies a transformation, and writes Parquet to another S3 location. The job must handle schema changes in the source JSON gracefully, and the engineer wants the job to automatically incorporate new columns without manual edits. Which Glue feature should the engineer enable?

Accepted Answer

Configure the DynamicFrame read with the mergeSchema option set to true so new columns are merged across files.. Glue's DynamicFrame reader supports merging schemas across multiple input files when mergeSchema is enabled. This lets a job that reads JSON incorporate columns that appear in newer files even if earlier files lacked them, avoiding manual schema edits. It is the appropriate mechanism for graceful schema evolution during a Glue Studio transformation rather than bookmarks, partition discovery, or type resolution.

Answer

Enable job bookmarks so that only new files are processed on each run.

Answer

Enable the AWS Glue Data Catalog schema evolution option on the table so new partitions are discovered.

Answer

Use the ResolveChoice transform with the makeStruct option to flatten nested JSON structures.

Question 11

A data engineer is using AWS Glue Studio to create a job that reads from an Amazon S3 bucket containing nested JSON data. The engineer needs to flatten the nested structure and write the output to Amazon S3 in Parquet format. The job must handle schema evolution, where new fields may be added to the JSON in the future. Which combination of steps should the engineer take to meet these requirements?

Accepted Answer

Use the Glue DynamicFrame 'Relationalize' transform to flatten the nested JSON, then write to Parquet with the 'updateBehavior' set to 'UPDATE_IN_DATABASE' in the Glue Data Catalog.. The Relationalize transform is specifically designed to flatten nested JSON into a relational structure. When writing the output, setting 'updateBehavior' to 'UPDATE_IN_DATABASE' allows the Glue Data Catalog to be updated with new fields, enabling schema evolution. This combination is the correct way to flatten nested JSON and handle future schema changes in a Glue job.

Answer

Use the Glue 'ResolveChoice' transform to flatten the nested JSON, then write to Parquet with the 'mergeSchema' option set to true.

Answer

Use the Glue 'ApplyMapping' transform to manually map all nested fields to a flat schema, then write to Parquet with the 'enableUpdateCatalog' option set to true.

Answer

Use the Glue 'Unnest' transform to flatten the JSON, then write to Parquet with the 'partitionKeys' option to enable schema evolution.

Question 12

A data engineer is using AWS Glue to transform data from an Amazon S3 bucket. The source data is in Parquet format and is partitioned by year, month, and day. The engineer needs to run a nightly job that processes only the previous day's partition. The job must be efficient and minimize data scanned. Which approach should the engineer take?

Accepted Answer

Create a Glue job that uses the AWS Glue Data Catalog partition index to push down a predicate for the previous day's partition.. Partition indexes in the AWS Glue Data Catalog enable efficient partition pruning by allowing the query engine to push down predicates and retrieve only relevant partition metadata. When a Glue job uses the Data Catalog as a source with a partition index on the date columns, it can directly access the previous day's partition without scanning the entire dataset, reducing both cost and runtime.

Answer

Use a Glue crawler to catalog the entire bucket, then create a job that reads all partitions and filters by date in the script.

Answer

Enable job bookmarks on the S3 source and set the transformation context to only process new files.

Answer

Configure the Glue job to use a custom JDBC driver to query the S3 bucket directly with a SQL WHERE clause.

Question 13

A data engineer is designing a pipeline that must ingest semi-structured JSON events from an Amazon Kinesis Data Stream into an Amazon S3 bucket, with each record landing as a separate object and the object key automatically including the event's partition key. The engineer wants to avoid writing any consumer application code. Which approach should the engineer use?

Accepted Answer

Create an Amazon Kinesis Data Firehose delivery stream with the Kinesis Data Stream as the source, enable record format conversion to Parquet, and set the S3 prefix to the partition key using dynamic partitioning.. Kinesis Data Firehose is the fully managed ingestion service that can consume directly from a Kinesis Data Stream and deliver to Amazon S3 without custom code. Dynamic partitioning lets Firehose derive the S3 prefix from a field in each record, so the event's partition key becomes part of the object key. This satisfies both the no-code requirement and the key-naming requirement.

Answer

Configure an Amazon S3 event notification on the Kinesis Data Stream to automatically copy records into objects using the partition key as the key name.

Answer

Use AWS Glue streaming ETL with a Kinesis Data Stream source and write to S3 using a DynamicFrame with a partition key column.

Answer

Deploy an AWS Lambda function with an event source mapping on the Kinesis Data Stream that calls the S3 PutObject API for each record.

Question 14

A data engineer is configuring AWS Glue DataBrew to clean a dataset stored in Amazon S3. The dataset contains a column named 'signup_date' with values in multiple formats (for example, '2024-01-15', '01/15/2024', and '15-Jan-2024'). The engineer needs to standardize all values to the ISO 8601 format 'YYYY-MM-DD'. Which DataBrew transformation should the engineer apply?

Accepted Answer

Use the 'Standardize date' transformation with the target format 'yyyy-MM-dd'.. AWS Glue DataBrew provides a dedicated 'Standardize date' transformation that parses date strings in various formats and outputs them in a chosen target format. For the scenario, selecting 'yyyy-MM-dd' ensures all signup_date values become ISO 8601 compliant, which is essential for downstream analytics and partitioning. Other transformations either do not exist or do not handle multi-format date parsing.

Answer

Use the 'Format date' transformation with the pattern 'yyyy-MM-dd'.

Answer

Use the 'Replace value' transformation with a regular expression to reformat each date variant.

Answer

Use the 'Change data type' transformation to convert the column to a date type with format 'yyyy-MM-dd'.

Question 15

A data engineer maintains an AWS Glue job that reads JSON from Amazon S3, flattens nested arrays, and writes Parquet to a target S3 prefix. Downstream consumers complain that records keep reappearing after each run and that some records are duplicated within a single run. The job processes files that are appended to the source prefix throughout the day, and the target prefix is overwritten on every run. Which combination of changes best resolves both symptoms?

Accepted Answer

Enable job bookmarks and change the write mode so the job appends only new records rather than overwriting the target.. Records reappearing across runs point to the job rereading all source objects each time, which job bookmarks prevent by tracking processed files. Duplicates within a run plus overwriting the target mean previously delivered data is rewritten, so switching to append semantics stops re-delivery. Applying both changes addresses the two distinct symptoms, while performance-oriented changes such as more workers or format conversion leave the ingestion semantics unchanged.

Answer

Convert the source JSON to Parquet first and then run the existing job against the Parquet source.

Answer

Add a DropDuplicates transform on the primary key and keep overwriting the target prefix.

Answer

Increase the number of DPUs and enable the Glue Data Catalog partition projection on the source table.