DEA-C01 Data Ingestion and Transformation Practice Test 50 — 15 Questions

Question 1

A data engineer needs to run a daily AWS Glue ETL job that transforms data in Amazon S3. The job must start at 2:00 AM UTC every day. The engineer wants to minimize operational overhead and ensure the job runs reliably. Which approach should the engineer use?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Create an AWS Glue trigger of type 'Schedule' with a cron expression '0 2 * * ? *' and attach it to the job.. AWS Glue provides built-in schedule triggers that use cron expressions to start jobs at specified times. Creating a trigger with the cron expression '0 2 * * ? *' and attaching it to the job is the simplest, most reliable method. It requires no additional services, reducing operational overhead and potential failure points, while ensuring the job runs daily at 2:00 AM UTC.

Answer

Configure an AWS Lambda function with a CloudWatch Events rule to start the Glue job daily at 2:00 AM UTC.

Answer

Use Amazon EventBridge to create a rule with a cron expression '0 2 * * ? *' that targets the AWS Glue job.

Answer

Use AWS Step Functions with a Wait state to delay execution until 2:00 AM UTC and then start the Glue job.

Question 2

A data engineer is using AWS Glue to transform data from Amazon S3. The source data is in CSV format with inconsistent date formats across files (e.g., 'MM/DD/YYYY' and 'YYYY-MM-DD'). The engineer needs to standardize all dates to 'YYYY-MM-DD' format in the output. Which AWS Glue transform should the engineer use to achieve this?

Accepted Answer

Map. The `Map` transform in AWS Glue enables custom record-level transformations. By writing a Python function that parses various date formats and outputs a standardized string, the engineer can handle inconsistent date formats. This approach is flexible and can be applied across all records, ensuring uniform output.

Answer

ApplyMapping

Answer

Filter

Answer

ResolveChoice

Question 3

A data engineer is using AWS Database Migration Service (AWS DMS) to replicate ongoing changes from an Amazon RDS for PostgreSQL database to an Amazon S3 bucket. The source table has a primary key and the engineer needs near-real-time change data capture (CDC) with minimal impact on the source. Which DMS task setting should the engineer configure to meet these requirements?

Accepted Answer

Set the migration type to full load plus CDC and enable logical replication on the source. AWS DMS uses PostgreSQL logical replication to read changes from the write-ahead log for CDC. Configuring the task as full load plus CDC performs the initial copy and then continuously applies changes, giving near-real-time replication with minimal source impact because changes are read from the log rather than by polling tables. Enabling logical replication on the source is a prerequisite for this mode.

Answer

Set the migration type to full load only and schedule hourly re-runs

Answer

Set the migration type to CDC only and configure a change table on the source

Answer

Set the migration type to full load only and enable binary logging on the source

Question 4

A data engineer runs an AWS Glue job that reads Parquet files from Amazon S3 partitioned by year/month/day and writes to another S3 prefix. The job currently processes all historical partitions on every run, causing long runtimes and high cost. The engineer wants subsequent runs to process only new data. Which configuration should the engineer apply?

Accepted Answer

Enable AWS Glue job bookmarks and ensure the job's source and target paths are stable across runs.. Job bookmarks maintain persistent state across runs, allowing Glue to skip files and partitions that were already processed in prior executions. This is the native mechanism for incremental processing in Glue ETL, directly reducing scanned data, runtime, and cost without changing the transformation logic or adding external state tracking.

Answer

Configure an S3 Lifecycle rule to transition old partitions to S3 Glacier Instant Retrieval so the Glue job reads them faster.

Answer

Increase the number of AWS Glue DPUs allocated to the job so it can process all partitions faster.

Answer

Set the Glue job's MaxConcurrentRuns parameter to 1 to prevent overlapping executions from reprocessing data.

Question 5

A data engineer runs an AWS Glue ETL job that reads semi-structured JSON from Amazon S3, flattens nested arrays, and writes Parquet to a partitioned S3 location. Job runs are becoming expensive because Glue reprocesses all historical partitions on every run. The engineer wants subsequent runs to process only newly arrived data. Which approach should the engineer take with the LEAST operational overhead?

Accepted Answer

Create an AWS Glue job bookmark and enable it on the S3 data source, then let the job read only new objects on each run.. Glue job bookmarks maintain run state and cause the reader to process only new or changed S3 objects since the last successful run. This eliminates repeated scanning of historical partitions with no custom code, unlike runtime filters or schedule changes, which still read all objects. Moving data to RDS is a much heavier architectural change than needed.

Answer

Schedule the job with a shorter interval so each run has less data to process.

Answer

Move the source data to an Amazon RDS table and use a Glue JDBC connection with a watermark column.

Answer

Add a WHERE clause to the DynamicFrame that filters records by the current date at runtime.

Question 6

A data engineer needs to load data from an Amazon DynamoDB table into an Amazon S3 data lake nightly. The table is large and the engineer wants to avoid consuming provisioned read capacity on the live table. Which approach should the engineer use?

Accepted Answer

Use the DynamoDB export to Amazon S3 feature to export a point-in-time snapshot to S3 without consuming read capacity.. DynamoDB export to S3 produces a point-in-time snapshot directly in Amazon S3 without consuming read capacity on the source table, making it ideal for nightly analytics loads that must not impact production traffic. The exported data is written in a format suitable for downstream processing by services such as Athena or Glue.

Answer

Create an Amazon ElastiCache for Redis cluster and replicate the DynamoDB table into it, then export from Redis to Amazon S3.

Answer

Enable DynamoDB Streams on the table and use an AWS Lambda function to write each change to Amazon S3 in near real time.

Answer

Use AWS Glue with a DynamoDB connection and read the table directly with the DynamoDB reader, accepting the consumed read capacity.

Question 7

A data engineer is loading data from Amazon S3 into an Amazon Redshift cluster using the COPY command. The S3 bucket contains 500 Parquet files, each about 200 MB, in a single prefix. The COPY job is running slowly and consuming excessive cluster resources. The engineer wants to improve performance without changing the data format or the cluster size. Which action should the engineer take?

Accepted Answer

Split the data into multiple prefixes and run multiple COPY commands in parallel, or use a manifest file to distribute the load across slices.. Redshift COPY achieves high throughput by having multiple slices read files in parallel. When hundreds of files sit in one prefix, the leader node may serialize listing and assignment, and some slices may read more files than others, causing skew and resource contention. Distributing files across prefixes or using a manifest gives COPY finer-grained control over parallel reads, balancing work across slices and improving load speed without altering the format or cluster size.

Answer

Use the COPY command with the PARALLEL OFF option to reduce the number of slices used.

Answer

Convert the Parquet files to CSV and load them with the CSV option to enable faster parsing.

Answer

Add the COMPUPDATE OFF and STATUPDATE OFF options to the COPY command to skip compression and statistics updates.

Question 8

A data engineer is configuring an AWS Glue crawler to catalog data stored in Amazon S3. The data is organized as Parquet files under prefixes named by year, month, and day, such as s3://analytics/events/year=2024/month=05/day=17/. Queries in Amazon Athena must use partition pruning to limit scanned data. Which crawler configuration should the engineer choose?

Accepted Answer

Create a crawler with the S3 path pointing to s3://analytics/events/ and enable 'Update all new and existing partitions with metadata from the table' so partition metadata stays current.. Hive-style prefixes such as year=/month=/day= are recognized as partition keys when an AWS Glue crawler targets the parent S3 prefix. To keep the catalog current as new date prefixes appear, the crawler should be configured to update all new and existing partitions with metadata from the table. Athena then reads the partition keys from the Data Catalog and prunes to only the relevant prefixes, reducing scanned bytes and query cost.

Answer

Create a crawler for each individual day prefix, such as s3://analytics/events/year=2024/month=05/day=17/, so each day becomes its own table.

Answer

Create a crawler with the S3 path pointing to s3://analytics/events/ and set the 'Schema change policy' to 'Delete tables and columns' so stale partitions are removed.

Answer

Create a crawler with the S3 path pointing to s3://analytics/events/ and enable the 'Add new columns only' option so new partitions are detected automatically.

Question 9

A data engineer needs to run an AWS Glue for Apache Spark ETL job that joins a 40 GB Amazon S3 Parquet dataset with a small 8 MB reference lookup table stored as CSV in Amazon S3. The reference table is read on every join and the job's executors are spending a large amount of shuffle time on the join. The reference table changes only once per month. Which approach MOST efficiently reduces shuffle overhead in the Glue job?

Accepted Answer

Broadcast the reference table by reading it with the Spark DataFrame API and applying broadcast() before the join.. The small, slowly changing reference table is an ideal broadcast candidate. Replicating it to every executor lets Spark perform a broadcast hash join where the large Parquet side stays in place and is never redistributed. Increasing DPUs, converting formats, or pre-partitioning all leave the shuffle of the large dataset intact, so they fail to address the actual bottleneck.

Answer

Partition both datasets by the join key and write them to S3 before the join so Spark can perform a partition-wise merge join.

Answer

Increase the number of Glue DPUs so that more executors are available to parallelize the shuffle stage of the join.

Answer

Convert the reference CSV to Parquet and enable Glue job bookmarks so the lookup is only read once per month.

Question 10

A data engineer is using AWS Glue Studio to create a visual ETL job that reads from an Amazon S3 bucket containing JSON files, applies a filter transformation, and writes the output to Amazon Redshift. The job must run daily. The engineer notices that the job is taking a long time to complete and wants to improve performance. Which action should the engineer take to optimize the job?

Accepted Answer

Convert the source JSON files to Parquet format before running the ETL job.. Converting JSON to Parquet improves ETL performance because Parquet is columnar, compressed, and requires less I/O and CPU to parse. AWS Glue jobs benefit significantly from columnar formats when reading from S3. While increasing DPUs or enabling bookmarks can help in some cases, the most direct and effective optimization for slow JSON processing is to use a more efficient file format.

Answer

Use a larger number of smaller files instead of fewer large files.

Answer

Increase the number of AWS Glue DPUs allocated to the job.

Answer

Enable job bookmarks to track processed files and avoid reprocessing.

Question 11

A data engineer needs to transform data in Amazon S3 using SQL statements without managing any infrastructure. The transformations are simple projections and filters, and the engineer wants the results written back to S3 in Parquet. Which AWS service should be used?

Accepted Answer

Amazon Athena with a CREATE TABLE AS SELECT statement. Amazon Athena is fully serverless and executes SQL directly against S3 data. A CREATE TABLE AS SELECT statement reads the source, applies projections and filters, and writes the result to S3 in a chosen format like Parquet. This satisfies the SQL and no-infrastructure requirements without cluster or job management.

Answer

AWS Glue ETL with a PySpark script

Answer

Amazon EMR with Hive on a transient cluster

Answer

Amazon Redshift Spectrum with an external schema

Question 12

A data engineer is using AWS Glue to process a large dataset stored in Amazon S3. The dataset is partitioned by year/month/day and consists of Parquet files. The engineer notices that the Glue job is running slowly and consuming excessive DPU hours. The job performs a join between two large tables and writes the output back to S3. Which optimization technique should the engineer implement to improve performance and reduce cost?

Accepted Answer

Use predicate pushdown to filter data before the join and partition the output by a common column.. Predicate pushdown reduces the amount of data read and shuffled by applying filters early in the execution plan. Partitioning the output by a common column improves data layout for subsequent queries and reduces write overhead. Together, these techniques optimize join performance and lower DPU consumption, directly addressing the slow job and high cost. Other options either do not target the join inefficiency or would worsen performance.

Answer

Increase the number of DPUs allocated to the job to speed up execution.

Answer

Convert the Parquet files to CSV to reduce storage size and improve read performance.

Answer

Enable AWS Glue job bookmarks to avoid reprocessing old data.

Question 13

A data engineer is using AWS Glue Studio to build a job that reads from an Amazon S3 bucket containing nested JSON, flattens arrays using Relationalize, and writes Parquet to another S3 bucket. The job must run only when new objects land in the source bucket. The engineer wants to avoid unnecessary job runs and minimize cost. Which approach should the engineer use?

Accepted Answer

Configure an Amazon EventBridge rule that matches S3 Object Created events on the source prefix and targets the Glue job as the rule's target.. S3 can emit Object Created events that EventBridge matches by bucket and prefix, and EventBridge supports an AWS Glue job as a direct target. This is the canonical event-driven pattern: the job starts only when a new object lands, eliminating idle scheduled runs. Polling and SNS notifications either add latency or require manual steps, and a fixed cron schedule still burns DPUs on empty runs.

Answer

Create an AWS Lambda function that polls the S3 bucket using the ListObjectsV2 API on a one-minute schedule and starts the Glue job when new keys appear.

Answer

Schedule the Glue job with a cron trigger every five minutes and rely on Glue job bookmarks to skip already-processed files.

Answer

Enable S3 Event Notifications to publish to an Amazon SNS topic and subscribe the data engineer's email address to the topic.

Question 14

A data engineer is configuring an AWS Glue ETL job that reads from an Amazon S3 bucket and writes to another S3 bucket. The job must be able to handle a large number of small files efficiently and minimize the number of output files to improve downstream query performance. Which two actions should the engineer take? (Choose two.)

Accepted Answer

Use the AWS Glue groupFiles option to group multiple small input files into a single partition for processing.. To handle many small input files, AWS Glue provides the groupFiles option, which combines small files into larger partitions for processing. To minimize output files, the coalesce transformation reduces the number of partitions before writing, resulting in fewer, larger files. Together, these actions improve both read efficiency and downstream query performance. Increasing DPUs, enabling bookmarks, or using the Data Catalog do not directly address these file-count issues.

Answer

Enable the job bookmark feature to track previously processed files and avoid reprocessing.

Answer

Increase the number of DPUs allocated to the job to allow more parallel processing of the small files.

Answer

Use the AWS Glue Data Catalog to store metadata about the small files and let the job read from the catalog.

Question 15

A data engineer is using AWS Glue Studio to build a job that reads from Amazon S3, applies a filter, and writes to Amazon Redshift. The engineer needs to ensure the job can be rerun safely without creating duplicate rows in Redshift if a previous run partially succeeded. Which design choice best meets this requirement?

Accepted Answer

Write the output to a staging table in Redshift and use a MERGE or upsert operation keyed on a business key to apply changes idempotently.. Loading into a staging table and then merging into the target on a business key is the canonical idempotent pattern for Redshift. Reruns re-execute the same merge deterministically, so partial failures do not leave duplicates. This decouples the risky load from the final apply step and supports safe retries.

Answer

Enable AWS Glue job bookmarks and rely on them to prevent duplicate writes to Redshift on rerun.

Answer

Configure the Redshift write to use a pre-action and post-action that truncate the target table before each load.

Answer

Set the Glue job's write mode to append and add a unique constraint on the Redshift target table to reject duplicates.