DEA-C01 Data Ingestion and Transformation Practice Test 24 — 15 Questions

Question 1

A data engineer needs to run a daily AWS Glue ETL job that reads from Amazon S3 and writes to Amazon Redshift. The job must run only after an upstream process has finished writing the day's files. The engineer wants to schedule the Glue job and have it start when the upstream process signals completion, rather than at a fixed time. Which approach should the engineer use?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Create an AWS Glue trigger of type 'EventBridge' (event-based) that starts the job in response to an Amazon EventBridge event emitted when the upstream process completes.. To start an AWS Glue job when an upstream process completes, the engineer should use an event-based trigger that responds to an Amazon EventBridge event. This allows the job to begin immediately after the completion signal instead of at a fixed time, avoiding reads of incomplete data. Scheduled, on-demand, and conditional triggers do not provide this event-driven behavior tied to the upstream process.

Answer

Create an AWS Glue trigger of type 'Scheduled' with a cron expression that runs every day at a fixed time after the upstream process usually completes.

Answer

Create an AWS Glue workflow with a 'Conditional' trigger that polls the S3 bucket every five minutes to check whether new files exist.

Answer

Create an AWS Glue trigger of type 'On-demand' and manually start the job each day after verifying the files are present.

Question 2

A data engineer is using AWS Glue to transform data from Amazon S3 and load it into Amazon Redshift. The engineer needs to ensure that the Glue job efficiently handles large datasets and minimizes the load time into Redshift. Which two actions should the engineer take to achieve these goals? (Choose two.)

Accepted Answer

Use the AWS Glue DynamicFrame write with the Redshift connection and enable the 'redshift_tmp_dir' parameter to stage data in S3 before loading.. To efficiently load large datasets into Redshift from Glue, using the 'redshift_tmp_dir' parameter to stage data in S3 and then leveraging the Redshift COPY command is essential. Converting data to Parquet before staging further optimizes the load by reducing data size and leveraging columnar storage. These two actions together minimize load time and resource usage. Other options, such as increasing DPUs or enabling auto-scaling, may improve processing but do not directly address the load efficiency into Redshift.

Answer

Enable auto-scaling for the Glue job to dynamically adjust resources based on workload.

Answer

Use AWS Glue job bookmarks to avoid reprocessing old data during each run.

Answer

Increase the number of DPUs for the Glue job to ensure faster data processing.

Question 3

A data engineer is building a transformation job in AWS Glue Studio that must read from an Amazon S3 bucket partitioned by year/month/day, join the data with a small reference table stored in Amazon DynamoDB, and write the result to Amazon Redshift. The engineer wants to minimize the amount of code written while keeping the job maintainable. Which approach should the engineer use to perform the join with the DynamoDB reference table?

Accepted Answer

Use the 'Join' transform with the S3 frame and a second frame created by reading the DynamoDB table with a Glue DynamicFrame source, configuring the join keys in the transform.. The Glue Studio visual Join transform can combine the S3 DynamicFrame with a DynamoDB-backed DynamicFrame produced by a DynamoDB source. This keeps the pipeline low-code while still performing the required join with the reference table. The engineer only needs to configure the join keys and join type in the visual editor.

Answer

Write a Python shell job that uses boto3 to scan DynamoDB and then calls the Glue 'Join' transform on the results.

Answer

Add a DynamoDB connection to the Glue job and use the Visual transform 'Join' to merge the S3 DynamicFrame with the DynamoDB table.

Answer

Use an AWS Glue custom connector for DynamoDB and then apply the 'Relationalize' transform before joining with the S3 data.

Question 4

A data engineer must ingest data from an Amazon DynamoDB table into an Amazon S3 bucket for analytics. The table has a high write volume, and the engineer needs incremental exports without consuming read capacity. The exports must capture the state of the table at a specific point in time. Which solution meets these requirements?

Accepted Answer

Use DynamoDB export to Amazon S3 with point-in-time recovery enabled, and schedule incremental exports.. DynamoDB export to Amazon S3 leverages point-in-time recovery to create a consistent snapshot without consuming read capacity, and it supports incremental exports by specifying a time range. This meets the high-write, incremental, and point-in-time requirements without impacting table performance. Other methods either consume capacity, lack snapshot consistency, or add unnecessary operational overhead.

Answer

Use AWS Database Migration Service (AWS DMS) with change data capture (CDC) to replicate changes to Amazon S3.

Answer

Use AWS Glue with a DynamoDB connection and enable job bookmarks to read only new items.

Answer

Enable DynamoDB Streams and use AWS Lambda to write each change to Amazon S3.

Question 5

A data engineer is using AWS Glue to transform data from an Amazon S3 bucket and write it to another S3 bucket in Parquet format. The source data is partitioned by year/month/day, and the job is configured to read from the source and write to the target with the same partitioning. The engineer notices that the job is producing a large number of small files in each partition, leading to poor query performance in Amazon Athena. The job uses the default Glue writer. Which change should the engineer make to reduce the number of small files?

Accepted Answer

Use the 'coalesce' or 'repartition' method in the Glue script to reduce the number of partitions before writing, and set the 'groupFiles' option to 'inPartition' or 'acrossPartitions'.. The number of output files in a Glue job is determined by the number of partitions in the DataFrame at write time. By coalescing or repartitioning to fewer partitions, and by using the groupFiles option to combine small input files, the job writes fewer, larger files. This improves Athena query performance by reducing the overhead of reading many small files. Increasing DPUs or enabling bookmarks does not address the root cause.

Answer

Enable job bookmarks to avoid reprocessing the same data and thus reduce the number of files written.

Answer

Increase the number of DPUs allocated to the job to allow more parallel tasks.

Answer

Change the output format to CSV and then use AWS Glue to convert to Parquet in a separate job.

Question 6

A data engineer configures an AWS Glue crawler to catalog a large S3 prefix containing Parquet files partitioned by year, month, and day. After the crawler runs, queries in Amazon Athena return zero rows even though the files contain data. The engineer verifies that the crawler created a single table with partition keys year, month, and day, but the partition locations in the Data Catalog point to the bucket root instead of the individual partition folders. What is the MOST likely cause?

Accepted Answer

The crawler's 'Include path' was set to the bucket root instead of the specific partition prefix, so partition locations were written relative to the wrong base path.. Partition locations in the Data Catalog are derived from the crawler's include path plus the discovered key structure. If the include path is the bucket root rather than the dataset prefix, partition storage descriptors can resolve to incorrect paths, so Athena scans empty locations and returns zero rows. Correcting the include path to the true prefix fixes partition locations and query output.

Answer

The IAM role used by the crawler lacks s3:GetObject permission on the partition folders, so the crawler could not enumerate keys.

Answer

The S3 prefix uses a non-Hive-style layout such as year=2024/month=01/, so the crawler cannot infer partition paths from the key structure.

Answer

The crawler was configured with 'Add new columns only' instead of 'Update all new and existing partitions with metadata from the table'.

Question 7

A data engineer runs an AWS Glue Spark job that reads a 2 TB Parquet dataset from Amazon S3 and writes an aggregated result. The job is taking many hours and CloudWatch shows most executors idle while a few do all the work. The engineer suspects a skewed join key. Which change most directly addresses the skew?

Accepted Answer

Repartition the larger input on the join key with a high partition count, or salt the hot key before the join.. Skew concentrates many rows with the same key into a single Spark partition, so only the executors owning that partition work. Repartitioning raises parallelism but a dominant key still lands on one partition, so salting the hot key (or otherwise splitting it) is the change that actually spreads the load and reduces the long tail.

Answer

Enable the Glue job bookmark and rerun the job so it only processes new data.

Answer

Increase the number of DPU capacity in the Glue job so more executors are available.

Answer

Change the output format from Parquet to CSV to reduce write time.

Question 8

A data engineer is designing a data ingestion pipeline using AWS Glue. The source data is in an Amazon S3 bucket, stored as nested JSON files. The engineer needs to flatten the nested structure and write the output to Amazon Redshift. The pipeline must handle schema changes gracefully and minimize manual intervention. Which two actions should the engineer take to meet these requirements? (Choose two.)

Accepted Answer

Use the AWS Glue Relationalize transform to flatten the nested JSON data.. The Relationalize transform flattens nested JSON into relational tables, which is essential for loading into Redshift. The ResolveChoice transform handles data type conflicts that can occur when schemas evolve, ensuring the job continues without failing. Together, they provide a robust way to process nested JSON with evolving schemas, minimizing manual intervention and making the pipeline resilient to changes.

Answer

Enable AWS Glue job bookmarks to track processed data and handle schema changes.

Answer

Create the AWS Glue table with a schema that includes all possible fields to accommodate future changes.

Answer

Configure the AWS Glue job to write to Redshift using the spark-redshift connector with pre-actions to drop and recreate the table.

Question 9

A data engineer is building an AWS Glue ETL job that joins a 40 GB Amazon S3 Parquet dataset with a 2 MB CSV lookup table containing currency exchange rates that changes daily. The engineer wants to avoid shuffling the tiny lookup dataset across the cluster on every run. Which AWS Glue transformation approach is MOST appropriate?

Accepted Answer

Use a broadcast join by passing the small DataFrame to the broadcast() function before the join.. Broadcasting the small lookup DataFrame is the standard Spark optimization when one side of a join is tiny. It avoids shuffling the large Parquet dataset by replicating the small table to each executor, which is far cheaper than a sort-merge join. The other choices either reintroduce shuffles, change services unnecessarily, or target scan efficiency instead of join behavior.

Answer

Write the lookup table to Amazon Redshift and use a Glue connection to join in Redshift.

Answer

Convert the CSV lookup to Parquet and rely on predicate pushdown to reduce shuffle.

Answer

Repartition both DataFrames to 200 partitions and then perform the join.

Question 10

A data engineer is using AWS Glue Studio to build a job that reads from an Amazon Kinesis Data Stream, performs a stateful aggregation over a 5-minute window, and writes results to Amazon S3. The job must handle late-arriving data up to 2 minutes and ensure exactly-once processing. Which combination of Glue features should the engineer use?

Accepted Answer

Use Glue streaming ETL with a 5-minute sliding window, set a watermark of 2 minutes, and enable checkpointing to an S3 bucket.. For streaming ETL with stateful windowed aggregations and exactly-once semantics, AWS Glue streaming jobs use sliding windows, watermarks for late data, and checkpointing to Amazon S3. A 2-minute watermark allows late events, and checkpointing ensures that each record is processed exactly once by storing state and stream offsets. The other options either use batch processing, omit checkpointing, or misapply job bookmarks, failing the requirements.

Answer

Use Glue streaming ETL with a 5-minute tumbling window and enable job bookmarks for exactly-once processing.

Answer

Use Glue streaming ETL with a 5-minute tumbling window, set a watermark of 2 minutes, and disable checkpointing to reduce latency.

Answer

Use Glue batch ETL with a trigger that runs every 5 minutes and enable job bookmarks.

Question 11

A data engineer is building a Glue ETL job that reads JSON files from Amazon S3 where each file contains nested arrays and a top-level 'id' field. The job must flatten the nested arrays and write Parquet output partitioned by a 'year' column derived from a timestamp field. The engineer wants to minimize the amount of code written and avoid manual schema definition. Which approach should the engineer use?

Accepted Answer

Use the AWS Glue DynamicFrame with the Relationalize transform and then write the output with partitionKeys set to the derived year column.. Relationalize is the Glue transform purpose-built for unnesting complex JSON structures, producing related DynamicFrames that can then be written as Parquet. Using partitionKeys on the write operation derives and applies partitioning without manual schema work, directly satisfying the requirement to minimize custom code and avoid hand-written schemas.

Answer

Use Spark SQL with explode() functions inside a Glue job, manually defining the schema with StructType before writing partitioned Parquet.

Answer

Configure a Glue crawler with a custom classifier for nested JSON and let the crawler create the flattened table, then run a Glue job with applymapping.

Answer

Load the JSON into an AWS Glue DataBrew recipe and apply a flatten transformation, then output to S3 as Parquet with partitioning.

Question 12

A data engineer is designing a pipeline to ingest data from an Amazon RDS for PostgreSQL database into Amazon S3. The data volume is large, and the engineer wants to minimize the impact on the source database during the initial full load. The engineer plans to use AWS Database Migration Service (DMS). Which two actions should the engineer take to minimize the impact on the source database? (Choose two.)

Accepted Answer

Use a read replica as the source for the DMS full load.. To minimize impact on the source database during a full load with AWS DMS, use a read replica to offload read traffic from the primary, and schedule the load during off-peak hours to avoid contention with production workloads. These two actions reduce the performance impact. Other options like enabling CDC or increasing threads do not reduce impact during the full load.

Answer

Enable multi-threaded full load with a high number of threads.

Answer

Configure DMS to use a smaller batch size for the full load.

Answer

Enable change data capture (CDC) on the source database.

Question 13

A data engineer must load a 500 GB uncompressed CSV dataset from Amazon S3 into Amazon Redshift using the COPY command. The cluster has four ra3.4xlarge nodes. Which configuration will MOST improve load throughput?

Accepted Answer

Split the CSV into multiple files of roughly equal size and run COPY with the PARALLEL ON option.. Redshift COPY achieves maximum throughput when input is split into multiple files of similar size so each node slice can read in parallel. A single 500 GB file constrains parallelism. Disabling compression or statistics updates only trims maintenance time, raising MAXERROR changes error tolerance, and GZIP requires re-encoding and still favors multiple files.

Answer

Run COPY with COMPUPDATE OFF and STATUPDATE OFF to reduce post-load overhead.

Answer

Use a single COPY command with the GZIP option to reduce data transfer.

Answer

Set the MAXERROR option to 100000 to allow more rows to be skipped.

Question 14

A data engineer is using AWS Glue to transform data and needs to handle a column that contains values of mixed types (some integers, some strings) in the source data. The engineer wants to keep both types in separate columns for further processing. Which AWS Glue transform should be used?

Accepted Answer

ResolveChoice. ResolveChoice is the AWS Glue transform built to handle columns with mixed data types. It can split a column into multiple columns, each containing one type, or cast to a single type. The engineer can use the 'make_struct' option to keep both types in a struct, or 'project' to create separate columns. This directly solves the problem of mixed types.

Answer

ApplyMapping

Answer

Filter

Answer

Relationalize

Question 15

A data engineer is designing an ingestion pipeline where an AWS Glue job reads from a JDBC source and writes to Amazon S3 in Parquet. The job must handle schema evolution when new columns are added to the source table. The engineer wants to use Glue job bookmarks to avoid reprocessing old data. Which two configurations or practices are required to meet these requirements? (Choose two.)

Accepted Answer

Use the Glue DynamicFrame resolveChoice transform with the 'make_cols' option to merge schemas and add new columns as they appear.. Job bookmarks require a reliable ordering key for JDBC sources; without a primary key or increasing column, incremental tracking fails. Separately, resolveChoice with make_cols reconciles catalog and data schemas so newly added source columns are retained. Together these two configurations satisfy incremental processing and schema evolution in a Glue JDBC-to-S3 pipeline.

Answer

Configure the JDBC connection with the 'customJdbcDriverS3Path' parameter to allow dynamic schema refresh at runtime.

Answer

Disable the Glue Data Catalog update and rely on the Spark DataFrame API to infer schema on every run.

Answer

Set the job parameter --enable-auto-scaling to true so that new columns are detected dynamically during the run.