DEA-C01 Data Ingestion and Transformation Practice Test 26 — 15 Questions

Question 1

A data engineer is building a transformation job in AWS Glue Studio that must join a 40 GB Parquet dataset in Amazon S3 with a 2 MB lookup table of product codes also stored in S3. The job runs nightly and must complete with minimal shuffle overhead. Which join strategy should the engineer configure for this join in the Glue job's transform node?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Broadcast join. The 40 GB side is far too large to broadcast, while the 2 MB lookup table is small enough to replicate cheaply to every executor. Broadcasting the small table keeps the large Parquet scan in place and eliminates the network-wide redistribution that a sort-merge join would impose, so it is the only strategy that both produces the enriched output and minimizes shuffle.

Answer

Left semi join

Answer

Nested-loop join

Answer

Sort-merge join

Question 2

A data engineer is using AWS Glue to process data from an Amazon S3 bucket. The job reads CSV files, applies transformations, and writes the output to another S3 bucket in Parquet format. The engineer notices that the job is running slowly and wants to optimize performance. The job has 10 DPUs allocated. Which two actions should the engineer take to improve the job's performance? (Choose two.)

Accepted Answer

Use the Glue 'groupFiles' option to combine small files into larger ones for more efficient processing.. Increasing DPUs provides more compute resources for parallel processing, which can speed up a job that is resource-constrained. Using the 'groupFiles' option reduces the overhead of reading many small files by combining them into larger partitions. Both actions directly target common performance bottlenecks in Glue jobs: insufficient parallelism and inefficient small file handling.

Answer

Enable job bookmarks to track processed data and avoid reprocessing.

Answer

Convert the input CSV files to Parquet before processing to reduce I/O.

Answer

Set the '--conf' parameter to increase the Spark executor memory.

Question 3

A data engineer is designing an AWS Glue ETL job that reads from an Amazon S3 bucket containing nested JSON files and writes to Amazon Redshift. The engineer needs to flatten the nested structure and handle schema evolution across daily files. Which two AWS Glue features should be used to meet these requirements? (Choose two.)

Accepted Answer

Use the AWS Glue Data Catalog with a crawler configured to update the table schema as new files arrive.. Relationalize flattens nested JSON into relational tables, and a crawler-updated Data Catalog table accommodates schema evolution by refreshing column definitions as new files arrive. Together they address both the flattening and the changing-schema requirements. Job bookmarks, auto-scaling, and ResolveChoice do not fulfill these needs.

Answer

Enable job bookmarks to track processed S3 objects and avoid reprocessing.

Answer

Use the ResolveChoice transform to automatically merge columns with different data types.

Answer

Set the job's '--enable-auto-scaling' parameter to allow the job to adapt to schema changes.

Question 4

A data engineer is building a Glue ETL pipeline that reads semi-structured JSON event logs from Amazon S3, and downstream analysts need to query the results with Amazon Athena using partition pruning on event date. The JSON contains inconsistent keys across records. Which two actions should the engineer take to make the output efficiently queryable in Athena? (Choose two.)

Accepted Answer

Run an AWS Glue crawler on the output location so the Data Catalog captures the schema and partition metadata.. Athena performance and cost depend on scanning as little data as possible. Storing results as columnar Parquet means only referenced columns are read, and Hive-style date prefixes let Athena skip whole partitions. Because Athena resolves tables, columns, and partitions through the AWS Glue Data Catalog, running a crawler over the Parquet output registers that metadata. Bookmarks and compression address ingestion and storage size, not query efficiency.

Answer

Compress the output using GZIP and keep a single large file per day.

Answer

Enable job bookmarks on the Glue job so only new event files are processed.

Answer

Store the output as newline-delimited JSON partitioned by event date only.

Question 5

A data engineer is designing a pipeline to ingest data from an Amazon Kinesis Data Stream into an Amazon S3 bucket using AWS Glue Streaming ETL. The engineer needs to ensure that the job processes records in near real-time and handles checkpoints to avoid reprocessing. Which two configurations are required to achieve this? (Choose two.)

Accepted Answer

Specify a checkpoint location in Amazon S3 for the streaming job.. For AWS Glue Streaming ETL jobs, checkpointing is managed by the underlying Spark Structured Streaming engine. A checkpoint location in Amazon S3 must be specified to store progress and state, and using the 'foreachBatch' sink with a unique checkpoint directory ensures reliable exactly-once processing. Other options like job bookmarks or starting position do not provide checkpointing.

Answer

Enable auto-scaling for the Glue job to handle variable throughput.

Answer

Set the job's '--startingPosition' parameter to 'LATEST'.

Answer

Enable job bookmarks for the Glue streaming job.

Question 6

A data engineer is building an AWS Glue ETL job that reads a large Parquet dataset stored in Amazon S3 and writes the results to a target S3 location. The job must add a new column named `ingest_date` containing the current date to every row and must run nightly in an idempotent way. Which approach BEST accomplishes this in the AWS Glue job script?

Accepted Answer

Convert the DynamicFrame to a Spark DataFrame, use `withColumn` with `lit(datetime.date.today().isoformat())`, and write in `overwrite` mode with `partitionBy` on the date column.. Adding a constant column is most reliably done with the Spark DataFrame `withColumn` and `lit` function after converting from a DynamicFrame. Writing with `overwrite` mode and `partitionBy` on the date column ensures each nightly run replaces the same partition, making the job idempotent. The other transforms either do not add columns or use append semantics that duplicate data on reruns.

Answer

Use the Glue DynamicFrame `Map` transform with a Python function that appends the `ingest_date` column, then write with `glueContext.write_dynamic_frame.from_options`.

Answer

Use the Glue `ResolveChoice` transform to add the `ingest_date` column and write using `append` mode to the target S3 location.

Answer

Use the Glue `Relationalize` transform to flatten the schema, then use `write_dynamic_frame.from_catalog` with the default `append` mode.

Question 7

A data engineer is building an AWS Glue ETL job that reads from an Amazon S3 bucket and writes to Amazon Redshift. The engineer wants to improve job performance by reducing the amount of data shuffled and by pushing down predicates to the source. Which two strategies should the engineer apply? (Choose two.)

Accepted Answer

Use pushdown predicates on the DynamicFrame read to filter rows before the data is loaded into the job.. Pushdown predicates, whether configured as a read option or applied at the source, filter data before it enters the Glue job and allow partition pruning on S3. Both reduce the volume of data read and shuffled. Increasing partitions or filtering after the read would add shuffle rather than reduce it, and job bookmarks address incremental runs, not single-run shuffle.

Answer

Apply a filter transform immediately after reading rather than relying on the source to filter.

Answer

Call repartition on the DynamicFrame before the join to increase the number of partitions.

Answer

Enable job bookmarks so that only new data is processed on subsequent runs.

Question 8

A data engineer needs to ingest data from a REST API that returns up to 500 MB of JSON per invocation. The API requires pagination and rate limiting. The engineer wants to use a serverless AWS service to call the API, handle pagination, and store the raw JSON in Amazon S3 without managing servers. The data must be ingested every 15 minutes. Which solution meets these requirements with the LEAST operational overhead?

Accepted Answer

Use AWS Lambda with a function that calls the API, handles pagination, and writes to S3; schedule it with Amazon EventBridge every 15 minutes.. The requirement is for a serverless, low-overhead solution to ingest API data every 15 minutes. AWS Lambda scheduled by Amazon EventBridge is the most straightforward approach: it runs code on a schedule, handles pagination and rate limiting within the function, and writes directly to S3. Other services either require more management or do not support pulling from an API with pagination.

Answer

Use AWS Step Functions with a Lambda function to call the API, handle pagination, and write to S3; schedule with EventBridge.

Answer

Use Amazon Kinesis Data Firehose with an HTTP endpoint source to pull data from the API and deliver to S3.

Answer

Use AWS Glue Python shell job with the requests library to call the API, handle pagination, and write to S3.

Question 9

A data engineer needs to create an AWS Glue Data Catalog table for data stored in Amazon S3. The data is in Parquet format and partitioned by year, month, and day. The engineer wants to ensure that the table can be queried efficiently with Amazon Athena and that new partitions are automatically added when new data arrives. Which approach should the engineer use?

Accepted Answer

Use an AWS Glue crawler to crawl the S3 bucket and create the table, then schedule the crawler to run periodically to detect new partitions.. An AWS Glue crawler automatically infers the schema and partitions from S3 data and updates the Glue Data Catalog. Scheduling it to run periodically ensures new partitions are detected and added, making them queryable in Athena. This is the standard, automated approach for cataloging partitioned data in S3.

Answer

Create an AWS Glue ETL job that reads the S3 data and writes it to a new location, then use the job to update the Data Catalog with new partitions.

Answer

Use AWS Lake Formation to create the table and configure a partition projection, then use Athena to query the data.

Answer

Manually create the table in the Glue Data Catalog using the AWS Management Console, specifying the partition columns, and use Athena's MSCK REPAIR TABLE command to add new partitions.

Question 10

A data engineer is using AWS Glue to transform data from an Amazon S3 bucket. The job reads JSON files, applies complex transformations, and writes to another S3 bucket in Parquet format. The job is taking too long and occasionally fails with out-of-memory errors. The engineer needs to improve performance and reliability without changing the output format. Which action should the engineer take?

Accepted Answer

Repartition the data in the Glue job before writing, and use a larger worker type.. Out-of-memory errors in AWS Glue often stem from data skew or insufficient memory per worker. Repartitioning the data ensures even distribution, and switching to a larger worker type increases available memory. This combination directly addresses both performance and reliability while keeping the Parquet output. Other options like adding DPUs or grouping files may not resolve skew-related memory issues.

Answer

Enable job bookmarks to track processed data.

Answer

Increase the number of DPUs allocated to the job.

Answer

Use the groupFiles and groupSize parameters to combine small files.

Question 11

A data engineer is using AWS Glue Studio to build a visual ETL job that joins a 50 GB Amazon S3 Parquet dataset with a 2 MB reference lookup table stored as a CSV in Amazon S3. The lookup table is updated weekly and must reflect the latest values on every job run. The engineer needs the join to execute efficiently without shuffling the large dataset. Which approach should the engineer use in the Glue job?

Accepted Answer

Leave the default Join transform; Glue will automatically broadcast the small table because it is under the autoBroadcastJoinThreshold.. Spark's broadcast join strategy activates automatically when one side of the join is smaller than spark.sql.autoBroadcastJoinThreshold, which defaults to 10 MB in AWS Glue. Because the reference CSV is only 2 MB, the engine broadcasts it to every executor, eliminating the shuffle of the 50 GB Parquet dataset. No manual broadcast configuration is needed in Glue Studio.

Answer

Configure the Join transform with the small CSV as the left side and enable the 'Broadcast' option for the small table.

Answer

Repartition both datasets on the join key before the Join transform to co-locate matching rows.

Answer

Convert the large Parquet dataset to CSV first so the Join transform can perform a sort-merge join more efficiently.

Question 12

A data engineer is building a streaming ingestion pipeline where Amazon Kinesis Data Firehose delivers records to Amazon S3, and a downstream AWS Glue job processes them. The engineer needs the Firehose-delivered objects to be efficiently queryable by Amazon Athena and wants to minimize small-file overhead. Which two configuration choices best achieve these goals? (Choose two.)

Accepted Answer

Enable dynamic partitioning in the Firehose delivery stream using a JQ expression on a timestamp field in each record.. Dynamic partitioning with a JQ expression on a timestamp field writes records into Hive-style prefixes that Athena can prune, reducing listing and scan cost. Enabling record format conversion to Parquet or ORC with a Glue table as the schema source produces columnar objects that Athena reads far more efficiently than line-delimited JSON. Together they address both query efficiency and small-file overhead.

Answer

Disable S3 object-level encryption so Firehose can write larger objects without encryption overhead.

Answer

Attach an AWS Lambda function to the delivery stream that merges incoming objects into larger files before the S3 write.

Answer

Set the Firehose buffer interval to 60 seconds and the buffer size to 1 MB to flush records as quickly as possible.

Question 13

A data engineer is using AWS Glue to transform data from Amazon S3 and load it into Amazon Redshift. The engineer needs to ensure that the data is compressed to reduce storage and improve query performance. Which compression format should the engineer choose for the Redshift COPY command when the source files are in Parquet format?

Accepted Answer

No compression option should be specified; Redshift automatically handles Parquet compression.. Amazon Redshift COPY automatically handles compression for Parquet files because Parquet stores data in a compressed columnar format. Specifying a compression option like GZIP or LZO is unnecessary and can cause errors. The engineer should simply load the Parquet files without an explicit compression parameter.

Answer

Snappy

Answer

GZIP

Answer

LZO

Question 14

A data engineer is designing an ingestion pipeline for a data lake on Amazon S3. Source systems deliver files continuously throughout the day, and analysts query the data with Amazon Athena. The engineer wants new data to be queryable within minutes while keeping small-file count low and avoiding duplicate rows when a delivery is retried. Which combination of design choices best meets these requirements?

Accepted Answer

Land raw files in S3, run an hourly AWS Glue job that writes Parquet partitioned by date with job bookmarks enabled, and register the output in the AWS Glue Data Catalog. Hourly transformation into partitioned Parquet consolidates small incoming files, the Data Catalog exposes the result to Athena within minutes, and job bookmarks ensure a retried delivery is not processed twice. That trio satisfies freshness, file sizing, and idempotency, whereas the alternatives either slow queries, ignore deduplication, or change the query engine.

Answer

Stream files through Amazon Kinesis Data Firehose with a one-hour buffer interval into S3 and query with Athena

Answer

Write each incoming file directly to S3 as JSON and create an Athena table over the raw prefix with partition projection

Answer

Load all files into Amazon Redshift using COPY and let analysts query Redshift instead of Athena

Question 15

A data engineer is designing an ingestion pipeline where application logs are written as JSON files to an Amazon S3 bucket every few minutes. The engineer must catalog the data with AWS Glue so it can be queried in Amazon Athena, and the schema must handle new fields that appear over time without breaking existing queries. (Choose two.)

Accepted Answer

Use AWS Glue Data Catalog partition projection with a JSON SerDe and define the schema using an AWS Glue crawler that runs on a schedule.. The two effective approaches are a Glue crawler that updates the Data Catalog schema on each run with a JSON classifier, and partition projection combined with a scheduled crawler to keep partitions and columns current. Both allow new JSON fields to become queryable in Athena. Manually fixed schemas, SerDe error handling, and Parquet conversion without schema updates do not expose evolving fields.

Answer

Convert the JSON files to Parquet using AWS Glue and create an external table in Athena pointing to the Parquet files with a fixed schema.

Answer

Define the table manually in the Glue Data Catalog with a schema of only the known fields and rely on Athena to ignore unknown fields.

Answer

Enable schema evolution by setting the table's `SerDeInfo` to use `org.openx.data.jsonserde.JsonSerDe` with `ignore.malformed.json` set to true.