DEA-C01 Data Ingestion and Transformation Practice Test 54 — 15 Questions

Question 1

A data engineer needs to run a Python-based transformation on each object as it lands in an Amazon S3 bucket. The objects are small (under 10 MB), arrive sporadically, and must be processed within seconds. Which approach is MOST appropriate?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Configure an S3 event notification to invoke an AWS Lambda function that performs the transformation.. S3 event notifications triggering Lambda is the standard serverless pattern for reacting to individual object uploads in near real time. It avoids polling, scales automatically, and suits small files and sporadic arrivals. Glue, Firehose, and EMR introduce batching, buffering, or cluster overhead that conflicts with the seconds-level, per-object requirement.

Answer

Schedule an AWS Glue crawler and ETL job to run every 15 minutes over the bucket.

Answer

Run an Amazon EMR cluster with a step function that polls the bucket every minute.

Answer

Use Amazon Kinesis Data Firehose to deliver objects to a processing endpoint.

Question 2

A data engineer needs to run a one-time AWS Glue ETL job that reads data from an Amazon S3 bucket and writes transformed Parquet files to another S3 bucket. The job does not need a schedule and should be run immediately after creation. Which method should the engineer use to start the job?

Accepted Answer

Use the AWS Glue console or AWS CLI to start the job run directly.. For a one-time, unscheduled AWS Glue job, the simplest way to start it is to invoke StartJobRun through the console, AWS CLI, or SDK. Triggers and workflows are used for scheduling or orchestration and are not needed when the job should run immediately after creation.

Answer

Configure a scheduled trigger with a cron expression that runs once.

Answer

Create a trigger with type ON_DEMAND and activate it.

Answer

Create a workflow and add the job to it, then start the workflow.

Question 3

A data engineer is configuring an AWS Glue ETL job that reads from an Amazon S3 bucket containing nested JSON and writes to a partitioned Parquet table. The engineer wants to reduce job cost and improve read performance. (Choose two.)

Accepted Answer

Enable job bookmarks to avoid reprocessing previously processed files.. Job bookmarks cut cost by skipping already-processed S3 objects, which is important for incremental pipelines. Partitioning the output Parquet table by low-cardinality filter columns enables partition pruning, reducing I/O and improving read performance for downstream queries. Crawlers, worker scaling, and pre-conversion do not directly deliver both cost reduction and read performance for this job configuration.

Answer

Convert the source JSON to Parquet before the Glue job runs.

Answer

Increase the number of workers to the maximum allowed for the job.

Answer

Use the Glue crawler to infer the schema and store it in the Data Catalog.

Question 4

A data engineer needs to run a one-time transformation on a 500 GB dataset stored in Amazon S3. The transformation is written in Python and uses pandas, which cannot handle the full dataset in memory. The engineer wants a serverless option that can parallelize the work without managing servers. Which AWS service should the engineer use?

Accepted Answer

Amazon EMR Serverless with a Spark job that uses Spark DataFrame operations instead of pandas.. EMR Serverless provides a serverless Spark environment that can distribute a large transformation across workers, removing the single-node memory limit of pandas. By converting the logic to Spark DataFrames, the engineer can process 500 GB in parallel without provisioning or managing servers, which matches the stated constraints.

Answer

AWS Lambda with a function that streams the S3 objects and processes them in 15-minute increments.

Answer

AWS Glue with a Python shell job and the pandas library, splitting the data into chunks.

Answer

Amazon Athena with a CREATE TABLE AS SELECT statement that applies the transformation in SQL.

Question 5

A data engineer runs an AWS Glue ETL job that joins a 4 TB Parquet dataset in Amazon S3 with a small 40 MB reference lookup table stored as a single CSV file in S3. The join is taking hours and the job frequently fails with executor out-of-memory errors. The engineer wants to reduce shuffle and memory pressure with the least development effort. Which approach should the engineer take?

Accepted Answer

Convert the small lookup CSV into a broadcast join by loading it with a broadcast hint, avoiding a shuffle of the large dataset.. Because one side of the join is only 40 MB, broadcasting that lookup table to every executor removes the need to shuffle the multi-terabyte dataset, cutting network I/O and eliminating the memory spikes that cause executor OOM failures. This is a minimal code change, satisfying the least-effort constraint, whereas scaling workers or repartitioning the large side still incurs a full shuffle.

Answer

Increase the number of Glue DPU workers so more executors can hold the shuffled join partitions.

Answer

Repartition the large Parquet dataset by the join key using a Glue transform before the join.

Answer

Enable the AWS Glue job bookmark and rerun the job so only new files are processed.

Question 6

A data engineer is building an AWS Glue ETL job that reads records from an Amazon Kinesis Data Stream and writes them to Amazon S3 in Parquet format. The job must checkpoint its progress so that it can resume without reprocessing data after a failure. Which AWS Glue mechanism should the engineer configure to track the stream position?

Accepted Answer

Enable job bookmarks with the transformation_ctx parameter set on the create_data_frame_from_options call.. Glue job bookmarks maintain state between runs, storing the last processed Kinesis sequence number per shard. With transformation_ctx set on the read operation, the job can resume from where it stopped after a failure, avoiding duplicate writes to S3. The other options address throughput, concurrency, or retry orchestration, none of which track record-level progress.

Answer

Use AWS Glue Studio's auto-generated script and enable the 'retry on failure' option in the job's trigger.

Answer

Set the Glue job's MaxConcurrentRuns parameter to 1 and let the job restart from the beginning on failure.

Answer

Configure a Kinesis Data Stream consumer with enhanced fan-out and rely on the stream's 24-hour retention to replay records.

Question 7

A data engineer must load a 2 GB uncompressed CSV file from Amazon S3 into Amazon Redshift using the COPY command. The cluster is a two-node ra3.xlplus cluster, and the load is running far slower than expected. The engineer wants the fastest reliable improvement without changing the cluster. What should the engineer do?

Accepted Answer

Split the single CSV file into multiple files and load them in parallel with a single COPY command.. Redshift distributes a COPY workload across slices, but each input file is read by only one slice. A single 2 GB CSV therefore runs on one slice while the rest idle. Splitting the file into several smaller files, ideally a multiple of the slice count, allows all slices to read concurrently, which is the fastest reliable fix that does not alter the cluster.

Answer

Convert the CSV to GZIP before loading and keep it as one file.

Answer

Add the COMPUPDATE OFF and STATUPDATE OFF parameters to the COPY command.

Answer

Increase the cluster to four nodes so more slices can read the same file simultaneously.

Question 8

A data engineer needs to run a transformation in AWS Glue where each record must be processed independently and the output schema is known ahead of time. The transformation should operate on a DynamicFrame and return a DynamicFrame. Which Glue transform is designed for this row-by-row operation?

Accepted Answer

Map. The Map transform is purpose-built for applying a function to each record and returning a DynamicFrame, which matches the need for independent row processing with a known output schema. Relationalize, ResolveChoice, and DropNullFields are schema or structure utilities, not general row-level mapping transforms.

Answer

DropNullFields

Answer

Relationalize

Answer

ResolveChoice

Question 9

A data engineer is using AWS Glue to read a large dataset from Amazon S3 and write it to Amazon Redshift. The job intermittently fails with 'Communication link failure' errors during the write phase. The dataset is several hundred gigabytes and the Redshift cluster is under heavy query load. Which change is MOST likely to resolve the failures while preserving data integrity?

Accepted Answer

Write the transformed data to Amazon S3 in Parquet, then use a Redshift COPY command from S3 instead of a direct JDBC write.. Staging data in S3 and using the Redshift COPY command is the recommended pattern for bulk loads. It minimizes open connections, leverages Redshift's parallel load capability, and provides transactional integrity. Increasing workers, extending timeouts, or switching to Spectrum does not resolve the underlying contention or integrity concerns during direct writes.

Answer

Switch the write operation to use Redshift Spectrum and query the S3 data directly without loading.

Answer

Increase the Glue job's number of workers and enable auto-scaling.

Answer

Configure the Glue connection to use a Redshift JDBC URL with a longer socketTimeout and retry the write.

Question 10

A data engineer is configuring an AWS Glue crawler to catalog CSV files stored in Amazon S3. The files are organized in prefixes by year and month, and the engineer wants the crawler to detect new partitions automatically and avoid re-crawling unchanged partitions. (Choose two.)

Accepted Answer

Configure the crawler to use incremental crawling so it processes only folders added since the last crawl.. Incremental crawling limits each crawler run to newly added folders, so previously cataloged partitions are not rescanned, reducing time and cost. Hive-style key=value prefixes make the year and month directories recognizable as partitions so the crawler can populate them correctly in the Data Catalog. Together they deliver automatic partition detection without reprocessing unchanged data.

Answer

Set the crawler's schema change policy to update the table definition in the Data Catalog.

Answer

Add a path in the crawler configuration that points to each year and month prefix individually.

Answer

Create a partition index on the Data Catalog table to speed up partition filtering.

Question 11

A data engineer is using AWS Glue DataBrew to clean a dataset stored in Amazon S3. The recipe must replace all null values in a specific column with the string 'UNKNOWN' and then convert the column to uppercase. The engineer wants to apply these steps in a repeatable recipe. Which combination of DataBrew transforms should be used?

Accepted Answer

Use the 'Fill missing values' transform with a custom value, then the 'Upper case' transform on the column.. DataBrew provides a dedicated 'Fill missing values' transform for substituting nulls with a constant, and an 'Upper case' transform for case normalization. Together they form a repeatable recipe that preserves rows and produces the required output. Removing nulls or using regex replacement does not achieve the same result.

Answer

Use the 'Remove nulls' transform, then the 'Upper case' transform.

Answer

Use the 'Replace value' transform with a regex for null, then the 'Upper case' transform.

Answer

Use the 'Custom formula' transform with an IFNULL expression, then the 'Upper case' transform.

Question 12

A data engineer is building an AWS Glue ETL job that reads from an Amazon S3 bucket containing nested JSON files and must flatten the nested structures before writing to Amazon Redshift. The job uses the Glue DynamicFrame API. The engineer wants the transformation to run as a single pass without an intermediate shuffle. Which operation should the engineer use?

Accepted Answer

Relationalize. Relationalize is the Glue DynamicFrame transform specifically designed to unnest nested JSON into multiple flat frames in one traversal. It produces a root frame and one frame per nested array, each with a join key back to the parent. This matches the requirement to flatten nested structures before loading into Redshift without an intermediate shuffle.

Answer

ResolveChoice

Answer

ApplyMapping

Answer

DropNullFields

Question 13

A data engineer is using AWS Glue Studio to build a job that reads from an Amazon S3 bucket containing nested JSON files. The job must flatten the nested structure and write the output to Amazon Redshift. The engineer notices that the job fails with an error indicating that the Redshift table does not exist. The engineer has already created the Redshift cluster and database. What should the engineer do to resolve the error?

Accepted Answer

Create the target table in Amazon Redshift with the appropriate schema before running the job.. When writing to Amazon Redshift from AWS Glue, the target table must already exist. Glue does not automatically create Redshift tables. The engineer should create the table with the correct schema in Redshift, then rerun the job. The other transforms mentioned are for data manipulation within the job and do not address the missing table.

Answer

Add a 'ResolveChoice' transform before the Redshift writer to handle nested types.

Answer

Use the 'ApplyMapping' transform to map columns to Redshift data types.

Answer

Configure the Redshift writer to use the 'CREATE TABLE' option and specify the table name.

Question 14

A data engineer needs to run a transformation on streaming data using SQL-like queries without managing servers, and the output must be written to Amazon S3 in near real time. The source is an Amazon Kinesis Data Stream. Which AWS service is the MOST appropriate to perform the transformation?

Accepted Answer

Amazon Managed Service for Apache Flink with a SQL application.. Amazon Managed Service for Apache Flink provides a managed environment for running SQL-based stream processing applications against Kinesis Data Streams and can deliver results to Amazon S3 in near real time. Because AWS operates the underlying infrastructure, the engineer avoids cluster management while still getting continuous, low-latency transformation, which is precisely what the scenario requires.

Answer

AWS Glue with a Python shell job triggered hourly by Amazon EventBridge.

Answer

AWS Lambda with a function triggered by Kinesis records and writing directly to S3.

Answer

Amazon EMR running Apache Spark Structured Streaming on a persistent cluster.

Question 15

A data engineer is configuring an AWS Glue crawler against an Amazon S3 path that contains CSV files with inconsistent column counts across files. The crawler keeps creating multiple tables for the same data and the engineer wants a single table with a merged schema. Which crawler configuration should the engineer change?

Accepted Answer

Set the crawler's 'Create a single schema for each S3 path' option to true.. The 'Create a single schema for each S3 path' crawler setting merges compatible schemas found under the same S3 path into one table. For CSV files with inconsistent column counts, enabling this option produces a single table with the union of compatible columns, eliminating the multiple tables the crawler was creating.

Answer

Increase the crawler's 'Maximum number of tables' limit in the crawler configuration.

Answer

Enable the crawler's 'Update all new and existing partitions with metadata from the table' option.

Answer

Change the crawler's classifier to a custom classifier that forces all columns to string type.