DEA-C01 Practice Test 41 — 15 Questions

Question 1

A data engineer is using AWS Glue to run an ETL job that reads from an Amazon S3 bucket and writes to another S3 bucket. The job processes data in CSV format and the engineer wants to ensure the output is partitioned by year, month, and day based on a timestamp column in the data. The engineer needs to optimize the job for performance and cost. Which approach should the engineer take?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Use the AWS Glue DynamicFrame partitionColumns parameter to specify year, month, and day as partition keys.. The partitionColumns parameter in AWS Glue's write operation is the correct way to partition output data by specified columns. It creates a hierarchical directory structure in S3, enabling efficient querying with services like Athena and Redshift Spectrum. Other options either misuse transformations or misunderstand the purpose of bookmarks and single-file output.

Answer

Write the output to a single CSV file and then use Amazon Athena to create a partitioned table.

Answer

Use the AWS Glue job's bookmark feature to automatically partition the output by timestamp.

Answer

Use the AWS Glue ResolveChoice transformation to split the timestamp column into separate year, month, and day columns, then write to S3 with partitioning.

Question 2

A data engineer runs an AWS Glue for Apache Spark job that writes partitioned Parquet output to Amazon S3. The engineer notices thousands of small output files in each partition, which slows downstream Amazon Athena queries. The job reads from a large S3 source and uses default partitioning. Which change should the engineer make to reduce the number of small files?

Accepted Answer

Use repartition or a controlled partition count before writing, or enable Glue's grouping of output files.. The number of output files equals the number of partitions written, so consolidating partitions with repartition or using Glue's output file grouping reduces small files without sacrificing parallelism. Coalescing to one partition kills scalability, bookmarks and DPUs do not affect file count, and CSV does not change partitioning while harming query efficiency.

Answer

Set the Glue job's output to a single partition using coalesce(1) before writing.

Answer

Enable Glue job bookmarks and increase the number of DPUs allocated to the job.

Answer

Convert the output to CSV so fewer files are created per partition.

Question 3

A data engineer is building an AWS Glue job that reads from a large Parquet dataset in Amazon S3 partitioned by year/month/day and writes aggregated results to Amazon Redshift. The job currently reads all partitions and takes several hours. The engineer wants the job to process only partitions from the last seven days and reduce runtime. Which change should the engineer make?

Accepted Answer

Pass a pushdown predicate in the from_catalog call that filters on the year, month, and day partition columns for the last seven days.. Partition pruning is the key optimization for partitioned datasets on S3. Supplying a pushdown predicate that references the year, month, and day partition columns lets the Glue reader skip non-matching partitions entirely, so only seven days of data are listed and read. More workers, format conversion, and bookmarks do not restrict the read window and therefore do not solve the runtime problem.

Answer

Convert the source data to ORC format to reduce the bytes read per partition.

Answer

Increase the number of DPUs allocated to the Glue job and enable auto scaling.

Answer

Enable job bookmarks so the job processes only new files since the last successful run.

Question 4

A data engineer needs to run a transformation on a large dataset stored in Amazon S3 using AWS Glue Studio. The transformation is a simple column rename and filter that can be expressed visually. The engineer wants to minimize development time and avoid writing PySpark code. Which approach should the engineer use?

Accepted Answer

Create a Glue Studio visual job, add a source node for the S3 bucket, add a Transform node for the rename and filter, and add a target node for the output location.. Glue Studio's visual job editor lets engineers build ETL pipelines by connecting source, transform, and target nodes without writing code. For a simple column rename and filter, this is the fastest path because Glue generates the Spark code and manages execution. The other options either require manual coding, use a different tool not integrated with Glue Studio jobs, or use query-based approaches that do not provide the same visual job experience.

Answer

Use AWS Glue DataBrew to create a recipe that renames columns and filters rows, then run a DataBrew job.

Answer

Use an AWS Glue crawler to infer the schema, then query the data with Amazon Athena and save the results to a new S3 location.

Answer

Develop an AWS Glue ETL script in PySpark using the DynamicFrame API, then run it as a job with the 'glueetl' job type.

Question 5

A data engineer is using AWS Glue to process a large dataset where a small number of partitions contain disproportionately more rows than others, causing some executors to run much longer than others and the job to take hours. The engineer wants to redistribute the data across partitions before a join operation to improve performance. Which technique should the engineer apply?

Accepted Answer

Call repartition on the DataFrame using a join key column before the join, which triggers a full shuffle and distributes rows evenly across partitions.. Repartitioning on the join key before the join causes a shuffle that spreads rows evenly across partitions, which is the direct fix for skewed partitions. Increasing DPUs does not help because the largest partition remains a single bottleneck, coalesce can worsen skew by merging without redistribution, and job bookmarks only reduce data across runs. The key is to redistribute data within the run before the expensive join.

Answer

Increase the number of AWS Glue DPUs allocated to the job so that more executors are available to process the skewed partitions.

Answer

Use coalesce to reduce the number of partitions, which merges partitions without a full shuffle and balances the data.

Answer

Enable the AWS Glue job bookmarks feature so that only new data is processed on subsequent runs, reducing the data volume.

Question 6

A data engineer must transform data in an AWS Glue ETL job. The transform requires calling an external REST API for each record to enrich the data. The Glue job runs on AWS Glue 4.0 with Python. The engineer wants to minimize the number of API calls and improve performance. Which approach should the engineer take?

Accepted Answer

Convert the DynamicFrame to a Spark DataFrame, use foreachPartition to batch records and call the API once per batch.. To minimize API calls in an AWS Glue job, the engineer should batch records and call the API once per batch. Converting the DynamicFrame to a Spark DataFrame and using foreachPartition allows processing each partition, accumulating records into batches, and making a single API call per batch. This reduces the number of calls and improves performance. Other options either call the API per record or do not support API calls.

Answer

Use Glue DynamicFrame's map method to call the API for each record.

Answer

Use AWS Glue's built-in transform 'ApplyMapping' to call the API.

Answer

Configure the Glue job to use a larger number of DPUs to parallelize API calls.

Question 7

A company needs to transfer 20 TB of historical data from an on-premises Hadoop cluster to Amazon S3. The network bandwidth is limited and the transfer must complete within one week. Which service should the company use?

Accepted Answer

AWS Snowball Edge. AWS Snowball Edge is a petabyte-scale data transport solution that uses physical devices to transfer large amounts of data into and out of AWS, bypassing network bandwidth limitations. For 20 TB over a limited network within one week, Snowball Edge is the most practical and cost-effective choice. It supports up to 80 TB usable capacity per device and provides encryption and tamper-resistant security.

Answer

Amazon S3 Transfer Acceleration

Answer

AWS Direct Connect with DataSync

Answer

AWS DataSync over a VPN connection

Question 8

A data engineer uses AWS Glue DataBrew to clean a dataset stored in Amazon S3. The recipe includes a 'Filter' step that removes rows where the 'status' column equals 'INVALID'. After running the recipe, the engineer notices that the output still contains rows with status 'INVALID'. The recipe was published and the job ran successfully. What is the most likely cause?

Accepted Answer

The filter condition used a case-sensitive match and the actual values are 'invalid' in lowercase, so the condition did not match any rows.. DataBrew filter conditions are case-sensitive, so a condition matching 'INVALID' will not remove rows containing 'invalid'. The engineer should either standardize the case of the status column or adjust the filter condition to match the actual values. This explains why the job succeeded but the unwanted rows remained.

Answer

The filter step was configured as a 'Remove' transformation instead of a 'Filter' transformation, so it only flagged rows without deleting them.

Answer

The recipe was applied to a sample of the data during job execution, so only a subset was filtered.

Answer

The DataBrew job was run in profile mode instead of recipe mode, so transformations were not applied.

Question 9

A data engineer is building an AWS Glue ETL job that reads a large JDBC table from Amazon RDS for PostgreSQL. The job must read the table in parallel to reduce runtime, but the table has no numeric primary key or monotonically increasing column. Which AWS Glue connection property should the engineer configure to enable parallel reads?

Accepted Answer

hashpartitions and hashfield. For JDBC sources without a numeric or monotonic column, AWS Glue provides hashpartitions and hashfield to enable parallel reads. hashpartitions sets the number of partitions, and hashfield names a column used for hashing rows across those partitions. This avoids the need for a numeric partitionColumn and is the documented approach for this exact situation.

Answer

partitionColumn with lowerBound and upperBound

Answer

hashfield

Answer

enableParallelRead with numPartitions

Question 10

A data engineer is building a pipeline that ingests records from an Amazon Kinesis data stream and writes them to Amazon S3 in Parquet format. The engineer wants to use AWS Glue to perform the transformation and needs the pipeline to handle records that arrive out of order and to deduplicate based on a record ID. Which combination of features should the engineer use?

Accepted Answer

Use AWS Glue streaming ETL with a Kinesis source, apply a windowed deduplication using the record ID and event timestamp, and write the results to S3 in Parquet.. Glue streaming ETL reading from Kinesis, combined with windowed deduplication on record ID and event timestamp, handles both out-of-order arrival and duplicates. Writing Parquet to S3 meets the format goal. Batch ETL from S3 adds latency and misses cross-batch duplicates, Firehose lacks built-in deduplication and does not use Glue for transformation, and ResolveChoice is not a deduplication tool.

Answer

Use Amazon Kinesis Data Firehose with a Lambda function for record transformation, and rely on Firehose's built-in deduplication to handle duplicates.

Answer

Use AWS Glue batch ETL with an S3 source, schedule the job every minute, and use the DropDuplicates transform on the record ID.

Answer

Use AWS Glue streaming ETL with a Kinesis source, set the job to process records in order, and use the ResolveChoice transform to merge duplicate records.

Question 11

A company uses Amazon Redshift for analytics. The data engineer notices that queries are slow due to many small inserts. Which technique would improve write performance?

Accepted Answer

Use the COPY command to load data from Amazon S3.. The COPY command is the recommended way to load data into Amazon Redshift because it performs bulk inserts in parallel across all nodes, leveraging the cluster's distributed architecture. Small individual INSERT statements cause high overhead due to transaction logging and commit processing, leading to slow write performance. By loading data from Amazon S3 using COPY, you bypass these per-row overheads and achieve optimal throughput.

Answer

Define DISTKEY and SORTKEY on the table.

Answer

Increase the number of nodes in the cluster.

Answer

Configure workload management (WLM) queues.

Question 12

A company uses Amazon Redshift for analytics. They notice that some queries are slow due to data redistribution. The data engineer wants to minimize data movement across nodes. Which table design strategy should be used? (Choose TWO.)

Accepted Answer

Choose a distribution key that matches the join key for large tables.. Option C is correct because choosing a distribution key that matches the join key on large tables colocates matching rows on the same compute node slice, so joins between those tables can be performed locally without broadcasting or redistributing data across nodes. Option E is correct because using distribution style ALL replicates small dimension tables to every node, eliminating the need to redistribute the dimension during joins with large fact tables and thereby minimizing cross-node data movement. Option A is not ideal here because AUTO lets Redshift decide and may still choose EVEN or KEY distribution that results in redistribution for some workloads, rather than guaranteeing join-key alignment. Option B addresses sort keys, which optimize range-restricted scans and merge joins but do not control how rows are distributed across nodes, so they do not directly reduce redistribution. Option D is incorrect because EVEN distribution spreads rows round-robin regardless of join keys, which typically forces data redistribution during joins and can worsen the problem.

Answer

Set the distribution style to AUTO for all tables.

Answer

Define compound sort keys on frequently filtered columns.

Answer

Use EVEN distribution for all tables.

Question 13

A data engineer must load a 500 MB CSV file from Amazon S3 into an existing Amazon Redshift table once per day. The file has a header row and uses a pipe delimiter. The engineer wants the fastest load and the least operational overhead. Which approach should the engineer use?

Accepted Answer

Use the Redshift COPY command with the IAM role, CSV, HEADER, and DELIMITER options specified.. The Redshift COPY command is purpose-built for bulk loading from S3 and parallelizes the read, making it the fastest and simplest choice for a single delimited file. The CSV, HEADER, and DELIMITER parameters align the load with the file format, so no staging or transformation layer is needed.

Answer

Use the Redshift Data API to run individual INSERT statements for each row in the CSV file.

Answer

Use an AWS Glue ETL job to read the CSV from S3 and write to Redshift through a JDBC connection.

Answer

Use AWS Database Migration Service with a change data capture task to replicate the S3 file into Redshift.

Question 14

A data engineer is using AWS Glue to process a large dataset stored in Amazon S3 in Parquet format. The Glue job performs a join between two tables and writes the result back to S3. The engineer notices that the job is running slowly and consuming excessive DPU hours. The job has 10 workers of type G.1X. Which action should the engineer take to improve performance and reduce cost?

Accepted Answer

Partition the Parquet data by the join key and use broadcast join if one table is small.. The performance issue is likely due to inefficient join operations causing large data shuffles. Partitioning the Parquet data by the join key aligns data with the join operation, reducing shuffling. If one table is small enough to fit in memory, a broadcast join eliminates the shuffle entirely. These optimizations reduce runtime and DPU consumption. Other options either do not address the join bottleneck or could increase cost without solving the problem.

Answer

Use the 'ApplyMapping' transformation to rename columns before the join.

Answer

Enable AWS Glue job bookmarks to avoid reprocessing old data.

Answer

Increase the number of workers to 20 and keep the worker type as G.1X.

Question 15

A data engineer needs to run a transformation in AWS Glue where each record must be processed independently and the output schema is known ahead of time. The transformation should operate on a DynamicFrame and return a DynamicFrame. Which Glue transform is designed for this row-by-row operation?

Accepted Answer

Map. The Map transform is purpose-built for applying a function to each record and returning a DynamicFrame, which matches the need for independent row processing with a known output schema. Relationalize, ResolveChoice, and DropNullFields are schema or structure utilities, not general row-level mapping transforms.

Answer

DropNullFields

Answer

Relationalize

Answer

ResolveChoice