DEA-C01 Data Ingestion and Transformation Practice Test 52 — 15 Questions

Question 1

A data engineer is using AWS Glue to transform data from an Amazon S3 bucket. The Glue job reads JSON files, applies complex transformations, and writes the output to another S3 bucket in Parquet format. The job runs daily and must complete within a 2-hour window. The engineer notices that the job is taking longer than expected and wants to optimize performance. The source data is partitioned by date, and the job uses a dynamic frame. Which optimization should the engineer implement to improve performance?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Use pushdown predicates to filter data at the source based on partition columns.. Pushdown predicates enable AWS Glue to filter data at the source based on partition columns, significantly reducing the amount of data read from S3. Since the data is partitioned by date, specifying a predicate for the desired date range ensures only relevant partitions are processed, leading to faster job completion and lower cost. This is a targeted optimization that addresses the performance bottleneck without unnecessary resource scaling.

Answer

Increase the number of DPUs allocated to the Glue job.

Answer

Enable job bookmarks to track processed data and avoid reprocessing.

Answer

Convert the dynamic frame to a Spark DataFrame and use Spark SQL for transformations.

Question 2

A data engineer is configuring an AWS Glue ETL job that reads from an Amazon S3 bucket and writes transformed records to an Amazon Redshift cluster. The job must load data in parallel and use an Amazon Redshift IAM role for authentication. Which connection option should the engineer configure in the Glue job to enable parallel loading and IAM-based authentication?

Accepted Answer

Create an AWS Glue connection of type Amazon Redshift, specify the cluster, database, and an IAM role with permissions to access both S3 and Redshift, and enable the 'Use Redshift COPY' option.. The correct configuration uses an AWS Glue connection of type Amazon Redshift with an IAM role and the Redshift COPY option enabled. This lets Glue stage data in Amazon S3 and issue a COPY command, which parallelizes loading across Redshift slices and authenticates via IAM instead of database credentials. Other connection types may provide connectivity but do not combine parallel COPY-based loading with IAM role authentication in the way Glue's native Redshift integration does.

Answer

Use a JDBC connection with the Redshift cluster endpoint, database name, and a database user credential stored in AWS Secrets Manager.

Answer

Configure an AWS Glue connection of type JDBC with the Redshift JDBC URL and set the 'redshift_tmp_dir' parameter to an S3 path, using a database password stored in AWS Glue.

Answer

Use an AWS Glue connection of type Amazon S3 and configure the Redshift cluster endpoint in the job's script using boto3.

Question 3

A data engineer runs an AWS Glue for Apache Spark job that writes partitioned Parquet output to Amazon S3. The engineer notices thousands of small output files in each partition, which slows downstream Amazon Athena queries. The job reads from a large S3 source and uses default partitioning. Which change should the engineer make to reduce the number of small files?

Accepted Answer

Use repartition or a controlled partition count before writing, or enable Glue's grouping of output files.. The number of output files equals the number of partitions written, so consolidating partitions with repartition or using Glue's output file grouping reduces small files without sacrificing parallelism. Coalescing to one partition kills scalability, bookmarks and DPUs do not affect file count, and CSV does not change partitioning while harming query efficiency.

Answer

Set the Glue job's output to a single partition using coalesce(1) before writing.

Answer

Enable Glue job bookmarks and increase the number of DPUs allocated to the job.

Answer

Convert the output to CSV so fewer files are created per partition.

Question 4

A data engineer needs to run a transformation on a large dataset stored in Amazon S3 using AWS Glue Studio. The transformation is a simple column rename and filter that can be expressed visually. The engineer wants to minimize development time and avoid writing PySpark code. Which approach should the engineer use?

Accepted Answer

Create a Glue Studio visual job, add a source node for the S3 bucket, add a Transform node for the rename and filter, and add a target node for the output location.. Glue Studio's visual job editor lets engineers build ETL pipelines by connecting source, transform, and target nodes without writing code. For a simple column rename and filter, this is the fastest path because Glue generates the Spark code and manages execution. The other options either require manual coding, use a different tool not integrated with Glue Studio jobs, or use query-based approaches that do not provide the same visual job experience.

Answer

Use AWS Glue DataBrew to create a recipe that renames columns and filters rows, then run a DataBrew job.

Answer

Use an AWS Glue crawler to infer the schema, then query the data with Amazon Athena and save the results to a new S3 location.

Answer

Develop an AWS Glue ETL script in PySpark using the DynamicFrame API, then run it as a job with the 'glueetl' job type.

Question 5

A data engineer is building an AWS Glue job that reads from a large Parquet dataset in Amazon S3 partitioned by year/month/day and writes aggregated results to Amazon Redshift. The job currently reads all partitions and takes several hours. The engineer wants the job to process only partitions from the last seven days and reduce runtime. Which change should the engineer make?

Accepted Answer

Pass a pushdown predicate in the from_catalog call that filters on the year, month, and day partition columns for the last seven days.. Partition pruning is the key optimization for partitioned datasets on S3. Supplying a pushdown predicate that references the year, month, and day partition columns lets the Glue reader skip non-matching partitions entirely, so only seven days of data are listed and read. More workers, format conversion, and bookmarks do not restrict the read window and therefore do not solve the runtime problem.

Answer

Convert the source data to ORC format to reduce the bytes read per partition.

Answer

Increase the number of DPUs allocated to the Glue job and enable auto scaling.

Answer

Enable job bookmarks so the job processes only new files since the last successful run.

Question 6

A company needs to ingest streaming data from thousands of IoT devices into Amazon S3 for long-term storage and analytics. The data arrives continuously at a rate of 5 MB per second and must be stored in a compressed format to reduce storage costs. The solution should be highly available and require minimal management. Which AWS service should the company use?

Accepted Answer

Amazon Kinesis Data Firehose with a delivery stream to Amazon S3, enabling compression.. Amazon Kinesis Data Firehose is designed for streaming data ingestion to destinations like S3. It automatically handles scaling, buffering, compression, and encryption. With compression enabled, data is stored in GZIP format, reducing storage costs. It requires no infrastructure management, making it the simplest and most cost-effective solution for continuous IoT data ingestion to S3.

Answer

Amazon Kinesis Data Streams with an AWS Lambda consumer that writes to Amazon S3.

Answer

Amazon Managed Streaming for Apache Kafka (Amazon MSK) with a Kafka Connect S3 sink connector.

Answer

AWS Glue streaming ETL job that reads from Amazon Kinesis Data Streams and writes to Amazon S3.

Question 7

A data engineer is using AWS Glue to process a large dataset where a small number of partitions contain disproportionately more rows than others, causing some executors to run much longer than others and the job to take hours. The engineer wants to redistribute the data across partitions before a join operation to improve performance. Which technique should the engineer apply?

Accepted Answer

Call repartition on the DataFrame using a join key column before the join, which triggers a full shuffle and distributes rows evenly across partitions.. Repartitioning on the join key before the join causes a shuffle that spreads rows evenly across partitions, which is the direct fix for skewed partitions. Increasing DPUs does not help because the largest partition remains a single bottleneck, coalesce can worsen skew by merging without redistribution, and job bookmarks only reduce data across runs. The key is to redistribute data within the run before the expensive join.

Answer

Increase the number of AWS Glue DPUs allocated to the job so that more executors are available to process the skewed partitions.

Answer

Use coalesce to reduce the number of partitions, which merges partitions without a full shuffle and balances the data.

Answer

Enable the AWS Glue job bookmarks feature so that only new data is processed on subsequent runs, reducing the data volume.

Question 8

A data engineer must transform data in an AWS Glue ETL job. The transform requires calling an external REST API for each record to enrich the data. The Glue job runs on AWS Glue 4.0 with Python. The engineer wants to minimize the number of API calls and improve performance. Which approach should the engineer take?

Accepted Answer

Convert the DynamicFrame to a Spark DataFrame, use foreachPartition to batch records and call the API once per batch.. To minimize API calls in an AWS Glue job, the engineer should batch records and call the API once per batch. Converting the DynamicFrame to a Spark DataFrame and using foreachPartition allows processing each partition, accumulating records into batches, and making a single API call per batch. This reduces the number of calls and improves performance. Other options either call the API per record or do not support API calls.

Answer

Use Glue DynamicFrame's map method to call the API for each record.

Answer

Use AWS Glue's built-in transform 'ApplyMapping' to call the API.

Answer

Configure the Glue job to use a larger number of DPUs to parallelize API calls.

Question 9

A data engineer needs to ingest data from an Amazon S3 bucket into an Amazon Redshift cluster. The data is stored as CSV files and is updated daily. The engineer wants to load only new data each day without duplicating existing records. Which AWS service or feature should the engineer use to automate this process?

Accepted Answer

AWS Glue with job bookmarks enabled to track processed files.. AWS Glue job bookmarks are designed to track processed data across job runs, enabling incremental processing. For daily loads from S3 to Redshift, a Glue ETL job with bookmarks ensures only new files are processed each day, preventing duplication. This automates the process and integrates with Redshift via the COPY command. Other options do not provide automatic incremental load tracking for S3 files.

Answer

Amazon Redshift Spectrum to query S3 data directly and insert new records.

Answer

Amazon Kinesis Data Firehose to stream S3 data to Redshift.

Answer

AWS Database Migration Service (DMS) with ongoing replication from S3 to Redshift.

Question 10

A data engineer uses AWS Glue DataBrew to clean a dataset stored in Amazon S3. The recipe includes a 'Filter' step that removes rows where the 'status' column equals 'INVALID'. After running the recipe, the engineer notices that the output still contains rows with status 'INVALID'. The recipe was published and the job ran successfully. What is the most likely cause?

Accepted Answer

The filter condition used a case-sensitive match and the actual values are 'invalid' in lowercase, so the condition did not match any rows.. DataBrew filter conditions are case-sensitive, so a condition matching 'INVALID' will not remove rows containing 'invalid'. The engineer should either standardize the case of the status column or adjust the filter condition to match the actual values. This explains why the job succeeded but the unwanted rows remained.

Answer

The filter step was configured as a 'Remove' transformation instead of a 'Filter' transformation, so it only flagged rows without deleting them.

Answer

The recipe was applied to a sample of the data during job execution, so only a subset was filtered.

Answer

The DataBrew job was run in profile mode instead of recipe mode, so transformations were not applied.

Question 11

A data engineer is building a pipeline that ingests records from an Amazon Kinesis data stream and writes them to Amazon S3 in Parquet format. The engineer wants to use AWS Glue to perform the transformation and needs the pipeline to handle records that arrive out of order and to deduplicate based on a record ID. Which combination of features should the engineer use?

Accepted Answer

Use AWS Glue streaming ETL with a Kinesis source, apply a windowed deduplication using the record ID and event timestamp, and write the results to S3 in Parquet.. Glue streaming ETL reading from Kinesis, combined with windowed deduplication on record ID and event timestamp, handles both out-of-order arrival and duplicates. Writing Parquet to S3 meets the format goal. Batch ETL from S3 adds latency and misses cross-batch duplicates, Firehose lacks built-in deduplication and does not use Glue for transformation, and ResolveChoice is not a deduplication tool.

Answer

Use Amazon Kinesis Data Firehose with a Lambda function for record transformation, and rely on Firehose's built-in deduplication to handle duplicates.

Answer

Use AWS Glue batch ETL with an S3 source, schedule the job every minute, and use the DropDuplicates transform on the record ID.

Answer

Use AWS Glue streaming ETL with a Kinesis source, set the job to process records in order, and use the ResolveChoice transform to merge duplicate records.

Question 12

A company ingests application logs into Amazon S3 through Amazon Kinesis Data Firehose. The logs arrive as newline-delimited JSON, and analysts query them with Amazon Athena. Query performance is poor because the JSON files are small and uncompressed. The engineer must improve Athena query performance while keeping the raw JSON available for a downstream legacy system. Which change should the engineer make?

Accepted Answer

Add a Firehose record format conversion to Apache Parquet and keep the raw JSON in a separate S3 prefix using a second delivery stream.. Athena performance improves when it scans less data, and converting the stream to Parquet with a record format conversion makes the stored objects columnar and compressed. A second delivery stream that leaves the raw JSON in place satisfies the legacy consumer, so the engineer gets both performance and compatibility.

Answer

Enable S3 Transfer Acceleration on the target bucket to speed up Athena scans.

Answer

Configure the Athena table with a SerDe for JSON and add a WHERE clause on the timestamp column in every query.

Answer

Increase the Firehose buffer size and interval so larger objects are delivered to S3.

Question 13

A data engineer is using AWS Glue to process a large dataset stored in Amazon S3. The dataset is partitioned by year, month, and day. The engineer notices that the Glue job is taking a long time and consuming many DPUs. The job reads all partitions, filters the data, and writes the result to another S3 location. The engineer wants to optimize the job to process only the required partitions and reduce cost. Which action should the engineer take?

Accepted Answer

Use predicate pushdown in the Glue job by specifying a push_down_predicate when reading from the AWS Glue Data Catalog.. Predicate pushdown in AWS Glue allows the job to read only the partitions that match a specified condition, reducing data scanned and DPU usage. By using push_down_predicate when reading from the Data Catalog, the job filters at the source. Other options either do not reduce data read (bookmarks, more DPUs) or are not sufficient alone (Parquet conversion).

Answer

Convert the dataset to Parquet format before processing.

Answer

Increase the number of DPUs to speed up the job.

Answer

Enable AWS Glue job bookmarks to track processed partitions.

Question 14

A data engineer needs to run a PySpark transformation on a 2 TB dataset stored in Amazon S3 and write the output back to S3 in Parquet. The team wants to use AWS Glue but does not want to manage clusters or tune Spark configuration manually. They also want to pay only for the time the job runs. Which AWS Glue component should the engineer use?

Accepted Answer

AWS Glue for Apache Spark job. AWS Glue for Apache Spark provides a serverless, distributed Spark runtime. The engineer submits a PySpark script, Glue handles cluster provisioning and Spark tuning, and billing is based on DPU-hours consumed during the run. This fits the need for custom PySpark on a 2 TB S3 dataset with no cluster management and pay-per-run cost, unlike single-node shell jobs or self-managed EMR clusters.

Answer

AWS Glue DataBrew recipe job

Answer

Amazon EMR on EC2 cluster with a bootstrap action

Answer

AWS Glue Python shell job

Question 15

A data engineer needs to transform JSON data from Amazon S3 into Parquet using AWS Glue. The JSON is nested and contains arrays. The engineer wants to flatten the nested structure and write the result to S3 partitioned by a 'region' field. Which combination of Glue transforms should the engineer use?

Accepted Answer

Use the 'Relationalize' transform to flatten the nested JSON, then write the resulting DynamicFrame with partition keys set to 'region'.. Relationalize is designed to flatten nested JSON and arrays into relational tables, producing a collection of DynamicFrames. The engineer can then select the relevant table, set the 'region' field as a partition key, and write Parquet to S3. This is the intended Glue transform for nested-to-relational conversion.

Answer

Use the 'ResolveChoice' transform to flatten nested data, then write with partition keys.

Answer

Use the 'DropNullFields' transform to remove nulls, then write with partition keys.

Answer

Use the 'Unbox' transform to extract nested fields, then use 'ApplyMapping' to rename columns, and write with partition keys.