DEA-C01 Data Ingestion and Transformation Practice Test 53 — 15 Questions

Question 1

A data engineer needs to ingest streaming data from an Amazon Kinesis Data Stream into an Amazon S3 bucket. The data must be delivered in near real-time and stored in Parquet format for efficient querying. The engineer wants to minimize custom code. Which solution should the engineer use?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Use Amazon Kinesis Data Firehose to deliver the stream to S3, and enable record format conversion to Parquet using an AWS Glue table.. Kinesis Data Firehose provides a fully managed way to deliver streaming data from Kinesis Data Streams to S3. It supports automatic conversion to Parquet using a schema from the AWS Glue Data Catalog, requiring no custom code. This meets the near real-time and Parquet requirements while minimizing development effort, unlike solutions that require writing Lambda functions, Flink applications, or Glue scripts.

Answer

Use Amazon Kinesis Data Analytics to process the stream and write Parquet to S3 using a Flink application.

Answer

Use AWS Lambda to read from the Kinesis Data Stream and write Parquet files to S3 using the AWS SDK for pandas (awswrangler).

Answer

Use AWS Glue streaming ETL job to read from the Kinesis Data Stream and write Parquet to S3.

Question 2

A data engineer is building an AWS Glue ETL job that reads a large JDBC table from Amazon RDS for PostgreSQL. The job must read the table in parallel to reduce runtime, but the table has no numeric primary key or monotonically increasing column. Which AWS Glue connection property should the engineer configure to enable parallel reads?

Accepted Answer

hashpartitions and hashfield. For JDBC sources without a numeric or monotonic column, AWS Glue provides hashpartitions and hashfield to enable parallel reads. hashpartitions sets the number of partitions, and hashfield names a column used for hashing rows across those partitions. This avoids the need for a numeric partitionColumn and is the documented approach for this exact situation.

Answer

partitionColumn with lowerBound and upperBound

Answer

hashfield

Answer

enableParallelRead with numPartitions

Question 3

A data engineer is designing an AWS Glue ETL job that reads data from an Amazon S3 bucket containing many small JSON files and writes the output to Amazon S3 in Parquet format. The engineer wants to improve read performance and reduce the number of output files. Which two actions should the engineer take? (Choose two.)

Accepted Answer

Call repartition or coalesce on the DynamicFrame before writing to control the number of output files.. Grouping small files during the read reduces file-open overhead and improves read performance, while repartitioning or coalescing before writing controls the number of output files. Job bookmarks only help across runs, adding DPUs does not fix per-file overhead or output count, and pre-converting files with Lambda does not address the small-file or output-file problems within the Glue job.

Answer

Increase the number of DPUs to allow more executors to read the small files in parallel.

Answer

Enable job bookmarks to skip previously processed files and reduce the number of files read.

Answer

Convert the source JSON files to Parquet before the Glue job runs using an AWS Lambda function triggered on each S3 object.

Question 4

A data engineer is building an AWS Glue ETL job that must read from an Amazon S3 bucket in the same account and write to an Amazon Redshift cluster in a private VPC. The job must not traverse the public internet and must use least-privilege credentials. (Choose two.)

Accepted Answer

Grant the Glue job an IAM role with an S3 bucket policy that allows access only to the required prefixes.. Private connectivity to Redshift requires the Glue job to run inside the VPC through a VPC connection with correct subnet routing and security group rules. Least privilege is then enforced by a scoped IAM role and bucket policy, so the two together satisfy both the no-public-internet and least-privilege requirements.

Answer

Store the Redshift user password in the job script as a plaintext variable so the connection can authenticate.

Answer

Set the Glue job's connection type to JDBC and rely on the default Glue service role for all S3 and Redshift permissions.

Answer

Create an internet gateway and a NAT gateway in the VPC so the Glue job can reach Redshift endpoints.

Question 5

A data engineer is designing an ingestion pipeline that uses Amazon Kinesis Data Firehose to deliver streaming records into an Amazon S3 bucket. The records arrive as JSON, and downstream consumers require Parquet with a stable schema. The engineer must configure the Firehose delivery stream so records are converted to Parquet before landing in S3. (Choose two.)

Accepted Answer

Enable record format conversion on the Firehose delivery stream and select Apache Parquet as the output format.. Firehose record format conversion converts JSON to Parquet in flight, and it requires a schema reference. The supported schema source is an AWS Glue table in the Data Catalog that describes the incoming JSON structure. Enabling conversion with Apache Parquet output and pointing to a matching Glue table together produce Parquet files in S3 for downstream analytics.

Answer

Enable Amazon S3 Object Lambda access points on the destination bucket to convert objects to Parquet on read.

Answer

Configure an AWS Lambda function as the delivery stream's transformation to rewrite each record into Parquet bytes.

Answer

Set the S3 destination prefix to include a .parquet file extension so Firehose writes columnar files.

Question 6

A data engineer runs an AWS Glue Studio job that reads JSON from Amazon S3 and writes to a partitioned Parquet table. The job currently runs for six hours. Profiling shows that a small number of partitions contain millions of rows while most contain a few hundred. Which change will MOST improve runtime?

Accepted Answer

Apply a salting technique to the skewed keys before the write, or repartition on a composite key.. Data skew, not compute capacity, is causing the long runtime because a few partitions hold most of the rows. Salting hot keys or repartitioning on a composite key spreads those rows across many tasks so no single task becomes a straggler. Increasing DPUs, changing worker type, or enabling bookmarks does not change how rows are distributed within a run.

Answer

Increase the number of DPUs allocated to the Glue job from 10 to 100.

Answer

Enable job bookmarks and re-run the job to skip already-processed files.

Answer

Switch the job's worker type from G.1X to G.2X to get more memory per executor.

Question 7

A data engineer is using AWS Glue to run a job that reads from an Amazon S3 bucket and writes to another S3 bucket. The job must be scheduled to run every day at 2:00 AM UTC. Which AWS Glue feature should the engineer use to set up this schedule?

Accepted Answer

AWS Glue Triggers with a schedule type of 'Scheduled' and a cron expression.. AWS Glue Triggers of type 'Scheduled' accept cron expressions and run jobs on a defined schedule in UTC. Creating a scheduled trigger with the appropriate cron expression is the direct, native way to run a Glue job daily at 2:00 AM UTC without external orchestration.

Answer

AWS Glue job bookmarks with a time-based configuration.

Answer

AWS Glue DataBrew schedules with a cron expression.

Answer

AWS Glue Workflows with an on-demand trigger started by an Amazon EventBridge rule.

Question 8

A data engineer is using AWS Glue to transform data from Amazon S3 and load it into Amazon Redshift. The engineer needs to ensure that the job handles data quality issues such as duplicate records and missing values before loading. The job must also minimize the amount of data shuffled across the network. Which two actions should the engineer take? (Choose two.)

Accepted Answer

Use the FillMissingValues transform to replace missing values with a default.. DropDuplicates removes duplicate records, and FillMissingValues replaces missing values with defaults, directly addressing the data quality requirements. Both are built-in AWS Glue transforms that can be applied without custom code. The other options do not handle duplicates or missing values, and repartitioning to a single partition would increase network shuffle, contrary to the requirement.

Answer

Repartition the DynamicFrame to a single partition before writing to Redshift.

Answer

Use the ResolveChoice transform to handle data type conflicts.

Answer

Enable job bookmarks to track processed data and avoid reprocessing.

Question 9

A data engineer is using AWS Glue to transform data from Amazon S3 and load it into Amazon Redshift. The job must run incrementally and process only new data since the last run. The source data is partitioned by date in S3, and new partitions are added daily. Which AWS Glue feature should the engineer enable to track previously processed data?

Accepted Answer

Job bookmarks. Job bookmarks are the AWS Glue feature designed to track previously processed data. When enabled, Glue maintains state for each job, so subsequent runs process only new data, such as new S3 partitions or files. This is the standard way to implement incremental ETL in Glue without reprocessing historical data.

Answer

Data Catalog partition index

Answer

Glue DataBrew profile

Answer

Glue FindMatches

Question 10

A data engineer runs an AWS Glue job that reads a large partitioned Parquet dataset from Amazon S3 and writes aggregated results to another S3 prefix. The job runs daily and currently reprocesses the entire dataset each time, which is becoming expensive. The engineer wants subsequent runs to process only data added since the last successful run, based on the job's state. Which feature should the engineer enable?

Accepted Answer

AWS Glue job bookmarks. AWS Glue job bookmarks maintain state across runs, recording which S3 objects, partitions, or JDBC rows were processed. When enabled, a subsequent run reads only new or modified data, avoiding full reprocessing. This is the built-in mechanism for incremental Glue jobs, unlike partition indexes, S3 Inventory, or retry settings, which do not track processed data across runs.

Answer

AWS Glue job retry with a smaller max retry count

Answer

AWS Glue Data Catalog partition indexes

Answer

Amazon S3 Inventory reports on the source prefix

Question 11

A data engineer is using AWS Glue to transform data from Amazon S3 and load it into Amazon Redshift. The engineer needs to improve the performance of the Glue job, which currently takes several hours to complete. The job reads large Parquet files, performs joins, and writes to Redshift. Which two actions should the engineer take to improve performance? (Choose two.)

Accepted Answer

Partition the source Parquet data in Amazon S3 by commonly filtered columns and use partition pruning in the Glue job.. Increasing DPUs adds compute resources to parallelize the job, and partitioning the source data enables partition pruning to reduce the amount of data read. Both actions directly address performance bottlenecks in a large Glue job. Other options either do not affect single-run performance or would degrade it.

Answer

Enable job bookmarks to avoid reprocessing previously processed data.

Answer

Convert the Parquet files to CSV to reduce storage size and speed up reading.

Answer

Use the 'Relationalize' transform to flatten the Parquet data before joining.

Question 12

A company uses AWS Glue DataBrew to clean and normalize data from an Amazon S3 bucket before loading it into Amazon Redshift. The data contains PII such as social security numbers. A compliance policy requires that PII be masked in the DataBrew output. Which DataBrew transformation should the engineer use to replace the last four digits of each SSN with asterisks?

Accepted Answer

Use the 'Mask value' transformation with a custom pattern to mask the last four characters.. AWS Glue DataBrew provides a 'Mask value' transformation that can apply custom masking patterns to specific characters. By configuring a pattern to mask the last four digits of the SSN with asterisks, the engineer preserves the first five digits for analytics while complying with the PII masking policy. Other transformations either over-mask, encrypt, or remove the data, failing the specific requirement.

Answer

Use the 'Encrypt' transformation with AWS KMS to encrypt the SSN column.

Answer

Use the 'Remove columns' transformation to drop the SSN column entirely.

Answer

Use the 'Replace value or pattern' transformation to replace the entire SSN with a fixed string.

Question 13

A data engineer must load a 500 MB CSV file from Amazon S3 into an existing Amazon Redshift table once per day. The file has a header row and uses a pipe delimiter. The engineer wants the fastest load and the least operational overhead. Which approach should the engineer use?

Accepted Answer

Use the Redshift COPY command with the IAM role, CSV, HEADER, and DELIMITER options specified.. The Redshift COPY command is purpose-built for bulk loading from S3 and parallelizes the read, making it the fastest and simplest choice for a single delimited file. The CSV, HEADER, and DELIMITER parameters align the load with the file format, so no staging or transformation layer is needed.

Answer

Use the Redshift Data API to run individual INSERT statements for each row in the CSV file.

Answer

Use an AWS Glue ETL job to read the CSV from S3 and write to Redshift through a JDBC connection.

Answer

Use AWS Database Migration Service with a change data capture task to replicate the S3 file into Redshift.

Question 14

A data engineer is using AWS Glue to process a large dataset stored in Amazon S3 in Parquet format. The Glue job performs a join between two tables and writes the result back to S3. The engineer notices that the job is running slowly and consuming excessive DPU hours. The job has 10 workers of type G.1X. Which action should the engineer take to improve performance and reduce cost?

Accepted Answer

Partition the Parquet data by the join key and use broadcast join if one table is small.. The performance issue is likely due to inefficient join operations causing large data shuffles. Partitioning the Parquet data by the join key aligns data with the join operation, reducing shuffling. If one table is small enough to fit in memory, a broadcast join eliminates the shuffle entirely. These optimizations reduce runtime and DPU consumption. Other options either do not address the join bottleneck or could increase cost without solving the problem.

Answer

Use the 'ApplyMapping' transformation to rename columns before the join.

Answer

Enable AWS Glue job bookmarks to avoid reprocessing old data.

Answer

Increase the number of workers to 20 and keep the worker type as G.1X.

Question 15

A data engineer is using AWS Glue Studio to create an ETL job that reads from an Amazon S3 bucket and writes to another S3 bucket. The source data is in JSON format and contains nested structures. The engineer needs to flatten the nested structures and write the output in Parquet format. The job must be efficient and scalable. Which transformation should the engineer use in Glue Studio to flatten the nested data?

Accepted Answer

Use the 'Relationalize' transform.. The Relationalize transform in AWS Glue is specifically designed to flatten nested JSON data. It unnests arrays and structs into a relational schema, producing multiple tables. This is the correct transform for the scenario. ApplyMapping, Filter, and Join do not flatten nested structures. Relationalize is efficient and scalable for this purpose.

Answer

Use the 'ApplyMapping' transform.

Answer

Use the 'Filter' transform.

Answer

Use the 'Join' transform.