DEA-C01 Data Ingestion and Transformation Practice Test 48 — 15 Questions

Question 1

A data engineer is building an AWS Glue ETL job that reads from an Amazon DynamoDB table and writes to Amazon S3. The DynamoDB table has a large number of items, and the engineer needs to ensure the job reads the data efficiently without consuming too much provisioned throughput. Which method should the engineer use to read from DynamoDB?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Use the DynamoDB export to S3 feature to export the table to S3, then read from S3 in the Glue job.. DynamoDB export to S3 allows you to export a full table snapshot to S3 without consuming read capacity, making it ideal for large-scale ETL reads. The Glue job can then read the exported data from S3 in parallel, achieving high throughput without impacting the DynamoDB table's performance.

Answer

Use Amazon Kinesis Data Streams to stream changes from DynamoDB to S3, then read from S3.

Answer

Use AWS Glue's native DynamoDB connector with the read throughput parameter set to a high value.

Answer

Use AWS Glue's DynamoDB connector with the 'dynamodb.scan' option to scan the table.

Question 2

A data engineer needs to ingest data from an Amazon Kinesis Data Stream into an Amazon S3 bucket. The data must be delivered in near real time, and the engineer wants to minimize operational overhead by using a fully managed service that can also transform the data format from JSON to Parquet. Which AWS service should be used?

Accepted Answer

Amazon Kinesis Data Firehose. Amazon Kinesis Data Firehose is the correct choice because it is fully managed, can ingest from Kinesis Data Streams, and supports built-in conversion of JSON to Parquet before delivering to Amazon S3. It requires minimal operational effort and provides near real-time delivery. The other options either require custom code, are batch-oriented, or do not support the required source and transformations.

Answer

AWS Glue ETL job

Answer

Amazon Kinesis Client Library (KCL) application

Answer

AWS Database Migration Service (AWS DMS)

Question 3

A data engineer must load a 250 GB uncompressed CSV dataset from Amazon S3 into Amazon Redshift. The data must be loaded daily, and the engineer wants to minimize load time and avoid saturating the cluster's leader node. Which approach meets these requirements?

Accepted Answer

Use the COPY command with a manifest file that references multiple files in Amazon S3.. The COPY command is the most efficient way to load large datasets from Amazon S3 into Amazon Redshift. By using a manifest file that lists multiple S3 objects, the load is parallelized across all slices, reducing load time and preventing the leader node from becoming a bottleneck. This is the standard best practice for bulk data ingestion into Redshift.

Answer

Use the COPY command with a single large file and the COMPUPDATE OFF option.

Answer

Use Amazon Redshift Spectrum to query the CSV files directly from Amazon S3 without loading them.

Answer

Use AWS Database Migration Service (AWS DMS) with a full load task to copy the CSV data into Amazon Redshift.

Question 4

A data engineer is configuring an AWS Glue ETL job that reads from an Amazon S3 bucket containing sensitive customer records. The job must remove all columns flagged as PII before writing to a target S3 location. The PII columns are not known in advance and vary by file. Which approach should the engineer use to ensure the PII is removed dynamically?

Accepted Answer

Use a custom PySpark script within the Glue job that scans column names and data samples against a regex pattern for PII, then drops matching columns.. The requirement is to dynamically remove PII columns that vary by file. AWS Glue does not have a native PII detection transform, and the Data Catalog does not store PII classifications unless manually added. A custom PySpark script within the Glue job can inspect schema and data, apply regex patterns, and drop matching columns. This provides the necessary flexibility and scalability.

Answer

Use AWS Glue's `DetectPII` transform to automatically identify and redact PII columns based on built-in patterns.

Answer

Use the AWS Glue DynamicFrame's `drop_fields` method with a list of PII column names derived from the Glue Data Catalog metadata.

Answer

Use AWS Glue DataBrew to create a recipe that identifies PII columns using pattern matching and apply it as a transformation step in the Glue job.

Question 5

A data engineer is using AWS Glue to transform data from Amazon S3 and load it into Amazon Redshift. The job needs to run daily and process only new data since the last run. The data in S3 is partitioned by date in the format `year=YYYY/month=MM/day=DD/`. Which feature should the data engineer use to track processed partitions and avoid reprocessing old data?

Accepted Answer

AWS Glue job bookmarks. AWS Glue job bookmarks are designed to track the state of data processed by a job. When reading from partitioned S3 data, bookmarks record the last processed partition and only process new partitions in subsequent runs. This enables incremental processing without manual intervention, making it the correct choice for avoiding reprocessing of old data.

Answer

AWS Glue crawler update schedules

Answer

AWS Glue triggers with a schedule

Answer

AWS Glue Data Catalog partition indexes

Question 6

A data engineer needs to ingest data from an Amazon S3 bucket into an Amazon Redshift cluster. The data is in CSV format and is updated daily. The engineer wants to use a fully managed service that can handle the load without managing infrastructure. Which AWS service should the engineer use?

Accepted Answer

AWS Glue. AWS Glue is a fully managed ETL service that can read CSV files from Amazon S3, apply transformations if needed, and load the data into Amazon Redshift. It handles the underlying infrastructure, scales automatically, and can be scheduled to run daily. This makes it the ideal choice for the scenario.

Answer

AWS Database Migration Service (AWS DMS)

Answer

Amazon Redshift Spectrum

Answer

Amazon Kinesis Data Firehose

Question 7

A data engineer is configuring an AWS Glue ETL job to read data from an Amazon S3 bucket that contains nested JSON files. The job must flatten the nested structure and write the output to Amazon Redshift. The engineer wants to optimize the job for performance and cost. Which two actions should the engineer take? (Choose two.)

Accepted Answer

Enable job bookmarks to track processed data and avoid reprocessing.. The Relationalize transform is purpose-built to flatten nested JSON, simplifying the ETL code and improving performance. Enabling job bookmarks ensures that only new or changed data is processed on subsequent runs, reducing cost and runtime. Together, these actions optimize the job for both performance and cost. The other options either do not address flattening, are not cost-effective, or misunderstand service capabilities.

Answer

Increase the number of DPUs to the maximum allowed to ensure faster execution.

Answer

Convert the JSON files to Parquet using an AWS Glue crawler before running the ETL job.

Answer

Use the AWS Glue ResolveChoice transform to handle data type conflicts.

Question 8

A data engineer is using AWS Glue to process a large dataset in Amazon S3. The dataset consists of many small JSON files (average 100 KB each) stored in a single prefix. The Glue job reads these files, performs transformations, and writes the output to Parquet in another S3 location. The job is running slowly and consuming many DPUs. Which action should the data engineer take to improve performance?

Accepted Answer

Use AWS Glue's `groupFiles` option to group multiple small files into larger chunks for processing.. The `groupFiles` option in AWS Glue allows the job to group multiple small files into a single partition, reducing the number of tasks and the overhead of reading many small files. This directly addresses the performance bottleneck caused by many small JSON files, improving job speed and reducing DPU consumption. Other options do not solve the small file issue.

Answer

Convert the JSON files to Parquet using an AWS Glue crawler before processing.

Answer

Enable AWS Glue job bookmarks to skip already processed files.

Answer

Increase the number of DPUs allocated to the job to enable more parallel tasks.

Question 9

A data engineer is using AWS Glue DataBrew to clean a dataset stored in Amazon S3. The dataset contains inconsistent date formats in a column named 'transaction_date'. The engineer needs to standardize all dates to ISO 8601 format (YYYY-MM-DD) and then write the cleaned data to a new S3 location. Which transformation should the engineer apply?

Accepted Answer

Use the 'Format date' transformation with a custom date format pattern.. The 'Format date' transformation in AWS Glue DataBrew is designed to parse and reformat date values. By specifying the input format(s) and setting the output format to 'yyyy-MM-dd', the engineer can standardize inconsistent date strings into ISO 8601. This ensures consistency for downstream processing and analytics, and it handles multiple input patterns without manual string manipulation.

Answer

Use the 'Change data type' transformation to convert the column to a date type.

Answer

Use the 'Replace value' transformation to replace all date separators with hyphens.

Answer

Use the 'Split column' transformation to separate day, month, and year into individual columns.

Question 10

A data engineer must ingest a 4 TB Oracle database into Amazon S3 nightly. The database is on-premises and the network link supports only 200 Mbps. The engineer wants to minimize the total transfer time and avoid impacting production. Which approach should the engineer use?

Accepted Answer

Use AWS Snowball Edge devices to transfer the data offline and import into Amazon S3.. For multi-terabyte datasets and limited bandwidth, offline transfer with AWS Snowball Edge is the recommended approach. It bypasses the network bottleneck, reduces transfer time, and avoids impacting production systems. Online methods like DMS, DataSync, or S3 Transfer Acceleration are constrained by the available bandwidth and are not optimal for this volume.

Answer

Use Amazon S3 Transfer Acceleration with multipart uploads from an on-premises script.

Answer

Use AWS DataSync with an on-premises agent to transfer the data over the internet.

Answer

Use AWS Database Migration Service (AWS DMS) with a replication instance and an S3 target endpoint.

Question 11

A data engineer is using AWS Glue to run an ETL job that reads from Amazon S3, performs a join between two large datasets, and writes the result to Amazon Redshift. The job is taking longer than expected, and the engineer suspects data skew. Which technique can help mitigate data skew in the join?

Accepted Answer

Enable the 'spark.sql.adaptive.enabled' configuration and set 'spark.sql.adaptive.skewJoin.enabled' to true.. Enabling Adaptive Query Execution with skew join handling allows Spark to dynamically detect and split skewed partitions during the join. This redistributes the workload more evenly across executors, mitigating the performance bottleneck caused by data skew.

Answer

Use AWS Glue's 'groupFiles' option to combine small files before the join.

Answer

Increase the number of DPUs to provide more memory for the join operation.

Answer

Use a broadcast join to replicate the smaller dataset across all nodes.

Question 12

A data engineer is using AWS Glue to transform data from Amazon S3 and load it into Amazon S3 in Parquet format. The source data is in JSON format and contains nested structures. The engineer needs to flatten the nested data and write it to Parquet. Which AWS Glue transform should the engineer use to flatten the nested structure?

Accepted Answer

Relationalize. The Relationalize transform in AWS Glue is designed to flatten nested JSON structures into a relational format. It automatically creates multiple DynamicFrames for nested arrays and objects, generating keys to join them. This makes it the correct choice for the scenario, as it simplifies the flattening process without custom code.

Answer

Map

Answer

ApplyMapping

Answer

Filter

Question 13

A data engineer is using AWS Glue to transform data from Amazon S3 and load it into Amazon Redshift. The job runs daily and processes only new data added since the last run. The engineer notices that the job is reprocessing old data, increasing runtime and cost. Which AWS Glue feature should be enabled to ensure only new data is processed?

Accepted Answer

Job bookmarks. Job bookmarks are the AWS Glue feature designed to track processed data and enable incremental processing. When enabled, the job reads only new data based on S3 object metadata. This reduces runtime and cost by avoiding reprocessing. The other options do not provide this capability: DataBrew is for data preparation, crawler incremental crawl updates the catalog, and S3 event notifications are for triggering actions.

Answer

AWS Glue DataBrew

Answer

Amazon S3 event notifications

Answer

AWS Glue Crawler incremental crawl

Question 14

A data engineer maintains an AWS Glue ETL job that reads JSON files from Amazon S3, applies transformations using a DynamicFrame, and writes Parquet to another S3 bucket. The job currently reads the entire source prefix on every run, but the source is now partitioned by year/month/day and only new partitions need processing. The engineer wants to process only new partitions without changing the job's transformation logic. Which change should the engineer make?

Accepted Answer

Enable job bookmarks on the Glue job so it tracks previously processed S3 objects and skips them on subsequent runs. Job bookmarks are the native Glue mechanism for incremental processing. When enabled, Glue tracks the state of previously processed S3 objects and partitions, so each run reads only new data. This meets the requirement to process only new partitions without rewriting the transformation logic or manually specifying dates.

Answer

Create a Glue crawler that catalogs only the new partitions and point the job at the crawler's output table

Answer

Configure the job to use a smaller number of DPUs so each run scans less data

Answer

Add a pushdown predicate to the DynamicFrame that filters on the year, month, and day partition columns using hardcoded values

Question 15

A data engineer is using AWS Glue DataBrew to clean a dataset stored in Amazon S3. The dataset contains a column with inconsistent date formats (for example, '2023-01-15', '01/15/2023', and '15-Jan-2023'). The engineer needs to standardize all values to ISO 8601 format and then write the cleaned data back to S3. Which approach should the engineer use?

Accepted Answer

Use the 'Standardize date' transform in DataBrew, select the column, and choose the target format as ISO 8601.. AWS Glue DataBrew includes a 'Standardize date' transform that parses multiple date formats and converts them to a specified target format such as ISO 8601. This transform is designed for exactly this scenario, where a column contains mixed date representations. Using it avoids custom code and ensures consistent, repeatable cleaning across the dataset.

Answer

Use the 'Format' transform in DataBrew to change the column data type to date.

Answer

Use the 'Replace value' transform to manually replace each non-ISO date string with its ISO equivalent.

Answer

Write a custom Python function in a DataBrew recipe step using the 'Custom transform' option.