DEA-C01 Data Operations and Support Practice Test 15 — 15 Questions

Question 1

A data engineer needs to schedule a daily AWS Glue ETL job that extracts data from an Amazon RDS for MySQL database and writes to Amazon S3. The engineer wants to ensure the job runs reliably every day at 2:00 AM UTC. Which AWS service should the engineer use to schedule the Glue job?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

AWS Glue Triggers with a schedule. AWS Glue Triggers are the native scheduling mechanism for Glue jobs. They support cron expressions, allowing precise scheduling. Creating a trigger with the schedule 'cron(0 2 * * ? *)' will start the job daily at 2:00 AM UTC. This is the simplest and most integrated solution, avoiding additional services or custom code.

Answer

AWS Step Functions with a Wait state

Answer

AWS Lambda function with a CloudWatch Events schedule

Answer

Amazon EventBridge rule with a cron expression

Question 2

A data engineer is using Amazon Athena to query data stored in Amazon S3. The engineer notices that queries are returning incorrect results for a table that is partitioned by date. Upon investigation, the engineer finds that new data is being written to S3 with a different date format than the existing partitions. The table is defined with partitions in the AWS Glue Data Catalog. What should the engineer do to ensure Athena queries return correct results?

Accepted Answer

Run MSCK REPAIR TABLE to update the partition metadata in the Glue Data Catalog.. When new partitions are added to S3 with a different format, they are not automatically registered in the Glue Data Catalog. Running MSCK REPAIR TABLE scans the S3 location and adds any new partitions to the catalog, allowing Athena to query them. This is the standard method to synchronize partitions.

Answer

Modify the table's schema to change the partition column data type to string to accommodate any format.

Answer

Use AWS Glue crawler to re-crawl the S3 location and update the table definition.

Answer

Update the table's partition projection settings to include the new date format.

Question 3

A data engineer is configuring an AWS Glue job to read from an Amazon Kinesis Data Stream and write to Amazon S3 in Parquet format. The job must handle stream data continuously and ensure exactly-once processing. Which two configurations should the engineer implement? (Choose two.)

Accepted Answer

Enable job bookmarks to track processed records.. Exactly-once processing in AWS Glue streaming jobs requires both job bookmarks and Spark checkpointing. Job bookmarks track the last processed record from Kinesis, while checkpointing stores the state of the stream processing to recover from failures without duplication. Together, they ensure each record is processed once.

Answer

Set the job's Glue version to 3.0 or later.

Answer

Set the job's worker type to G.1X to handle the stream throughput.

Answer

Configure the job to use the 'kinesis' connection type with enhanced fan-out.

Question 4

A data engineer is using AWS Step Functions to orchestrate a series of AWS Glue jobs. During a recent run, a Glue job failed due to a transient network error. The engineer wants to implement a retry mechanism that retries the failed job up to three times with an exponential backoff. Which TWO configurations in the Step Functions state machine will achieve this? (Choose two.)

Accepted Answer

Define a Retry field in the state that invokes the Glue job, specifying MaxAttempts as 3 and BackoffRate as 2.0.. In AWS Step Functions, the Retry field within a state definition enables automatic retries for that state. To achieve up to three retries with exponential backoff, you specify MaxAttempts as 3 and BackoffRate as 2.0. The IntervalSeconds sets the initial delay. Both configurations that include these parameters are correct. Other options either do not use the built-in Retry mechanism or rely on unsupported features.

Answer

Set the Step Functions state machine's timeout to allow for retries and rely on Step Functions' automatic retry for all states.

Answer

Use a Catch field to transition to a retry state that increments a counter and loops back to the Glue job state until the counter reaches 3.

Answer

Configure the Glue job itself to have a retry policy with MaxRetries set to 3 and a backoff interval.

Question 5

A data engineer is using AWS Glue DataBrew to clean a dataset stored in Amazon S3. The dataset contains a column 'order_date' with values in mixed formats (e.g., '2023-01-15', '01/15/2023', '15-Jan-2023'). The engineer needs to standardize all values into ISO 8601 format (YYYY-MM-DD) before loading into Amazon Redshift. Which DataBrew transformation should be used?

Accepted Answer

Use the 'Date parse' transformation with the appropriate input formats and output format.. The 'Date parse' transformation is designed to handle multiple input date formats and convert them to a single output format. It allows the engineer to specify all observed formats and the desired ISO 8601 output, ensuring consistent date values. Other transformations either do not exist or assume a consistent input format.

Answer

Use the 'Convert type' transformation to change the column to date type.

Answer

Use the 'Format date' transformation and specify the output pattern.

Answer

Use the 'Standardize date' transformation and select the target format.

Question 6

A data engineer manages an AWS Glue ETL job that reads JSON files from an Amazon S3 bucket and writes Parquet files to another S3 bucket. The job was working for months, but now it fails with an error indicating an 'InvalidInputException: Unable to infer schema'. The S3 bucket contains many JSON files, and some files have a different structure than the original files. The engineer needs to resolve the error without changing the job's output format. What should the engineer do?

Accepted Answer

Add a transformation that applies a fixed schema to the DynamicFrame using the ApplyMapping transform, specifying the expected columns and data types.. When AWS Glue infers schema from JSON files with varying structures, it can fail with an InvalidInputException. Explicitly defining the schema using a transform like ApplyMapping ensures the job uses a consistent schema, allowing it to process all files regardless of variations. This approach is robust and maintains the output format, directly addressing the root cause.

Answer

Modify the job to read only files that match the original schema by filtering S3 paths using a glob pattern.

Answer

Use the AWS Glue crawler to update the Data Catalog table with the new schema, then rerun the job.

Answer

Increase the number of AWS Glue DPUs allocated to the job to provide more memory for schema inference.

Question 7

A data engineer is using AWS Step Functions to orchestrate a complex ETL workflow that includes AWS Glue jobs, Amazon EMR steps, and AWS Lambda functions. The workflow intermittently fails with a States.Timeout error on a Glue job task. The Glue job itself completes successfully in about 45 minutes. The engineer needs to ensure the workflow does not time out prematurely. Which action should the engineer take?

Accepted Answer

Increase the TimeoutSeconds parameter on the Glue job task in the Step Functions state machine definition.. The States.Timeout error on a Step Functions task occurs when the task exceeds its configured timeout. For a Glue job that takes 45 minutes, the default task timeout of 60 seconds is insufficient. Increasing TimeoutSeconds on the task allows Step Functions to wait for the job's completion. Other options do not address the specific timeout configuration of the Step Functions task.

Answer

Configure a retry policy with exponential backoff on the Glue job task to handle transient timeouts.

Answer

Increase the Glue job's timeout parameter in the job definition to allow more time for completion.

Answer

Change the Step Functions state machine to use Express Workflows instead of Standard Workflows.

Question 8

A data engineer is using Amazon Kinesis Data Firehose to deliver streaming data to an Amazon S3 bucket. The engineer notices that some records are being delivered to S3 with a delay of several minutes, even though the Firehose delivery stream is configured with a buffer interval of 60 seconds. What is the most likely reason for this delay?

Accepted Answer

The Firehose delivery stream is configured to transform records with an AWS Lambda function, which adds processing time.. Kinesis Data Firehose delivers data to S3 based on buffer size or buffer interval, whichever comes first. If a Lambda function is used for record transformation, its processing time, including cold starts, can add latency. This can cause deliveries to take longer than the configured buffer interval. Cross-region delivery is not supported, buffer size does not override the interval, and S3 versioning does not add latency.

Answer

The Firehose delivery stream's buffer size is set too high, causing it to wait until the buffer is full before delivering.

Answer

The S3 bucket is in a different AWS region than the Firehose delivery stream.

Answer

The S3 bucket has versioning enabled, which adds latency to the delivery.

Question 9

A data engineer needs to schedule an AWS Glue crawler to run every day at 2:00 AM UTC. The engineer wants to minimize operational overhead and ensure the crawler runs reliably. Which AWS service should the engineer use to schedule the crawler?

Accepted Answer

AWS Glue triggers.. AWS Glue triggers are the native scheduling mechanism for Glue crawlers and jobs. They support cron expressions and require no additional services or code, making them the lowest-overhead solution. Other options introduce unnecessary complexity for a simple scheduled task.

Answer

AWS Step Functions with a Wait state.

Answer

Amazon EventBridge Scheduler with a universal target.

Answer

AWS Lambda with a CloudWatch Events rule.

Question 10

A data engineer needs to schedule an AWS Glue ETL job to run every day at 2:00 AM UTC. The job extracts data from Amazon S3, transforms it, and loads it into Amazon Redshift. The engineer wants to minimize operational overhead and ensure the job runs reliably. Which method should be used to schedule the job?

Accepted Answer

Use AWS Glue triggers with a schedule type of 'Scheduled' and a cron expression.. AWS Glue triggers provide a native scheduling capability for Glue jobs. A scheduled trigger uses a cron expression to run the job at specified times, with built-in retry and monitoring. This minimizes operational overhead compared to custom solutions like Lambda or Step Functions.

Answer

Use AWS Step Functions with a Wait state to delay execution until 2:00 AM.

Answer

Configure an AWS Lambda function with a CloudWatch Events (EventBridge) rule to start the Glue job.

Answer

Create an Amazon EventBridge rule with a cron expression and set the target to the Glue job.

Question 11

A data engineer needs to schedule a daily AWS Glue ETL job that extracts data from an Amazon RDS for MySQL database and writes to Amazon S3. The engineer wants to ensure the job runs reliably every day at 2:00 AM UTC. Which AWS service should the engineer use to schedule the Glue job?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

AWS Glue Triggers. AWS Glue Triggers are the native scheduling mechanism for Glue jobs. They support cron expressions to define when a job should run, and they can be configured to retry on failure and send notifications. While other services like EventBridge can also schedule jobs, Glue Triggers are the most direct and integrated solution for this scenario.

Answer

AWS Lambda with a cron expression

Answer

AWS Step Functions

Answer

Amazon EventBridge (Amazon CloudWatch Events)

Question 12

A data engineer is using AWS Glue DataBrew to clean a dataset stored in Amazon S3. The dataset contains a column 'order_date' with values in various formats, such as '2023-01-15', '01/15/2023', and '15-Jan-2023'. The engineer wants to standardize all dates to the ISO format 'YYYY-MM-DD'. Which DataBrew transformation should the engineer use to achieve this?

Accepted Answer

Use the 'Custom formula' transformation with a SQL expression that handles multiple formats.. DataBrew does not have a built-in transformation that automatically handles multiple date formats. The most reliable way is to use a 'Custom formula' transformation with a SQL expression that checks the pattern of each value and applies the appropriate conversion to ISO format. This allows the engineer to handle all variations in one step.

Answer

Use the 'Standardize date' transformation with the 'ISO 8601' format.

Answer

Use the 'Format date' transformation with a custom format string.

Answer

Use the 'Parse date' transformation with the 'auto-detect' option.

Question 13

A data engineer is designing a data pipeline that uses Amazon Kinesis Data Streams to ingest clickstream data. The engineer needs to ensure that the data is processed in near real-time and that records are not lost if the consumer application fails. The engineer also wants to minimize the number of API calls to Kinesis. Which two actions should the engineer take to meet these requirements? (Choose two.)

Accepted Answer

Use AWS Lambda to process the stream with event source mapping and configure a dead-letter queue (DLQ) for failed records.. The Kinesis Client Library (KCL) with checkpointing to DynamoDB and AWS Lambda with event source mapping and a DLQ both provide reliable, near real-time processing with minimal API calls. KCL handles checkpointing and failure recovery, while Lambda's event source mapping polls the stream and a DLQ captures failed records. The other options either do not address record loss or increase API calls.

Answer

Enable Kinesis Data Streams' server-side encryption to protect data at rest.

Answer

Configure the consumer to use enhanced fan-out with HTTP/2 to get dedicated throughput per consumer.

Answer

Increase the number of shards in the stream to improve throughput and reduce latency.

Question 14

A data engineer is using AWS Glue DataBrew to clean and transform a dataset stored in Amazon S3. The dataset contains a column with inconsistent date formats (e.g., '2023-01-15', '01/15/2023', '15-Jan-2023'). The engineer needs to standardize all dates to ISO 8601 format ('YYYY-MM-DD') for downstream processing. Which approach should the engineer take in DataBrew?

Accepted Answer

Apply a series of 'Parse date' transforms, each targeting a specific input format, followed by a 'Format date' transform to output YYYY-MM-DD.. When a column contains multiple date formats, a single standardize transform may not suffice. DataBrew's 'Parse date' transform lets you define the input format for each variation, converting them into a consistent date type. Then, a 'Format date' transform outputs the desired ISO 8601 string. This two-step approach handles mixed formats accurately and is repeatable for production pipelines.

Answer

Use the 'Standardize date' transform and select the target format as YYYY-MM-DD.

Answer

Use the 'Replace value' transform to manually replace each non-ISO date string with its ISO equivalent.

Answer

Use the 'Column from examples' transform to let DataBrew infer the desired date format from a few examples.

Question 15

A data engineer manages an AWS Glue ETL job that processes JSON files from Amazon S3, transforms them, and writes Parquet to another S3 location. The job previously ran for about 10 minutes but now takes over 2 hours and occasionally fails with an executor out-of-memory error. The input data volume has grown from 5 GB to 200 GB, but the number of files remains roughly 500 small files of about 400 MB each. The job's worker type is G.1X with 10 workers. Which action should the engineer take FIRST to improve performance and stability?

Accepted Answer

Repartition the input data into larger files using an AWS Glue job or Amazon S3 compaction, then rerun the ETL job.. The job's performance degradation and memory errors are caused by inefficient processing of many small files. Compacting the input data into larger files reduces the number of partitions and per-task overhead, improving throughput and stability. Scaling workers or enabling bookmarks does not fix the underlying small-file issue. Repartitioning is the most direct and cost-effective first step.

Answer

Switch the job to use Python shell instead of Spark to reduce memory overhead.

Answer

Enable job bookmarks to avoid reprocessing previously processed data.

Answer

Increase the number of workers from 10 to 50 and change worker type to G.2X.