DEA-C01 Data Operations and Support Practice Test 32 — 15 Questions

Question 1

A data engineer maintains an AWS Glue ETL job that processes millions of small JSON files stored in Amazon S3. The job's runtime has increased significantly, and CloudWatch logs show many small executor tasks and frequent garbage collection. The engineer wants to improve job performance by reducing the number of small files processed per task. Which action should the engineer take?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Use the Glue ETL job's 'groupFiles' option to group multiple small files into a single partition.. Grouping small files into larger partitions reduces the number of tasks and the overhead of task scheduling and garbage collection. This is a specific optimization for AWS Glue ETL jobs that read many small files. Enabling job bookmarks or increasing DPUs does not address the root cause. Converting formats via a crawler is not a valid approach because crawlers do not transform data.

Answer

Convert the source files to Parquet format using an AWS Glue crawler before running the ETL job.

Answer

Enable job bookmarks to track previously processed files and skip them on subsequent runs.

Answer

Increase the number of DPUs allocated to the job to provide more memory per executor.

Question 2

A data engineer is troubleshooting an AWS Glue job that is reading from an Amazon Kinesis Data Stream. The job is configured with a 1-minute window and is supposed to process the latest records. However, the engineer notices that the job is reprocessing old data from the stream. What is the most likely cause of this issue?

Accepted Answer

The Glue job is configured with the starting position 'TRIM_HORIZON', which reads all available records from the beginning of the stream.. When an AWS Glue job reads from Kinesis, the starting position determines where reading begins. 'TRIM_HORIZON' reads from the oldest record, while 'LATEST' reads only new records. If the job is reprocessing old data, the starting position is likely set to 'TRIM_HORIZON'. Adjusting this to 'LATEST' will make the job process only new data going forward.

Answer

The Glue job's checkpointing is disabled, so it restarts from the beginning each time it runs.

Answer

The Kinesis stream has multiple shards, and the Glue job is not parallelizing reads across all shards, causing it to reread from one shard.

Answer

The Glue job's bookmarking is enabled, causing it to track processed records and reprocess them if the job fails.

Question 3

A data engineer is monitoring an AWS Glue job that reads from Amazon S3 and writes to Amazon Redshift. The job runs daily and recently started taking significantly longer to complete. The engineer checks the job metrics and notices that the number of DPUs used is consistently at the maximum allocated, and the job's Spark UI shows many tasks spilling to disk. Which action should the engineer take to improve performance?

Accepted Answer

Repartition the data or adjust the number of partitions to reduce skew and improve memory usage.. Disk spilling in Spark indicates that executors are running out of memory during processing, often due to data skew or insufficient partitions. Repartitioning the data or adjusting partition counts distributes the workload more evenly across executors, reducing memory pressure and eliminating spilling. This is a targeted fix for the observed symptom and improves job performance without unnecessary resource increases.

Answer

Change the output format from Parquet to CSV to reduce write overhead.

Answer

Increase the number of DPUs allocated to the job.

Answer

Enable job bookmarks to avoid reprocessing old data.

Question 4

A data engineer is using AWS Step Functions to orchestrate a series of AWS Glue jobs and Amazon EMR steps. The state machine has a task that starts a Glue job and waits for completion using the 'StartJobRun' API with the 'sync' integration. Occasionally, the Step Functions execution fails with the error: 'States.TaskFailed: Glue job failed with error: ResourceNumberLimitExceededException'. The engineer confirms that the Glue job itself runs successfully when triggered manually. What is the most likely cause of this intermittent failure?

Accepted Answer

The AWS Glue job is hitting the concurrent job run limit for the account.. The error 'ResourceNumberLimitExceededException' from AWS Glue indicates that the account has reached the maximum number of concurrent job runs. Step Functions may trigger the job multiple times concurrently, or other processes may be running Glue jobs. The engineer should check the Glue service quotas and either increase the concurrent runs limit or implement a queueing mechanism to serialize job starts.

Answer

The Glue job's script has a bug that causes it to fail only when run in parallel.

Answer

The IAM role used by Step Functions lacks permissions to start the Glue job.

Answer

The Step Functions state machine is exceeding the maximum number of concurrent executions.

Question 5

A data engineer is using AWS Step Functions to orchestrate a series of AWS Glue jobs. During a recent run, a Glue job failed due to a transient network issue, and the Step Functions execution stopped. The engineer needs to ensure that the workflow can automatically retry the failed Glue job up to three times before considering the step failed. Which Step Functions state configuration should the engineer implement?

Accepted Answer

Add a Retry field to the Task state with ErrorEquals set to States.ALL and MaxAttempts set to 3.. The Retry field in Step Functions is designed to automatically retry a task when specified errors occur. By setting ErrorEquals to States.ALL and MaxAttempts to 3, the Glue job will be retried up to three times on any error, including transient network issues. The Catch field is for fallback, not retry. Glue job definitions do not have a retry count, and timeout/heartbeat fields are for monitoring, not retry logic.

Answer

Set the TimeoutSeconds and HeartbeatSeconds fields in the Task state to trigger a retry on failure.

Answer

Add a Catch field to the Task state with ErrorEquals set to States.ALL and Next set to a retry state.

Answer

Configure the Glue job itself to have a maximum retry count of 3 in its job definition.

Question 6

A data engineer needs to schedule a recurring AWS Glue ETL job to run every day at 2:00 AM UTC. The job must be triggered automatically without manual intervention. Which AWS service should the engineer use to create this schedule?

Accepted Answer

AWS Glue triggers. AWS Glue triggers provide a built-in scheduling capability for Glue jobs. A time-based trigger uses a cron expression to define the schedule, such as 'cron(0 2 * * ? *)' for 2:00 AM UTC daily. This eliminates the need for external services and keeps the scheduling logic within AWS Glue, simplifying management and monitoring.

Answer

Amazon EventBridge (CloudWatch Events) rules

Answer

AWS Lambda with a CloudWatch Events schedule

Answer

AWS Step Functions with a Wait state

Question 7

A data engineer has an AWS Glue job that processes data from an Amazon S3 bucket and writes to an Amazon Redshift cluster. The job is scheduled to run daily. Recently, the job started failing with the error: 'java.sql.SQLException: [Amazon](500310) Invalid operation: Spectrum Scan Error: S3 Access Denied'. The engineer verifies that the IAM role associated with the Glue job has full access to the S3 bucket. What is the most likely cause of this error?

Accepted Answer

The Amazon Redshift cluster's IAM role lacks permission to access the S3 bucket.. The error occurs because Amazon Redshift, not AWS Glue, is attempting to access S3 during the COPY operation. The Redshift cluster's IAM role must have permissions to read the S3 bucket. Even if the Glue job's role has access, Redshift uses its own role. The engineer should attach a policy to the Redshift cluster's IAM role granting S3 read access.

Answer

The Glue job's script is using an incorrect S3 path for the Redshift COPY command.

Answer

The Amazon Redshift cluster is in a different AWS Region than the S3 bucket, causing access latency.

Answer

The S3 bucket policy explicitly denies access to the Glue job's IAM role.

Question 8

A data engineer is using AWS Step Functions to orchestrate a series of AWS Glue jobs. The engineer notices that when a Glue job fails, the Step Functions execution also fails, but the engineer wants to retry the failed job up to three times with exponential backoff before failing the entire workflow. The engineer needs to implement this with minimal changes to the state machine. What should the engineer do?

Accepted Answer

Add a Retry field with MaxAttempts: 3 and BackoffRate: 2.0 to the state that invokes the Glue job.. AWS Step Functions provides native retry capabilities via the Retry field in the state definition. Specifying MaxAttempts and BackoffRate on the state that calls the Glue job enables automatic retries with exponential backoff. This is the simplest and most direct method to meet the requirement without altering the Glue job or adding external components.

Answer

Configure the Glue job to automatically retry on failure by setting the MaxRetries parameter in the job's default arguments.

Answer

Create an Amazon CloudWatch alarm that triggers an AWS Lambda function to restart the Glue job on failure.

Answer

Modify the state machine to use a Map state that iterates over the Glue job three times.

Question 9

A data engineer is managing an Amazon Redshift cluster that experiences performance degradation during peak query hours. The engineer notices that many queries are waiting in the queue, and the WLM query queue wait time is high. The cluster uses automatic WLM. Which action should the engineer take to improve query throughput?

Accepted Answer

Enable concurrency scaling to automatically add transient clusters for bursts of queries.. Concurrency scaling automatically adds transient clusters to handle query bursts, reducing queue wait times. It is specifically designed to improve throughput during peak periods. Manually increasing concurrency is not possible with automatic WLM. Adding nodes or changing distribution style does not directly address queue wait times caused by concurrency limits.

Answer

Change the cluster's distribution style to EVEN to balance data across nodes.

Answer

Increase the number of nodes in the cluster to add more compute resources.

Answer

Modify the WLM configuration to increase the concurrency level for the queue.

Question 10

A data engineer maintains an AWS Glue ETL job that processes millions of small JSON files stored in Amazon S3. The job's runtime has increased significantly, and CloudWatch logs show many small executor tasks and frequent garbage collection. The engineer wants to improve job performance by reducing the number of small files processed per task. Which action should the engineer take?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Verify that the Kinesis Data Streams source is configured with the correct stream name and that the job's transformation does not include an unsupported operation that blocks bookmark propagation, such as a custom transformation that does not preserve the bookmark state.. Job bookmarks in AWS Glue track the last processed data to avoid reprocessing. When bookmarks fail to advance, it is typically due to an unsupported transformation or a misconfigured source. Ensuring the Kinesis source is correctly configured and that all transformations preserve bookmark state will allow the job to process only new data and advance the bookmark.

Answer

Disable job bookmarks and instead use an AWS Lambda function to track the last processed Kinesis sequence number in Amazon DynamoDB.

Answer

Increase the number of AWS Glue DPUs allocated to the job and enable auto-scaling.

Answer

Change the output format from Parquet to JSON, as Parquet does not support job bookmarks with Kinesis sources.

Question 11

A data engineer is using Amazon Kinesis Data Firehose to deliver streaming data to an Amazon S3 bucket. The engineer notices that some data records are missing from S3, and the Firehose delivery stream metrics show increased DeliveryToS3.DataFreshness. The engineer needs to ensure all records are delivered. Which action should the engineer take?

Accepted Answer

Enable error logging on the Firehose delivery stream and review the error output in Amazon CloudWatch Logs.. Enabling error logging on the Firehose delivery stream is crucial for diagnosing why records are not delivered to S3. The logs in CloudWatch will show specific errors such as access denied or invalid records, allowing the engineer to take corrective action. This directly addresses the missing data and high data freshness.

Answer

Increase the number of shards in the source Kinesis data stream to improve throughput.

Answer

Enable Amazon S3 server access logging on the destination bucket to capture failed deliveries.

Answer

Configure the Firehose delivery stream to use a longer buffering interval to reduce the number of S3 PUT requests.

Question 12

A data engineer is using Amazon Kinesis Data Streams to ingest real-time data. The engineer needs to ensure that records are delivered to consumers in the same order they were written and that each record is processed exactly once. Which combination of Kinesis Data Streams features should the engineer use?

Accepted Answer

Use a single shard and implement idempotent processing in the consumer application.. To guarantee order, all records must go to a single shard, as Kinesis preserves order only within a shard. For exactly-once processing, the consumer must be idempotent because Kinesis Data Streams provides at-least-once delivery. Combining a single shard with idempotent processing ensures both requirements are met.

Answer

Use partition keys to distribute records across shards and enable AWS Lambda as a consumer with checkpointing.

Answer

Use multiple shards with sequence numbers and enable Kinesis Client Library (KCL) with deduplication.

Answer

Use a single shard and enable enhanced fan-out for consumers.

Question 13

A data engineer is using AWS Step Functions to orchestrate a series of AWS Glue jobs that process data in Amazon S3. The engineer needs to ensure that if a Glue job fails, the entire workflow stops and an Amazon SNS notification is sent. Which Step Functions state type should be used to handle the error and send the notification?

Accepted Answer

A Task state with a Catch field that transitions to a state that publishes to an Amazon SNS topic.. In AWS Step Functions, the Catch field on a Task state allows you to define a fallback state when an error occurs. By transitioning to a state that publishes to an Amazon SNS topic, the workflow can stop and send a notification. This is the correct pattern for error handling and alerting in Step Functions.

Answer

A Choice state that evaluates the output of the Glue job and branches based on success or failure.

Answer

A Parallel state that runs the Glue job and an SNS notification task simultaneously.

Answer

A Task state with a Retry field that retries the Glue job up to three times.

Question 14

A data engineer runs an AWS Glue ETL job that reads CSV files from an Amazon S3 bucket, applies transformations, and writes Parquet output to another S3 bucket. The job fails with the error 'AnalysisException: Unable to infer schema for CSV. It must be specified manually.' The CSV files are stored with a header row, and the job's script uses the default Glue DynamicFrame reader without specifying format options. What is the MOST likely cause of the failure?

Accepted Answer

The CSV files have inconsistent or malformed data that prevents Glue from inferring a schema, or the header option is not set correctly.. The error 'Unable to infer schema for CSV' occurs when AWS Glue cannot determine column names and types from the source data. This often happens when the header option is not enabled or when the CSV data has inconsistencies such as varying column counts or mixed data types. Ensuring the reader is configured with 'withHeader' set to true and that data is well-formed allows Glue to infer the schema correctly.

Answer

The CSV files are compressed with gzip, and Glue cannot infer schema from compressed CSV files.

Answer

The S3 bucket containing the CSV files does not have the correct bucket policy allowing AWS Glue to read objects.

Answer

The Glue job's IAM role lacks permissions to read the CSV files from the S3 bucket.

Question 15

A data engineer is using AWS Database Migration Service (AWS DMS) to migrate an on-premises Oracle database to Amazon Aurora PostgreSQL. The migration is running but the engineer observes that some tables are not being replicated. The DMS task logs show no errors, and the task status is 'Running'. Which action should the engineer take to identify the missing tables?

Accepted Answer

Check the DMS task's table mappings to ensure all required tables are included in the selection rules.. Table mappings in AWS DMS determine which tables are migrated. If tables are missing and no errors are present, the most likely cause is that the tables are not included in the selection rules. Reviewing and correcting the table mappings is the direct way to ensure all required tables are replicated.

Answer

Enable CloudWatch Logs for the DMS task and look for warnings about table selection.

Answer

Increase the DMS replication instance size to handle more tables concurrently.

Answer

Restart the DMS task with the 'Reload target' option to force a full load of all tables.