DEA-C01 Data Operations and Support Practice Test 16 — 15 Questions

Question 1

A data engineer manages an AWS Glue ETL job that processes millions of small JSON files from Amazon S3 and writes Parquet to another S3 location. The job runs for hours and often fails with 'Container killed by YARN for exceeding memory limits' or 'No space left on device' during the shuffle phase. The engineer wants to improve reliability and performance without changing the source data. Which change is MOST effective?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Configure Glue to group small files and increase the shuffle partition size.. The job fails during shuffle due to many small input files and large shuffle partitions. Grouping small files reduces task count and overhead, while increasing shuffle partitions spreads data across more reducers, lowering per-task memory and disk usage. These AWS Glue settings directly mitigate the observed errors without altering source data format.

Answer

Enable job bookmarks to skip previously processed files.

Answer

Increase the number of DPUs and enable Glue's auto-scaling.

Answer

Change the output format from Parquet to CSV with compression.

Question 2

A data engineer is troubleshooting an AWS Glue ETL job that intermittently fails with the error 'An error occurred while calling o123.pyWriteDynamicFrame. The specified bucket does not exist.' The job writes to an Amazon S3 bucket that is in a different AWS account. The engineer has verified that the S3 bucket exists and the IAM role used by the Glue job has s3:PutObject permission on the bucket. What is the MOST likely cause of the error?

Accepted Answer

The S3 bucket was recently created, and the Glue job is experiencing eventual consistency issues.. The intermittent 'bucket does not exist' error when writing to a newly created cross-account S3 bucket is characteristic of S3's eventual consistency for bucket creation. Although S3 now provides strong read-after-write consistency for objects, bucket-level operations may still have propagation delays, especially across accounts. Waiting and retrying resolves the issue.

Answer

The Glue job's IAM role lacks permission to list the bucket, causing the write to fail.

Answer

The S3 bucket is in a different AWS Region, and the Glue job is not configured to use the correct endpoint.

Answer

The S3 bucket name is misspelled in the Glue job script.

Question 3

A data engineer is using AWS Glue DataBrew to clean and transform a dataset stored in Amazon S3. The engineer needs to ensure that the DataBrew job can access the S3 bucket securely and that the output is written to a different S3 bucket. The engineer must also ensure that the job can be monitored and that errors are captured. Which TWO actions should the engineer take to meet these requirements? (Choose two.)

Accepted Answer

Attach an IAM role to the DataBrew job that has read permissions on the input S3 bucket and write permissions on the output S3 bucket.. To securely access S3, the DataBrew job needs an IAM role with appropriate read and write permissions. To monitor and capture errors, enabling job metrics and setting up CloudWatch alarms is essential. These two actions directly address the requirements for secure access and monitoring.

Answer

Store the output in the same S3 bucket as the input to simplify permissions.

Answer

Configure the DataBrew job to use a VPC endpoint for Amazon S3 to ensure private network access.

Answer

Use AWS Glue DataBrew's built-in error handling to automatically retry failed jobs indefinitely.

Question 4

A data engineer is managing an Amazon Redshift cluster that experiences performance degradation during concurrent load and query operations. The engineer notices that some queries are waiting in the queue and that the cluster's workload management (WLM) configuration uses automatic WLM. The engineer needs to ensure that short, high-priority queries are not blocked by long-running queries. Which action should the engineer take?

Accepted Answer

Create a separate WLM queue for short queries and assign a higher priority to that queue, then enable short query acceleration (SQA).. In Amazon Redshift, automatic WLM dynamically manages queues but does not guarantee priority for short queries. Creating a separate queue with higher priority and enabling short query acceleration (SQA) ensures that short queries are routed to a dedicated queue and executed quickly, reducing latency and preventing blocking by long-running queries.

Answer

Enable concurrency scaling to automatically add transient clusters for bursts of queries.

Answer

Increase the number of nodes in the Redshift cluster to provide more compute resources for all queries.

Answer

Switch from automatic WLM to manual WLM and configure a single queue with a high concurrency level.

Question 5

A data engineer is configuring an AWS Glue job to read from an Amazon Kinesis Data Stream and write to Amazon S3 in Parquet format. The engineer needs to ensure the job handles stream data correctly and efficiently. Which two actions should the engineer take? (Choose two.)

Accepted Answer

Enable job bookmarks to track the last processed Kinesis sequence number.. To handle Kinesis stream data efficiently in AWS Glue, enabling job bookmarks tracks the last processed sequence number, preventing reprocessing. Specifying the starting position (e.g., TRIM_HORIZON or LATEST) is required to define where the job begins reading. These two actions ensure correct and efficient stream processing.

Answer

Configure the job to use the 'GlueStreaming' transformation to read from Kinesis.

Answer

Increase the number of shards in the Kinesis stream to match the Glue job's DPU count.

Answer

Set the job's --enable-metrics parameter to true to monitor job progress.

Question 6

A data engineer runs an AWS Glue ETL job that uses a dynamic frame to read from an Amazon S3 bucket and write to an Amazon Redshift table. The job fails with the error 'ConcurrentModificationException: Rate exceeded'. The engineer needs to resolve this error with minimal impact on the job's performance. Which solution should the engineer implement?

Accepted Answer

Enable retry on the job and set a longer timeout for Redshift write operations.. The ConcurrentModificationException with 'Rate exceeded' indicates that the Glue job is making too many concurrent API calls to Redshift, hitting service limits. Enabling job retries with a longer timeout allows the job to handle these transient throttling errors gracefully by retrying the operations after a backoff period. This approach maintains performance while resolving the issue. Other options either don't address the throttling or could worsen it.

Answer

Configure the job to use a larger number of smaller write batches to Redshift.

Answer

Increase the number of DPUs allocated to the job to provide more compute capacity.

Answer

Enable job bookmarks to track processed data and reduce the number of read operations.

Question 7

A data engineer maintains an AWS Glue ETL job that writes Parquet files to Amazon S3. Downstream Amazon Athena queries are returning duplicate rows. The engineer suspects that the Glue job is not handling job retries correctly. Which AWS Glue job configuration should be enabled to prevent duplicate data when a job is retried after a failure?

Accepted Answer

Enable the Job bookmark feature.. Job bookmarks are the AWS Glue feature designed to maintain state about what data has already been processed. When enabled, the job tracks the last processed record and only reads new data on subsequent runs, including retries. This ensures that a failed and retried job does not reprocess the same source records, preventing duplicate rows in the target. Other options do not address data reprocessing.

Answer

Increase the number of workers to improve performance.

Answer

Set the maximum concurrency of the job to 1.

Answer

Configure the job to write to a new S3 prefix on each run.

Question 8

A data engineer is using AWS Database Migration Service (AWS DMS) to migrate an on-premises Oracle database to Amazon Aurora PostgreSQL. During the migration, the engineer notices that some rows are not being replicated to the target. The task logs show no errors, but the target table has fewer rows than the source. The engineer needs to identify the cause of the missing rows. Which action should the engineer take first?

Accepted Answer

Review the DMS task's source filter or transformation rules for any row-level filtering conditions.. DMS row-level filtering can silently exclude rows if misconfigured. The first step is to review the task's filter and transformation rules to see if any conditions are filtering out rows. This is a common cause of missing rows without errors. Checking table mappings or constraints is less likely to cause row-level missing data. DEBUG logging is a follow-up action.

Answer

Enable DMS task logging at the DEBUG level to capture detailed information about each row processed.

Answer

Check the DMS task's table mappings to ensure that the source and target schemas are correctly mapped.

Answer

Verify that the target table's primary key or unique constraints are not causing duplicate rows to be dropped.

Question 9

A data engineer manages an AWS Glue ETL job that processes millions of small JSON files from Amazon S3 and writes Parquet to another S3 location. The job runs for hours and often fails with 'Container killed by YARN for exceeding memory limits' or 'No space left on device' during the shuffle phase. The engineer wants to improve reliability and performance without changing the source data. Which change is MOST effective?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Configure the Glue job to use the groupFiles and groupSize dynamic frame options.. The groupFiles and groupSize options in AWS Glue are designed to reduce the number of small files by grouping multiple input files into a single partition before processing. This results in fewer, larger output files, which improves Athena query performance. Other options such as job bookmarks, DPU scaling, or output mode changes do not address the underlying file compaction requirement.

Answer

Enable AWS Glue job bookmarks to skip previously processed files.

Answer

Set the Glue job's output mode to 'append' instead of 'overwrite'.

Answer

Increase the number of DPUs allocated to the Glue job.

Question 10

A data engineer is using AWS Step Functions to orchestrate a daily ETL pipeline that includes an AWS Glue job, an Amazon EMR step, and an Amazon Redshift stored procedure. The pipeline occasionally fails at the Redshift step with a timeout after 2 hours. The Step Functions state machine has a default timeout of 1 hour for the Redshift task. The engineer needs to ensure the pipeline can run for up to 4 hours without failing due to timeout. Which solution meets this requirement?

Accepted Answer

Modify the Redshift task in the Step Functions state machine to set TimeoutSeconds to 14400 and add a heartbeat timeout.. The Redshift task in Step Functions has a default timeout of 1 hour, which is shorter than the 2-hour runtime of the stored procedure. Setting TimeoutSeconds to 14400 (4 hours) on that specific task allows it to complete. Other options either do not change the task timeout or introduce inappropriate architectures. The correct fix is to adjust the task-level timeout.

Answer

Move the Redshift stored procedure execution to an AWS Lambda function with a 15-minute timeout and invoke it asynchronously.

Answer

Break the Redshift stored procedure into smaller steps and invoke them sequentially with Step Functions Map state.

Answer

Increase the Step Functions state machine timeout to 4 hours and add a retry policy with exponential backoff.

Question 11

A data engineer is responsible for monitoring an AWS Glue ETL job that extracts data from an Amazon RDS for MySQL database, transforms it, and loads it into Amazon S3. The job runs on a schedule every hour. The engineer needs to set up alerts for when the job fails or takes longer than expected. Which TWO actions should the engineer take to meet these requirements? (Choose two.)

Accepted Answer

Enable AWS Glue job metrics and create a CloudWatch alarm on the `glue.driver.aggregate.numFailedTasks` metric to trigger when the value is greater than zero.. To monitor AWS Glue job failures and duration, the engineer should use CloudWatch metrics. The `glue.driver.aggregate.numFailedTasks` metric can trigger an alarm on failures, and the `glue.driver.aggregate.elapsedTime` metric can trigger an alarm when the job runs longer than expected. Both metrics are available when job metrics are enabled.

Answer

Use AWS CloudTrail to log all AWS Glue API calls and create a CloudWatch Logs metric filter for the `JobRunFailed` event.

Answer

Set up an AWS Lambda function that polls the AWS Glue job's status using the `GetJobRun` API every minute and sends an Amazon SNS notification if the status is FAILED.

Answer

Create an Amazon EventBridge rule that matches AWS Glue job state changes to FAILED and triggers an Amazon SNS topic.

Question 12

A data engineer is using AWS Glue to crawl an Amazon S3 data lake. The crawler creates a table in the AWS Glue Data Catalog, but the table schema has all columns defined as strings, even though the underlying data contains integers and timestamps. The engineer needs the crawler to infer the correct data types. Which action should the engineer take?

Accepted Answer

Add a custom classifier to the crawler that specifies the correct data types for each column.. When an AWS Glue crawler misinfers data types, a custom classifier provides explicit schema definitions, ensuring the crawler creates the table with the correct types. This is the recommended approach for consistent and accurate schema inference. Other options either do not address the inference logic or provide only temporary fixes.

Answer

Modify the S3 bucket policy to allow the crawler to read the data in binary format.

Answer

Increase the crawler's sample size by setting a higher value for the 'Sample size' parameter.

Answer

Update the table schema manually in the AWS Glue Data Catalog after the crawler runs.

Question 13

A data engineer is monitoring an AWS Glue job that reads from an Amazon S3 bucket and writes to Amazon Redshift. The job's CloudWatch metrics show that the number of 'glue.driver.aggregate.bytesRead' is much higher than expected, and the job is taking longer than usual. The engineer suspects that the job is reading unnecessary data. Which action should the engineer take to reduce the amount of data read?

Accepted Answer

Partition the source data in Amazon S3 and add partition filters to the Glue job.. The high bytesRead metric indicates the job is scanning more data than necessary. Partitioning the S3 data and applying partition filters in the Glue job enables the job to read only the required partitions, drastically reducing I/O. This is the most effective way to reduce data read and improve job performance.

Answer

Enable job bookmarks to track previously processed data.

Answer

Increase the number of DPUs allocated to the job.

Answer

Convert the source data to a columnar format like Parquet.

Question 14

A company uses Amazon Kinesis Data Streams to ingest real-time clickstream data. The data engineer needs to ensure that all records are processed in the order they arrive and that no data is lost if the consumer application fails. The consumer uses AWS Lambda. Which configuration should the engineer implement to meet these requirements?

Accepted Answer

Use the Kinesis Client Library (KCL) with checkpointing to Amazon DynamoDB and configure the consumer to process records sequentially per shard.. The Kinesis Client Library (KCL) is designed to handle the complexities of consuming from Kinesis Data Streams, including load balancing across shards, checkpointing progress, and ensuring ordered processing per shard. By checkpointing to DynamoDB, the consumer can resume from the last successfully processed record after a failure, preventing data loss. This meets both the ordering and durability requirements.

Answer

Enable enhanced fan-out on the Kinesis stream and use multiple Lambda consumers.

Answer

Configure the Lambda event source mapping with a batch size of 1 and enable function-level error handling with a dead-letter queue.

Answer

Increase the number of shards in the Kinesis stream to improve throughput.

Question 15

A data engineer runs an AWS Glue job that reads JSON files from Amazon S3, transforms them, and writes Parquet to another S3 location. The job fails with the error 'AnalysisException: Unable to infer schema for JSON. It must be specified manually.' The input files are compressed with gzip and have no file extension. What is the MOST likely cause of this failure?

Accepted Answer

AWS Glue cannot infer schema from gzip-compressed JSON files without a file extension.. The failure occurs because AWS Glue's JSON reader relies on the file extension and compression metadata to select the correct input format and codec. Files with no extension and gzip compression cannot be automatically classified, so schema inference fails and the job requires an explicit schema or a properly configured classifier. Providing the schema or renaming files with a .json.gz extension resolves the error.

Answer

The Glue job's output Parquet path is in a different AWS Region than the input.

Answer

The AWS Glue Data Catalog table for the input lacks a partition projection setting.

Answer

The Glue job's IAM role lacks s3:GetObject permission on the input prefix.