DEA-C01 Data Operations and Support Practice Test 31 — 15 Questions

Question 1

A data engineer is using AWS Glue to process data from an Amazon Kinesis Data Stream. The Glue job is configured to run every 15 minutes and uses job bookmarks to track processed data. Recently, the job started reprocessing old data, leading to duplicate records in the target Amazon S3 bucket. The engineer verifies that the job bookmark is enabled and the job is not being run manually. Which TWO actions should the engineer take to resolve the duplicate processing issue? (Choose two.)

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Check that the Glue job's bookmark key is not being changed between runs, as a changed bookmark key resets the bookmark state.. Duplicate processing with AWS Glue job bookmarks often occurs when the bookmark key changes or when concurrent runs interfere with bookmark updates. Verifying the bookmark key remains constant and preventing concurrent runs with the same bookmark are key steps. These actions ensure the bookmark state is preserved and correctly updated, preventing reprocessing of already-processed data.

Answer

Increase the Kinesis stream's retention period to 168 hours to allow the Glue job more time to process data.

Answer

Configure the Glue job to use the '--job-bookmark-option' parameter set to 'job-bookmark-enable'.

Answer

Verify that the Kinesis stream's shard iterator type is set to LATEST in the Glue job's connection options.

Question 2

A data engineer manages an AWS Glue ETL job that processes JSON files from Amazon S3 and writes to Amazon Redshift. The job fails with the error 'Unable to find a suitable JDBC driver in the classpath'. The engineer has included the Redshift JDBC driver as a job parameter in the Glue job configuration. Which step should the engineer take to resolve the error?

Accepted Answer

Add the JDBC driver as an additional jar in the Glue job's 'Dependent jars path' and ensure the job's IAM role has s3:GetObject permission on the jar location.. To use a custom JDBC driver with AWS Glue, you must upload the driver jar to Amazon S3 and specify its path in the job's 'Dependent jars path' parameter. Additionally, the IAM role associated with the job must have s3:GetObject permission for that jar. This ensures the driver is available in the classpath at runtime, resolving the 'Unable to find a suitable JDBC driver' error.

Answer

Modify the job script to include the JDBC driver URL directly in the 'connection_options' when calling the write_dynamic_frame method.

Answer

Increase the number of workers in the Glue job to ensure the driver is loaded in parallel across all executors.

Answer

Convert the Glue job to use a development endpoint and manually install the JDBC driver on the underlying EC2 instance.

Question 3

A data engineer is building an AWS Glue ETL job that reads from an Amazon S3 bucket containing CSV files and writes to an Amazon Redshift table. The job runs successfully but the Redshift table ends up empty. The engineer checks the AWS Glue job run metrics and sees that the job processed 0 rows. The S3 bucket contains files under the prefix 'data/'. The Glue crawler created a table with the correct schema. What is the MOST likely cause of the empty output?

Accepted Answer

The Glue job's script is filtering out all records because of an incorrect partition predicate or a misconfigured pushdown predicate.. When an AWS Glue job completes successfully but processes zero rows, the issue is typically within the job's transformation logic rather than permissions or connectivity. A pushdown predicate or filter that does not match any data will cause the job to read no records. Checking the script for such filters is the first step.

Answer

The Redshift table is in a different AWS Region than the S3 bucket.

Answer

The Glue job's output is written to a temporary S3 location instead of Redshift because the connection is misconfigured.

Answer

The Glue job's IAM role lacks s3:GetObject permission on the bucket.

Question 4

A data engineer is using AWS Step Functions to orchestrate a series of AWS Glue jobs. The engineer notices that one of the Glue jobs occasionally fails due to a transient network error. The engineer wants to implement a retry mechanism that retries the failed job up to three times with an exponential backoff. Which Step Functions state configuration should the engineer use?

Accepted Answer

Add a 'Retry' field with 'MaxAttempts': 3, 'IntervalSeconds': 2, and 'BackoffRate': 2.0 to the task state.. The 'Retry' field in Step Functions is designed to handle transient errors by retrying the task with optional exponential backoff. Specifying 'MaxAttempts': 3, 'IntervalSeconds': 2, and 'BackoffRate': 2.0 achieves the desired retry behavior. Other options either require complex workarounds or do not support exponential backoff, making the 'Retry' field the correct choice.

Answer

Configure the Glue job itself to retry on failure by setting the 'MaxRetries' parameter in the job definition.

Answer

Use a 'Parallel' state to run the Glue job multiple times simultaneously and take the first successful result.

Answer

Add a 'Catch' field with 'ErrorEquals': ['States.TaskFailed'] and a 'Next' state that increments a counter and loops back.

Question 5

A data engineer is using AWS Step Functions to orchestrate an ETL workflow that includes an AWS Glue job. The Glue job occasionally fails due to transient issues, such as network timeouts. The engineer wants the Step Functions state machine to automatically retry the Glue job up to three times with exponential backoff before failing the workflow. Which Step Functions state configuration should the engineer use?

Accepted Answer

Use a Retry field with MaxAttempts: 3 and BackoffRate: 2.0 in the task state that invokes the Glue job.. The Retry field in AWS Step Functions is specifically designed to handle retries on task failures, including automatic backoff. By specifying MaxAttempts and BackoffRate, the state machine will retry the Glue job invocation with exponential backoff, meeting the requirement without custom logic.

Answer

Configure the Glue job to have a retry policy in its job definition, such as MaxRetries: 3.

Answer

Use a Catch field with ErrorEquals: ["States.ALL"] and Next: "RetryState" to redirect to a retry state.

Answer

Set the TimeoutSeconds and HeartbeatSeconds fields in the task state to trigger a retry.

Question 6

A data engineer is investigating intermittent failures in an AWS Step Functions state machine that orchestrates a nightly ETL workflow. The state machine invokes an AWS Glue job, then an Amazon EMR step, then an AWS Lambda function. Occasionally a task fails transiently and the entire workflow stops instead of retrying. The engineer needs the workflow to automatically retry failed tasks with exponential backoff before alerting. What should the engineer do?

Accepted Answer

Add a Retry field to the relevant state with ErrorEquals, IntervalSeconds, MaxAttempts, and BackoffRate values.. Step Functions state machines are defined in Amazon States Language, and each state can include a Retry field that matches specific error names and retries with an interval that grows by a BackoffRate multiplier. This declarative approach centralizes retry logic in the orchestrator, so transient failures in Glue, EMR, or Lambda are retried automatically before any alert is raised.

Answer

Set the state machine's execution role to include the 'states:Retry' IAM action and re-run the workflow.

Answer

Configure the Glue job, EMR step, and Lambda function each with their own internal retry logic and remove error handling from the state machine.

Answer

Enable the state machine's 'Retry on failure' setting in the Amazon CloudWatch console.

Question 7

A data engineer is monitoring an AWS Glue job that reads from an Amazon S3 bucket and writes to Amazon Redshift. The job has been running for 2 hours, which is longer than usual. The engineer checks the Glue job's metrics and sees that the number of active executors is high, but the job is not making progress. The engineer suspects a data skew issue. Which action should the engineer take to diagnose and mitigate the skew?

Accepted Answer

Use the Spark UI to inspect the stage details and identify tasks that are taking significantly longer than others.. Data skew in Spark jobs manifests as a few tasks taking much longer than others. The Spark UI is the primary tool to diagnose skew by showing task-level metrics. Once identified, mitigation strategies include salting keys, using broadcast joins, or repartitioning. The correct action here is to use the Spark UI to inspect stage details.

Answer

Increase the number of DPUs for the Glue job to add more executors and reduce the impact of skew.

Answer

Enable AWS Glue job metrics and examine the 'glue.driver.aggregate.bytesRead' metric to identify skewed partitions.

Answer

Modify the Glue job to use the 'groupFiles' option to combine small files and improve parallelism.

Question 8

A company stores raw clickstream data in an Amazon S3 bucket and uses AWS Glue crawlers to populate the AWS Glue Data Catalog. Analysts report that new partitions are not appearing in Amazon Athena queries even though new date-based folders exist in S3. The crawler runs successfully each night. What is the MOST likely cause?

Accepted Answer

The new folders do not follow the Hive-style partition naming convention (for example, year=/month=/day=) that the crawler and Athena expect.. Glue crawlers infer partitions from Hive-style key=value directory structures. When new folders use a plain date hierarchy without partition keys, the crawler treats them as ordinary prefixes and does not register partitions, so Athena cannot see the new data until the folder layout is corrected or partitions are added manually.

Answer

The crawler's 'Update the table definition in the Data Catalog' behavior is set to 'Create new partitions only' but the table was created manually without a partition key.

Answer

The S3 bucket has versioning enabled, which prevents the crawler from detecting new prefixes.

Answer

The crawler's output is configured to a different database, or the crawler has an exclude pattern that filters out the new date folders.

Question 9

A data engineer needs to ensure that an AWS Glue job has access to an Amazon RDS database in a private subnet. The Glue job will run in a VPC and requires a security group and subnet configuration. Which combination of steps should the engineer take?

Accepted Answer

Configure the Glue job with a VPC connection, specify a subnet and security group, and ensure the security group allows outbound traffic to the RDS database's security group.. For an AWS Glue job to access an Amazon RDS database in a private subnet, the job must be configured with a VPC connection that specifies the subnet and security group. The security group must allow outbound traffic to the RDS database's security group on the database port. This ensures network connectivity while maintaining security.

Answer

Use AWS Glue Studio to create a connection to RDS, which automatically configures the necessary VPC settings and security groups.

Answer

Attach an IAM role to the Glue job with permissions to access RDS, and configure the RDS database to be publicly accessible.

Answer

Create a VPC endpoint for AWS Glue, configure the Glue job with a VPC connection, and attach a security group that allows outbound traffic to the RDS database.

Question 10

A data engineer maintains an Amazon Kinesis Data Streams pipeline that feeds an AWS Lambda consumer. During traffic spikes, the Lambda function is throttled and records are reprocessed, causing duplicate entries in the downstream Amazon S3 sink. The engineer needs to reduce duplicates with the LEAST code change. What should the engineer do?

Accepted Answer

Make the Lambda function idempotent by using the Kinesis sequence number as a deduplication key before writing to S3.. Duplicate delivery in Kinesis-to-Lambda pipelines is expected because Lambda retries batches after throttling or errors, and Kinesis guarantees at-least-once delivery. Making the consumer idempotent by keying on the Kinesis sequence number ensures that reprocessed records are detected and not written twice, which requires only a small change inside the function.

Answer

Configure the event source mapping's StartingPosition to LATEST and enable enhanced fan-out on the stream.

Answer

Increase the Lambda function's reserved concurrency and set the event source mapping's MaximumRetryAttempts to a low value.

Answer

Enable the 'ReportBatchItemFailures' feature on the event source mapping and return the failed sequence numbers so only failed records are retried.

Question 11

A data engineer manages an AWS Glue ETL job that reads JSON files from Amazon S3 and writes to Amazon Redshift. The job recently started failing with the error: 'Unable to find catalog table' when trying to access a table in the AWS Glue Data Catalog. The engineer confirms that the table exists in the Data Catalog and that the IAM role used by the job has glue:GetTable permissions. What is the most likely cause of this error?

Accepted Answer

The table name in the Glue job script is misspelled or uses incorrect casing.. The error indicates that the Glue job cannot locate the specified table in the Data Catalog. Since the table exists and the IAM role has the necessary glue:GetTable permission, the most likely cause is a mismatch between the table name referenced in the job script and the actual table name in the catalog. The Data Catalog is case-sensitive, so even a minor typo or casing difference will result in this error.

Answer

The Glue job is running in a different AWS region than the Data Catalog.

Answer

The Glue job's IAM role lacks permissions to access the S3 bucket containing the table's data.

Answer

The Glue job's IAM role lacks permissions to access the AWS Glue Data Catalog.

Question 12

A data engineer is troubleshooting an Amazon Redshift cluster where nightly COPY loads from Amazon S3 are intermittently slow and sometimes fail with 'S3ServiceException' errors. The engineer suspects the cluster's network configuration and load design are contributing. Which TWO actions should the engineer take to improve load performance and reliability? (Choose two.)

Accepted Answer

Split large input files into multiple files sized roughly between 1 MB and 1 GB and load them in parallel.. Redshift COPY performance depends on parallelizing reads across slices and on a stable network path to S3. Splitting input into many files sized between 1 MB and 1 GB lets slices read concurrently, while enhanced VPC routing keeps S3 traffic inside the VPC and avoids public internet variability that produces intermittent S3ServiceException errors.

Answer

Convert all input files to a single large gzip file to minimize the number of S3 GET requests during COPY.

Answer

Increase the cluster's node count and change the distribution style of all tables to EVEN before every load.

Answer

Disable compression on the input files so Redshift can parse records faster.

Question 13

A data engineer manages an AWS Glue ETL job that processes JSON files from an S3 bucket and writes Parquet to another bucket. The job uses a Glue DynamicFrame with a specified schema. During execution, the job fails with the error: 'AnalysisException: cannot resolve column 'transaction_id' given input columns: [txn_id, amount, timestamp]'. The source data has a column named 'txn_id', but the Glue job's script references 'transaction_id'. The job's catalog table for the source points to the correct S3 location and has the correct schema. What is the most likely cause of this error?

Accepted Answer

The Glue job's script uses a hardcoded schema that does not match the actual data columns.. The error occurs because the Glue script references a column that does not exist in the source data. Even though the Glue Data Catalog is correct, the script may have been written with a hardcoded schema or mapping that expects 'transaction_id' instead of 'txn_id'. To resolve, the engineer should update the script to use the correct column name or apply a mapping to rename 'txn_id' to 'transaction_id'.

Answer

The Glue job's IAM role lacks permission to read the S3 bucket, resulting in an empty DynamicFrame.

Answer

The Glue Data Catalog table for the source has an incorrect schema definition.

Answer

The S3 bucket contains files with inconsistent schemas, causing Glue to infer a different column name.

Question 14

A data engineer maintains an AWS Glue ETL job that processes JSON files from Amazon S3 and writes Parquet to another S3 location. The job has been running successfully for months. Recently, the job started failing intermittently with the error 'Unable to infer schema for JSON'. The engineer confirms the source bucket contains valid JSON files. Which action should the engineer take to resolve the failure?

Accepted Answer

Define an explicit schema using a Glue Data Catalog table or a DynamicFrame with a specified schema, instead of relying on schema inference.. The error 'Unable to infer schema for JSON' occurs when AWS Glue cannot automatically determine a consistent schema from the source files. This often happens when JSON records have varying fields or data types. Defining an explicit schema through the Glue Data Catalog or programmatically ensures the job can parse the data without relying on inference, resolving the failure while maintaining data integrity.

Answer

Enable job bookmarks to track previously processed files and avoid reprocessing.

Answer

Increase the number of AWS Glue DPUs allocated to the job to handle larger file sizes.

Answer

Convert the source JSON files to CSV format before running the job, then update the job to read CSV.

Question 15

A data engineer manages an AWS Glue ETL job that processes millions of small JSON files in Amazon S3. The job is slow and often fails with an OutOfMemory error on the driver. The engineer wants to improve performance without changing the output format. Which solution should the engineer implement?

Accepted Answer

Use AWS Glue's groupFiles and groupSize options to combine small files into larger groups.. Grouping small files with groupFiles and groupSize is the most effective way to handle millions of small files in AWS Glue. It reduces the number of input splits, lowers driver memory pressure, and improves overall job performance. Other options either do not address the driver memory issue or require changing the output format.

Answer

Enable job bookmarks to track processed files and skip already-processed data.

Answer

Convert the JSON files to Parquet format using an AWS Glue crawler before running the ETL job.

Answer

Increase the number of DPUs allocated to the job to provide more memory per executor.