Troubleshooting Scenario Questions

Increase the number of DPUs for the Glue job.

Why wrong: More DPUs improve performance but not data consistency.

Use a staging table in Redshift with a transaction to commit.

Why wrong: Adds complexity; not a direct diagnostic step.

Verify the S3 bucket policy allows Firehose to write.

Why wrong: Permission errors would show as 'AccessDenied', not 'InvalidData'.

Increase the buffer size and interval in the Firehose delivery stream.

Why wrong: Buffer settings do not resolve data transformation errors.

Change the data format to CSV in the Firehose configuration.

Why wrong: Changing format may not solve the transformation issue.

Increase the number of DPUs and set 'coalesce(1)' in the script.

Why wrong: coalesce(1) reduces parallelism, increasing memory pressure on a single executor.

Decrease the number of DPUs and increase 'spark.shuffle.partitions'.

Why wrong: Decreasing DPUs reduces available memory, worsening the problem.

Set the 'RedshiftTempDir' parameter to a larger S3 bucket.

Why wrong: RedshiftTempDir is for staging data, not for resolving memory errors in Glue.

Check that the Glue job's VPC and security groups allow outbound traffic to the RDS instance

Why wrong: Network issues cause timeouts, not 'No suitable driver'.

Restart the Glue job with a higher timeout value

Why wrong: Restarting does not fix missing driver; the job will fail again.

Update the IAM role associated with the Glue job to allow 'rds:*' permissions

Why wrong: IAM permissions are for AWS API calls, not for driver loading.

The data location is not registered with Lake Formation.

Why wrong: If the table exists, the location is likely registered.

The S3 bucket policy does not grant the user access.

Why wrong: Lake Formation can grant access without explicit bucket policy.

The user does not have the aws:SourceArn condition in the IAM policy.

Why wrong: This is not required for accessing the Data Catalog.

Enable the 'upsert' feature in the Redshift connection by setting 'update' to true.

Why wrong: Upsert requires a unique key; if not properly configured, duplicates can still occur.

Use partition pruning on the S3 source to reduce the number of files read.

Why wrong: Partition pruning reduces data scanned but does not prevent duplicate writes.

Increase the number of DPUs (Data Processing Units) allocated to the job.

Why wrong: More DPUs improve performance but do not prevent duplicates.

Change the distribution style of both tables to ALL

Why wrong: ALL replicates the entire table to every node, which is inefficient for large tables.

Change the distribution style of both tables to EVEN

Why wrong: EVEN distribution spreads data round-robin, which does not collocate join keys.

Add a sort key on the join column

Why wrong: Sort keys improve data pruning but do not reduce redistribution during joins.

Disable S3 Block Public Access on the bucket.

Why wrong: Block Public Access does not affect IAM role access.

Grant the GlueServiceRole permission to write to the AWS Glue Data Catalog.

Why wrong: Catalog permissions are for metadata, not S3 write.

Ensure the Glue job is in the same VPC as the S3 bucket.

Why wrong: S3 is global; VPC endpoints are for private connectivity, not permissions.

The MySQL database has reached its maximum number of concurrent connections.

Why wrong: The error message specifically mentions transaction log, not connection limits.

The AWS Glue job does not have sufficient permissions to write to the S3 bucket.

Why wrong: The error is about the source database transaction log, not S3 write permissions.

The AWS Glue job is configured with an incorrect 'writeDynamicFrame' method.

Why wrong: The error is not related to the method call but to the source database condition.

Enable automatic backups with a 7-day retention.

Why wrong: Backups do not affect read latency.

Increase the allocated storage to 200 GB.

Why wrong: Increases baseline IOPS but does not directly reduce latency.

Convert the instance to a Multi-AZ deployment.

Why wrong: Multi-AZ improves availability, not read performance.

Poorly chosen sort keys; redefine sort keys

Why wrong: Sort keys affect query performance, not data distribution.

Some nodes are underutilized; add more nodes

Why wrong: Adding nodes does not fix skew; it may mask the issue.

Concurrency scaling is disabled; enable concurrency scaling

Why wrong: Concurrency scaling handles read queries, not storage.

Grant the crawler's IAM user access to the bucket

Why wrong: Crawlers use roles, not users.

Attach a VPC endpoint to the S3 bucket

Why wrong: VPC endpoints help connectivity, not permissions.

Enable S3 default encryption on the bucket

Why wrong: Encryption does not affect access permissions.

The Glue job is running in a VPC without a route to the internet.

Why wrong: Would cause timeout, not access denied.

The S3 bucket policy does not allow the Glue job to read the data.

Why wrong: Error is on write to RDS, not read from S3.

The RDS instance is encrypted with a KMS key that the Glue job cannot access.

Why wrong: Would cause decryption error, not access denied.

Run VACUUM command to reclaim space

Why wrong: Vacuum reclaims deleted space but does not add capacity.

Change distribution style to KEY

Why wrong: Distribution style affects data distribution, not disk space.

Apply compression encoding to tables

Why wrong: Compression reduces storage but does not solve current disk spill.

The table's read capacity is set too low

Why wrong: Throttling is for write requests, not read.

The table's auto scaling is disabled

Why wrong: Auto scaling is enabled as per stem.

The table is using global tables without conflict resolution

Why wrong: Global tables replicate writes but don't cause throttling.