A company is using AWS Glue to run ETL jobs that transform data from Amazon S3 to Amazon Redshift. The jobs are failing intermittently with 'Out of Memory' errors. The team wants to resolve this issue without increasing costs significantly. Which TWO actions should the team take?
Trap 1: Use DynamicFrame instead of Spark DataFrame for transformations.
DynamicFrame is built on Spark; memory usage is similar.
Trap 2: Increase the number of workers to maximum allowed.
More workers increase parallelism and cost but each worker still has limited memory; may not solve OOM if memory per worker is insufficient.
Trap 3: Switch from a Spark job to a Python shell job.
Python shell jobs are for lightweight processing; not suitable for large transformations.
- A
Increase the Spark memory overhead parameter in the Glue job configuration.
Allocates more memory per worker for Spark processing, reducing OOM errors.
- B
Use DynamicFrame instead of Spark DataFrame for transformations.
Why wrong: DynamicFrame is built on Spark; memory usage is similar.
- C
Increase the number of workers to maximum allowed.
Why wrong: More workers increase parallelism and cost but each worker still has limited memory; may not solve OOM if memory per worker is insufficient.
- D
Switch from a Spark job to a Python shell job.
Why wrong: Python shell jobs are for lightweight processing; not suitable for large transformations.
- E
Change the worker type from 'G.1x' to 'G.2x' to double memory per worker.
Doubles memory per worker, addressing OOM with moderate cost increase.