DEA-C01 Data Operations and Support Practice Question

Answer analysis

Option-by-option breakdown

For each option: why learners choose it and why it is or isn't the right answer here.

• ✗

  Switch from using Apache Spark to Python shell.

  Why it's wrong here

  Python shell jobs run single-node, single-driver scripts without Spark's distributed executors, so a 10 GB dataset exceeds one worker's memory rather than relieving pressure. It is tempting because Python shell suits small, lightweight transformation or orchestration tasks where distributed processing overhead is unnecessary.

• ✗

  Repartition the data into more partitions within the job.

  Why it's wrong here

  Repartitioning redistributes records across executors but does not reduce the memory consumed per partition or per executor, so skew-driven OOM persists. It is tempting because repartitioning genuinely resolves skewed partitions and improves parallelism when a few oversized partitions, not total memory, cause the failure.

• ✗

  Change the job type from Python to Java.

  Why it's wrong here

  Changing the language does not alter Glue's executor memory allocation; the JVM heap per worker remains the same, so the OOM recurs. It is tempting because Java can outperform Python for CPU-bound Spark transformations, making it a valid optimisation when compute, not memory, is the bottleneck.

• ✓

  Increase the number of DPUs allocated to the job.

  Why this is correct

  Insufficient memory in AWS Glue typically stems from insufficient worker capacity for the 10 GB dataset. Increasing the number of DPUs adds executors and memory, directly addressing the constraint before considering other tuning such as partitioning or worker type changes.

Go deeper

Related to this question