DEA-C01 Data Operations and Support Practice Question

Exhibit

Refer to the exhibit.

CloudWatch Logs log group: /aws/glue/jobs/error
Log stream: job-run-12345
Log event:
2024-01-15T10:30:45.000Z ERROR [Executor task launch worker for task 0.0 in stage 0.0 (TID 0)] : java.lang.OutOfMemoryError: Java heap space
2024-01-15T10:30:45.001Z ERROR [Executor task launch worker for task 0.0 in stage 0.0 (TID 0)] : at org.apache.spark.sql.execution.datasources.FileFormatWriter$WriteTask.execute(FileFormatWriter.scala:247)

A data engineer is running an AWS Glue ETL job that converts CSV files to Parquet. The job fails with the error shown in the exhibit. The input files are about 500 MB each. The job uses 5 workers of type G.1X (16 GB memory each). What is the MOST likely cause?

Answer choices

Why each option matters

Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.

Correct answer & explanation

✓

The output Parquet file size is too large for the executor memory

The error indicates an out-of-memory (OOM) error during the write phase. When converting large CSV files to Parquet, Spark must buffer the output data in memory before writing. With 500 MB input files and 5 workers of G.1X (16 GB each), the executor memory is limited. If the output Parquet file or partition is too large, it may exceed the executor's memory, causing OOM. Option B (data skew) could also cause OOM, but typically during shuffle, not write. Options C and D are unrelated to write OOM issues.

Answer analysis

Option-by-option breakdown

For each option: why learners choose it and why it is or isn't the right answer here.

✓
The output Parquet file size is too large for the executor memory
Why this is correct
Writing a large file requires memory proportional to file size; splitting into smaller files can help.
✗
The data is highly skewed causing a single partition to receive too much data
Why it's wrong here
Skew would cause OOM in shuffle stage, not in write stage.
✗
The Spark driver does not have enough memory to handle the schema inference
Why it's wrong here
Schema inference happens on driver; error is on executor.
✗
The input CSV files are corrupt or have inconsistent schema
Why it's wrong here
Corrupt files would cause parse errors, not OOM.

Go deeper

DEA-C01 Data Operations and Support Practice Question

Exhibit

Why each option matters

Option-by-option breakdown

Related to this question