MLA-C01 Data Preparation for Machine Learning Practice Question

A team is using Amazon SageMaker Processing for data preprocessing. They have a Parquet dataset in Amazon S3. Which configuration will provide the most efficient reading of the dataset during processing?

⚠ Common exam trap

AWS often tests the misconception that many small files improve parallelism, but in distributed systems like Spark on SageMaker, small files increase S3 API call overhead and scheduler latency, making larger Parquet files (e.g., 128 MB–1 GB) far more efficient for reading.

Answer choices

Why each option matters

Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.

Correct answer & explanation

✓

Read the Parquet files directly using SparkSession.read.parquet

SageMaker Processing natively integrates with Apache Spark, and reading Parquet files directly via `SparkSession.read.parquet` leverages columnar storage, predicate pushdown, and compression (e.g., Snappy) to minimize I/O and deserialization overhead. This approach is far more efficient than text-based or format-conversion methods, as Parquet is optimized for analytical workloads and preserves schema information.

Answer analysis

Option-by-option breakdown

For each option: why learners choose it and why it is or isn't the right answer here.

✗
Read the Parquet files as text using SparkContext.textFile
Why it's wrong here
textFile reads raw bytes line by line, so Parquet's binary structure and footer metadata are never parsed, yielding corrupt records rather than columns. It is tempting for plain-text or log ingestion where no schema exists; that scenario, not columnar analytics, is where textFile belongs.
✗
Split the dataset into many small Parquet files (e.g., 1 MB each)
Why it's wrong here
One-megabyte objects multiply S3 GET requests and per-file overhead, and each Spark task opens its own reader, so throughput collapses. Small files suit low-latency appends or streaming sinks; efficient Parquet scanning needs few large row groups, not fragmentation.
✗
Convert the Parquet files to CSV before processing
Why it's wrong here
Parquet's columnar encoding and predicate pushdown are lost once converted to CSV, forcing a full row-wise scan and inflating S3 transfer volume. Conversion is tempting when tooling lacks a Parquet reader, and would suit exporting a small subset for spreadsheet inspection, not efficient large-scale processing.
✓
Read the Parquet files directly using SparkSession.read.parquet
Why this is correct
SparkSession.read.parquet reads Parquet's columnar, compressed format directly, enabling predicate pushdown and column pruning so SageMaker Processing scans only needed columns and row groups from Amazon S3, avoiding full-dataset deserialisation and delivering the most efficient read.

Quick reference

AWS S3 Storage Class Comparison

Storage Class	Min Duration	Retrieval	Use Case
S3 Standard	None	Immediate	Frequently accessed data
S3 Standard-IA	30 days	Immediate	Infrequent access, rapid retrieval
S3 One Zone-IA	30 days	Immediate	Non-critical infrequent data
S3 Intelligent-Tiering	None	Immediate–hours	Unknown or changing access patterns
S3 Glacier Instant	90 days	Milliseconds	Archive with instant retrieval
S3 Glacier Flexible	90 days	Minutes–hours	Archive, flexible retrieval
S3 Glacier Deep Archive	180 days	Hours	Long-term compliance archive

Go deeper

MLA-C01 Data Preparation for Machine Learning Practice Question

Why each option matters

Option-by-option breakdown

AWS S3 Storage Class Comparison

Related to this question