A data scientist is performing exploratory data analysis on a large dataset stored in Amazon S3 (100 GB, CSV format, 500 columns). The dataset contains customer transaction records with features such as transaction amount, timestamp, customer ID, and numerous categorical variables (e.g., product category, payment method, location). The scientist wants to understand the distribution of transaction amounts across different product categories and identify any outliers. They have an Amazon SageMaker notebook instance with a ml.t3.medium instance and are using pandas. However, when trying to load the entire dataset into a DataFrame using pd.read_csv('s3://bucket/data.csv'), the notebook crashes with a memory error. Additionally, the scientist suspects that some categorical columns have high cardinality (e.g., product category has thousands of unique values), and there are missing values in several columns. What is the MOST efficient approach to perform the EDA without modifying the original dataset or using additional AWS services? Options: A) Use the SageMaker SDK to launch a parallel processing job with PySpark and read the data into a Spark DataFrame, then compute statistics and visualize with matplotlib. B) Use pandas with chunksize parameter to iterate through the dataset in chunks, compute per-chunk statistics, and aggregate results; for high-cardinality columns, use value_counts() with dropna=False and then plot the top 20 categories. C) Use the S3 Select API to filter rows and columns before loading into pandas, reducing the data size; then use pandas for EDA. D) Use SageMaker Data Wrangler to import the dataset, create a flow to handle missing values and reduce cardinality, and export a sample to the notebook for analysis.

Question

Accepted Answer

Use pandas with chunksize parameter to iterate through the dataset in chunks, compute per-chunk statistics, and aggregate results; for high-cardinality columns, use value_counts() with dropna=False and then plot the top 20 categories.. It addresses the memory issue by reading the data in chunks using the chunksize parameter, allowing processing without loading the entire dataset into memory. It computes per-chunk statistics and aggregates them, which is efficient for EDA. For high-cardinality categorical columns, it uses value_counts() with dropna=False to capture missing values, and then plots the top 20 categories, which is manageable and insightful. This approach stays within pandas and the existing SageMaker notebook without requiring additional services or changing the dataset. Option A is incorrect because launching a separate PySpark job adds complexity and extra cost, and is not the most efficient for an ad-hoc EDA. Option B (S3 Select) can reduce the data volume but cannot natively perform complex aggregations like distribution across categories without pulling all rows; it is more suited for simple filtering. Option C (SageMaker Data Wrangler) is a separate service that requires additional setup and is overkill for this simple EDA task; it also modifies the workflow and is not the most efficient for immediate analysis.

Answer

Use the SageMaker SDK to launch a parallel processing job with PySpark and read the data into a Spark DataFrame, then compute statistics and visualize with matplotlib.

Answer

Use the S3 Select API to filter rows and columns before loading into pandas, reducing the data size; then use pandas for EDA.

Answer

Use SageMaker Data Wrangler to import the dataset, create a flow to handle missing values and reduce cardinality, and export a sample to the notebook for analysis.

Storage Class	Min Duration	Retrieval	Use Case
S3 Standard	None	Immediate	Frequently accessed data
S3 Standard-IA	30 days	Immediate	Infrequent access, rapid retrieval
S3 One Zone-IA	30 days	Immediate	Non-critical infrequent data
S3 Intelligent-Tiering	None	Immediate–hours	Unknown or changing access patterns
S3 Glacier Instant	90 days	Milliseconds	Archive with instant retrieval
S3 Glacier Flexible	90 days	Minutes–hours	Archive, flexible retrieval
S3 Glacier Deep Archive	180 days	Hours	Long-term compliance archive

MLS-C01 Exploratory Data Analysis Practice Question

Why each option matters

Option-by-option breakdown

AWS S3 Storage Class Comparison

Related to this question