A machine learning team needs to create a training dataset by joining two large datasets (10 TB and 5 TB) stored in S3. The join key is 'user_id'. They want to minimize data movement and cost. Which approach should they use?
Trap 1: Launch an Amazon EMR cluster with Spark, read data from S3, perform…
Using Spark on EMR still moves the full 15 TB of data from S3 into the cluster’s ephemeral compute for the shuffle phase, incurring significant data-transfer and processing costs, which fails the stem’s requirement to minimise data movement and cost. This approach is tempting because Spark is a standard engine for large-scale joins, and it would be correct if the team needed complex transformations beyond a simple join, where in-memory processing justifies the data movement.
Trap 2: Use Amazon Athena to run a SQL query joining the two datasets…
Incorrect. Amazon Athena would scan the entire 15 TB of data, leading to high costs (approximately $75 per TB scanned) and potentially slow performance for large joins. It is not optimized for heavy ETL joins and lacks the processing power of Spark.
Trap 3: Load both datasets into Amazon Redshift using COPY commands, then…
Incorrect. Loading both datasets into Redshift first involves significant data movement and extra costs for storage and compute. This approach is not minimal in data movement or cost compared to processing in place with Glue.
- A
Use AWS Glue ETL to read both datasets, join them using Spark DataFrames, and write the result to S3.
Correct. AWS Glue ETL runs serverless Spark jobs that can efficiently join large datasets stored in S3 without moving data out of S3. It is cost-effective because you pay only for the resources consumed during the job execution, and it handles large volumes efficiently.
- B
Launch an Amazon EMR cluster with Spark, read data from S3, perform the join, and write results back to S3.
Why wrong: Using Spark on EMR still moves the full 15 TB of data from S3 into the cluster’s ephemeral compute for the shuffle phase, incurring significant data-transfer and processing costs, which fails the stem’s requirement to minimise data movement and cost. This approach is tempting because Spark is a standard engine for large-scale joins, and it would be correct if the team needed complex transformations beyond a simple join, where in-memory processing justifies the data movement.
- C
Use Amazon Athena to run a SQL query joining the two datasets directly on S3.
Why wrong: Incorrect. Amazon Athena would scan the entire 15 TB of data, leading to high costs (approximately $75 per TB scanned) and potentially slow performance for large joins. It is not optimized for heavy ETL joins and lacks the processing power of Spark.
- D
Load both datasets into Amazon Redshift using COPY commands, then perform the join in Redshift.
Why wrong: Incorrect. Loading both datasets into Redshift first involves significant data movement and extra costs for storage and compute. This approach is not minimal in data movement or cost compared to processing in place with Glue.