Using Partitioned Amazon S3 for Athena Log Queries
A data engineer needs to store semi-structured JSON logs from multiple microservices in a cost-effective manner for later analysis using Amazon Athena. The logs are generated continuously, and the total volume is about 1 TB per day. The data must be queryable within minutes of arrival. Which storage solution is most appropriate?
Quick Answer
The answer is an Amazon S3 bucket with partitioned folders. This is the correct choice because partitioning by time—such as year, month, day, and hour—enables Athena to use partition pruning, drastically reducing the data scanned per query and allowing near-real-time analysis of semi-structured JSON logs within minutes of arrival. On the AWS Certified Data Engineer Associate DEA-C01 exam, this scenario tests your understanding of cost-effective, serverless query architecture for high-volume log ingestion, where the common trap is to over-engineer with a data warehouse or streaming service when S3’s native Athena integration and lifecycle policies already handle 1 TB per day efficiently. Remember the key trade-off: S3 provides the cheapest storage, but without partitioning, Athena would scan the entire bucket, killing performance and cost. Memory tip: “Partition by time to make Athena shine”—if logs arrive continuously, always think hour-level prefixes for immediate queryability.
⚠ Common exam trap
AWS often tests the misconception that a data warehouse (Redshift) or a NoSQL database (DynamoDB) is required for analytical queries on semi-structured data, when in fact S3 with Athena is the most cost-effective and scalable solution for serverless ad-hoc analysis on raw logs.
Answer choices
Why each option matters
Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.
Correct answer & explanation
✓
Amazon S3 bucket with partitioned folders
Amazon S3 with partitioned folders is the most appropriate solution because it provides a cost-effective, scalable storage layer for semi-structured JSON logs, and integrates natively with Amazon Athena for serverless querying. By partitioning the data by time (e.g., year/month/day/hour), Athena can use partition pruning to minimize scanned data, enabling queries within minutes of arrival. S3's low cost per GB and lifecycle policies further optimize storage for the 1 TB/day volume.
Answer analysis
Option-by-option breakdown
For each option: why learners choose it and why it is or isn't the right answer here.
- ✗
Amazon DynamoDB table with JSON attribute
Why it's wrong here
DynamoDB stores items for key-based access patterns and cannot be queried directly by Athena without exporting to S3, adding latency and cost. It would be correct for low-latency application lookups by partition key, not for cost-effective storage of 1 TB daily JSON logs analysed through Athena.
- ✗
Amazon RDS for PostgreSQL table with JSON column
Why it's wrong here
RDS for PostgreSQL requires the JSON logs to be inserted into a table, and Athena cannot query that table directly; data would need extraction to S3 first. RDS suits transactional workloads needing relational integrity, not cheap bulk storage of semi-structured logs queried in place.
- ✓
Amazon S3 bucket with partitioned folders
Why this is correct
Amazon S3 stores JSON at low cost per terabyte and integrates natively with Athena, which queries data in place. Partitioning folders by date or service prunes scanned data, cutting query cost and latency, so logs become queryable within minutes of arrival.
- ✗
Amazon Redshift cluster with JSON ingestion
Why it's wrong here
Redshift requires loading data into provisioned or serverless compute before querying, so JSON logs arriving continuously would need staging and COPY operations, delaying availability beyond minutes. Redshift suits structured analytical workloads with complex joins, not cheap landing of raw semi-structured logs queried in place by Athena.
Quick reference
AWS S3 Storage Class Comparison
| Storage Class | Min Duration | Retrieval | Use Case |
|---|---|---|---|
| S3 Standard | None | Immediate | Frequently accessed data |
| S3 Standard-IA | 30 days | Immediate | Infrequent access, rapid retrieval |
| S3 One Zone-IA | 30 days | Immediate | Non-critical infrequent data |
| S3 Intelligent-Tiering | None | Immediate–hours | Unknown or changing access patterns |
| S3 Glacier Instant | 90 days | Milliseconds | Archive with instant retrieval |
| S3 Glacier Flexible | 90 days | Minutes–hours | Archive, flexible retrieval |
| S3 Glacier Deep Archive | 180 days | Hours | Long-term compliance archive |
Go deeper
Related to this question
About these practice questions
This DEA-C01 question is part of Courseiva's 1,321-question bank — original exam-style content with full explanations and wrong-answer analysis, never real exam questions or exam dumps. Learn why practice questions differ from exam dumps →
Same concept, more angles
2 more ways this is tested on DEA-C01
These questions test the same concept from different angles. Work through them to make sure you can recognise it however the exam phrases it.
Variation 1. A data engineer needs to store semi-structured JSON logs from multiple sources in a centralized data store for querying using SQL. The logs are immutable and need to be retained for 90 days. Which AWS service should be used?
easy- A.Amazon RDS for MySQL.
- B.Amazon DynamoDB.
- ✓ C.Amazon S3 with Amazon Athena.
- D.Amazon ElastiCache for Redis.
Why C: Amazon S3 with Amazon Athena is the correct choice because S3 provides durable, cost-effective storage for immutable semi-structured JSON logs, and Athena enables serverless SQL querying directly against the data in S3 without needing to load or transform it. This combination meets the 90-day retention requirement and supports querying semi-structured data using standard SQL via Athena's built-in JSON SerDe.
Variation 2. A data engineer needs to store semi-structured JSON logs from multiple microservices in a cost-effective manner for ad-hoc querying using SQL. Which AWS service should be used?
medium- ✓ A.Amazon Athena with data in S3
- B.Amazon DynamoDB
- C.Amazon RDS for MySQL
- D.Amazon Kinesis Data Analytics
Why A: Amazon Athena is the correct choice because it allows you to query semi-structured JSON logs stored in S3 directly using standard SQL, without needing to load or transform the data. Athena's schema-on-read approach and pay-per-query pricing make it highly cost-effective for ad-hoc analysis of large volumes of log data, as you only pay for the data scanned during queries.
JA
Written by Johnson Ajibi, MSc IT Security
Senior Network & Security Engineer · founder of Courseiva
This DEA-C01 practice question is part of Courseiva's free Amazon Web Services certification practice question bank. Courseiva provides original exam-style practice questions with explanations, topic-based practice, mock exams, readiness tracking, and study analytics to help learners prepare for the DEA-C01 exam.