A company collects data from multiple sources: IoT sensor streams, social media feeds, and CSV files from legacy systems. They want to store all this data in its original format without any transformation, so that data scientists can later apply machine learning models or run ad-hoc queries. Which data storage pattern best describes this approach?
Trap 1: Data warehouse
A data warehouse stores data that has been extracted, transformed, and loaded (ETL), organizing it into structured star schemas for business intelligence and reporting. This schema-on-write approach requires defining the schema before loading, making it impossible to preserve raw data in its original form; instead, warehouse data is curated and aggregated to answer known analytical questions.
Trap 2: Relational database
A relational database enforces a rigid schema and ACID-compliant transactions, which are essential for OLTP applications like order processing but ill-suited for ingesting raw IoT streams. Storing unstructured or semi-structured sensor data would force a transformation into tables, losing fidelity and creating scalability bottlenecks for high-velocity, heterogeneous data.
Trap 3: Data mart
A data mart is a logically separated subset of a data warehouse, tailored to a single business unit or process, such as sales or finance. It typically contains aggregated and pre-transformed data, so it inherits the warehouse's schema-on-write constraints and cannot accommodate raw, unmodified IoT streams; its scope is too narrow and its data too curated for exploratory big data analytics.
- A
Data warehouse
Why wrong: A data warehouse stores data that has been extracted, transformed, and loaded (ETL), organizing it into structured star schemas for business intelligence and reporting. This schema-on-write approach requires defining the schema before loading, making it impossible to preserve raw data in its original form; instead, warehouse data is curated and aggregated to answer known analytical questions.
- B
Data lake
A data lake is a centralized repository that stores raw data in its native format, from IoT sensor streams to structured files, without requiring a predefined schema. It employs schema-on-read, so data scientists can explore and run ad-hoc analytics before defining structure. This makes it ideal for diverse, high-volume streaming data where format and meaning may evolve over time.
- C
Relational database
Why wrong: A relational database enforces a rigid schema and ACID-compliant transactions, which are essential for OLTP applications like order processing but ill-suited for ingesting raw IoT streams. Storing unstructured or semi-structured sensor data would force a transformation into tables, losing fidelity and creating scalability bottlenecks for high-velocity, heterogeneous data.
- D
Data mart
Why wrong: A data mart is a logically separated subset of a data warehouse, tailored to a single business unit or process, such as sales or finance. It typically contains aggregated and pre-transformed data, so it inherits the warehouse's schema-on-write constraints and cannot accommodate raw, unmodified IoT streams; its scope is too narrow and its data too curated for exploratory big data analytics.