# Big data

> Source: Courseiva IT Certification Glossary — https://courseiva.com/glossary/big-data

## Quick definition

Big data is a term for very large sets of information that are too big or too complex for regular computers to handle. It includes data from many sources like social media, sensors, and business transactions. Special tools like distributed storage and parallel processing are needed to work with big data. The goal is to find patterns and insights that can help make better decisions.

## Simple meaning

Imagine you have a small notebook where you write down your daily expenses. You can easily flip through it, add up numbers, and find out how much you spent last month. That is like working with normal, small data. Now imagine instead that you have a warehouse filled with thousands of notebooks, each containing millions of pages of handwritten numbers, text, and pictures from different sources. Some pages are in different languages, some are blurry, and new notebooks are being added every second. Trying to find a single pattern or answer in that pile would be impossible with just your eyes and hands. That is big data. Big data is not just about size, it is also about the variety of information and how fast it comes in. For example, every time you like a post on social media, your phone sends a tiny piece of data to a giant server. Multiply that by billions of people and you get an ocean of data. Traditional databases, like the ones used by a small shop to track inventory, are not built to swim in such an ocean. They would crash or take years to process. Instead, big data uses special systems that break the work into tiny pieces and spread it across hundreds or thousands of computers that work together. This is called distributed computing. Think of it like organizing a huge library not with one librarian, but with a team of thousands, each responsible for a single shelf. When someone asks for a book, all the librarians search their shelves at the same time and report back. This makes it fast and possible. Big data helps companies understand customer behavior, predict weather patterns, detect fraud, and even recommend the next video you should watch. In IT certification exams, big data is a foundational concept that explains why new tools like Hadoop, Spark, and cloud data lakes were created.

## Technical definition

Big data is formally defined by three main characteristics, known as the three Vs: Volume, Velocity, and Variety. Volume refers to the sheer scale of data, often measured in terabytes, petabytes, or even exabytes. Traditional relational database management systems (RDBMS) are not designed to handle such quantities efficiently due to architectural limitations in storage and query processing. Velocity describes the speed at which data is generated and needs to be processed. For example, real-time sensor data from IoT devices or stock market feeds can generate millions of records per second. Variety covers the different forms of data: structured data (like tables in a database), semi-structured data (like JSON files or XML), and unstructured data (like text documents, images, videos, and social media posts). Later, two more Vs were added: Veracity, which refers to the quality and trustworthiness of the data, and Value, which is the business insight gained from analyzing the data.

Technically, big data solutions rely on distributed storage and parallel processing frameworks. Apache Hadoop is a foundational technology that uses the Hadoop Distributed File System (HDFS) to split files into blocks and replicate them across multiple nodes in a cluster. This provides fault tolerance and high throughput. The processing engine in Hadoop is MapReduce, which breaks a task into map and reduce phases that can run in parallel across many servers. However, MapReduce is disk-based and can be slow for iterative tasks, so newer engines like Apache Spark use in-memory processing through Resilient Distributed Datasets (RDDs) to achieve much faster performance. Data ingestion tools like Apache Kafka handle high-velocity streams by acting as a distributed messaging system. Data storage can also be implemented using NoSQL databases like Apache HBase (column-oriented) or MongoDB (document-oriented) which scale horizontally across commodity hardware.

In the cloud, big data services abstract much of the underlying complexity. For example, Microsoft Azure provides Azure Data Lake Storage for scalable data lakes, Azure Synapse Analytics for data warehousing and analytics, and Azure HDInsight for managed Hadoop and Spark clusters. The DP-900 exam (Microsoft Azure Data Fundamentals) covers these concepts at a foundational level. Candidates must understand the difference between structured and unstructured data, the role of batch vs. stream processing, and how data lakes differ from data warehouses. Big data environments typically follow a lambda architecture that combines batch processing for historical data with stream processing for real-time data. A key protocols include OAuth 2.0 for authentication to data sources and ODBC/JDBC for connecting analytics tools. Data governance is also critical, involving metadata management via tools like Apache Atlas and data lineage tracking. Overall, big data is not a single technology but an ecosystem of tools and practices designed to extract value from massive, diverse, and fast-moving data.

## Real-life example

Think about a giant public library in a big city. Every day, thousands of people visit, borrow books, return books, ask questions, and use the computers. The library keeps records of every single transaction: who borrowed what, when, what topics people searched for, which sections are most popular, and even how long people stay. Over time, this creates a huge collection of data about reading habits and preferences. Now imagine the library wants to decide which new books to buy. Looking at each record manually would take years. Instead, they use a system where all librarians are connected through a network. Every time a book is checked out, that information is immediately recorded in a central system. The system can instantly calculate that mystery novels are checked out three times more often than history books, and that most people borrow books on rainy days. That is similar to how big data works in real companies. A online streaming service like Netflix collects data every time you press play, pause, or stop. They also know what time of day you watch, what device you use, and what you searched for. When millions of users do this every second, the company can predict which shows to recommend to you personally. They can even decide which new series to produce based on what topics are trending. In retail, stores use big data to analyze customer purchases in real time. If a storm is coming, the system might detect that people are buying more flashlights and batteries, and automatically restock those items in stores. This kind of analysis would be impossible with a simple spreadsheet. Big data allows organizations to see patterns and make decisions that are faster, smarter, and more accurate than ever before. For IT professionals, understanding the principles behind big data is essential because most modern applications, from mobile games to banking systems, handle data at a scale that requires these specialized tools.

## Why it matters

Big data matters because it fundamentally changes how organizations make decisions. In the past, decisions were often based on intuition, limited surveys, or small samples of data. For example, a store manager might order more umbrellas because spring is coming, but they might not know that customers in a specific neighborhood prefer blue over red. With big data analytics, that same manager can analyze sales data from thousands of stores, weather forecasts, and even social media mentions to predict exactly what products will be needed, in what color, and in which location. This level of precision reduces waste, increases sales, and improves customer satisfaction. In healthcare, big data helps doctors identify disease patterns by analyzing millions of patient records, leading to earlier diagnosis and better treatments. In finance, big data is used to detect fraudulent transactions in real time by comparing each transaction against a huge database of known fraud patterns. For governments, big data helps in traffic management, disaster response, and public health monitoring. For IT professionals specifically, big data is a core skill because the infrastructure that supports big data is complex and requires specialized knowledge. Whether you are deploying a data lake in the cloud, writing ETL pipelines, or tuning a Spark job, you need to understand the underlying principles. The DP-900 exam is an entry-level certification that introduces these concepts. It teaches candidates about the different types of data processing, storage options, and analytics tools. Without a grasp of big data, an IT professional may struggle to design systems that are scalable, cost-effective, and capable of delivering insights. As the amount of data in the world continues to grow exponentially, the ability to work with big data is no longer a niche skill but a fundamental requirement for many roles in IT, data science, and business intelligence.

## Why it matters in exams

In the context of the Microsoft DP-900 exam (Azure Data Fundamentals), big data is a core concept that appears across several exam objectives. The exam tests your understanding of core data concepts, including the differences between structured, semi-structured, and unstructured data, as well as the characteristics of big data (the five Vs). You are expected to know that big data workloads often require distributed processing and that Azure offers specific services for these scenarios, such as Azure Data Lake Storage, Azure Synapse Analytics, and Azure HDInsight. The exam also covers batch processing versus stream processing, with big data often involving real-time streams. For example, you might see a question asking which Azure service is best for processing large volumes of unstructured data in real time. The correct answer would often reference Azure Stream Analytics or Apache Spark on HDInsight. Another common question pattern is identifying the correct storage solution for big data, where Azure Data Lake Storage Gen2 is preferred for its hierarchical namespace and compatibility with Hadoop. You also need to understand data pipelines and how big data fits into the modern data warehouse architecture, including the use of data lakes for storing raw data and data warehouses for structured analytics. The DP-900 exam includes scenario-based questions where you must recommend a data solution given volume, velocity, and variety requirements. For instance, if a company needs to store petabytes of sensor data arriving every second and then run analytics, you would need to recommend a combination of Azure Data Lake Storage for storage and Azure Stream Analytics for processing. The exam also touches on data governance and security in big data contexts, such as using Azure Purview for data cataloging. Although DP-900 is a foundational exam, a solid understanding of big data terminology and cloud services is crucial because it lays the groundwork for more advanced certifications like DP-203 (Data Engineering on Azure) and DP-100 (Data Science). You will not be asked to write code or configure clusters, but you must be able to read a scenario and choose the right service or concept. Big data questions in DP-900 are typically straightforward but require you to distinguish between batch and real-time, structured and unstructured, and relational and non-relational. The exam traps often involve confusing data lakes with data warehouses, or misunderstanding which processing model fits a given velocity requirement.

## How it appears in exam questions

In the DP-900 exam, big data topics appear in multiple-choice, multiple-select, and scenario-based questions. A typical scenario question might describe a company that collects millions of social media posts every minute. The question will ask which Azure service is best for storing this data. Option A might be Azure SQL Database, option B is Azure Data Lake Storage, option C is Azure Cosmos DB, and option D is Azure Table Storage. The correct answer is Azure Data Lake Storage because it is designed for massive volumes of unstructured data. Another question type asks you to identify the correct characteristic of big data. For example, which V describes the fact that data comes from many different sources like text, images, and video? The answer is Variety. You might also see a question about processing modes. If a scenario says data must be processed as soon as it arrives, the required mode is stream processing. If the scenario says data is processed in large chunks at the end of the day, it is batch processing. There are also questions about the components of big data processing. For instance, what is the role of Apache Spark in big data? The answer is in-memory data processing for faster analytics. Some questions will test your understanding of scaling. For example, which type of scaling is used by big data storage systems like HDFS? The answer is horizontal scaling (adding more nodes) as opposed to vertical scaling (upgrading a single server). Another common pattern is comparing big data to traditional data. A question might ask why a relational database is not suitable for big data workloads. The answer would highlight limitations in handling unstructured data and the lack of horizontal scalability. The exam may also include questions about data lake vs. data warehouse. For big data, the recommended approach is often a data lake because it can store raw data in its original format. You should be prepared to match Azure services to big data use cases, such as Azure HDInsight for managed Hadoop/Spark, Azure Synapse Analytics for data warehousing and big data analytics, and Azure Data Factory for orchestration of data pipelines. The exam does not ask for deep technical commands or configuration details, but it does require conceptual understanding and the ability to apply that understanding to practical business scenarios. Always read the scenario carefully and identify the key constraints: volume, velocity, variety, and whether the data is structured or unstructured.

## Example scenario

You are a data analyst at a large retail company called ShopWorld. The company has 10,000 stores across the country, and each store has hundreds of sensors that track inventory, customer traffic, and weather conditions. Every time a customer walks into a store, the sensor records the time, the entrance used, and the estimated age and gender of the customer. The point-of-sale system records every purchase with the item, quantity, price, and payment method. The company runs a popular website and mobile app that collect data on what customers browse, what they add to their cart, and what they ultimately buy. This results in billions of data points being generated every day. The CEO wants to understand why sales of winter jackets dropped by 15% last month compared to the same month last year. With older systems, you would have to take a sample of data from a few stores, export it to a spreadsheet, and manually look for trends. That might take weeks and the result could be inaccurate. With big data, you can use a cloud-based data lake that stores all this raw data. You then run a distributed query using Azure Synapse Analytics that processes all 10,000 stores data in minutes. The analysis reveals that in stores located in warmer regions, jackets were overpriced compared to local competitor prices. The app showed that customers who viewed jackets on the website but did not buy often complained about the lack of size availability. By combining inventory data, you find that smaller sizes were out of stock in many stores. The big data system can also process real-time data: as soon as a new shipment arrives, the inventory counts are updated, and the system can automatically suggest moving excess inventory from warm stores to cold stores. This kind of insight is only possible because big data tools can handle the volume, velocity, and variety of the data. In the DP-900 exam, you might be asked to identify that this scenario requires a data lake for storage and a distributed analytics service like Azure Synapse for processing.

## Common mistakes

- **Mistake:** Thinking big data only means very large amounts of data.
  - Why it is wrong: Big data is defined by multiple characteristics including volume, velocity, variety, veracity, and value. Simply having a lot of data does not make it big data if it does not also involve high speed or diverse formats.
  - Fix: Remember the five Vs. Big data is about the combination of size, speed, and diversity, not just size alone.
- **Mistake:** Confusing a data lake with a data warehouse for big data.
  - Why it is wrong: A data warehouse stores structured, processed data optimized for SQL queries. A data lake stores raw data in any format. Big data often requires a data lake because it deals with unstructured and semi-structured data.
  - Fix: Use a data lake when you need to store raw big data. Use a data warehouse when you have cleaned, structured data for reporting.
- **Mistake:** Believing that all big data processing must happen in real time.
  - Why it is wrong: Big data processing can be batch, real-time (streaming), or a combination of both. Batch processing is often used for large historical analysis, while stream processing handles live data. The choice depends on the use case.
  - Fix: Learn the difference: batch processes large chunks at intervals, stream processes data as it arrives. Choose based on whether the business needs immediate insights.
- **Mistake:** Assuming traditional relational databases can handle big data by adding more storage.
  - Why it is wrong: Relational databases are designed for vertical scaling (adding more power to a single machine), which has limits. Big data requires horizontal scaling (adding more machines) which is not natively supported by most traditional RDBMS.
  - Fix: For big data, use distributed systems like Hadoop, Spark, or Azure Data Lake Storage that are built for horizontal scaling.

## Exam trap

{"trap":"A question asks which Azure service is best for processing streaming data from IoT devices, and the options include Azure SQL Database, Azure Data Lake Storage, Azure Stream Analytics, and Azure Batch. A learner might choose Azure Data Lake Storage because they know it stores big data.","why_learners_choose_it":"They confuse storage with processing. Data Lake Storage is for storing, not processing streams.","how_to_avoid_it":"Match the action: if the scenario says 'process in real time,' choose a processing service like Azure Stream Analytics or Spark on HDInsight. Storage services store, processing services transform."}

## Commonly confused with

- **Big data vs Data warehouse:** A data warehouse is optimized for storing structured, cleaned data and running SQL queries for reporting and business intelligence. Big data often involves raw, unstructured data stored in data lakes. Data warehouses typically use a schema-on-write approach, while big data uses schema-on-read. (Example: If you have customer names and purchase amounts in a clean table, that is a data warehouse. If you have raw social media posts with typos and images, that is big data in a data lake.)
- **Big data vs Data lake:** A data lake is a centralized repository that can store all types of data at any scale, including big data. However, not all data lakes contain big data. A data lake can also hold small datasets. Big data is a characteristic of the data, not the storage system. (Example: A data lake is like a giant storage room that can hold anything from a single receipt to millions of video files. Big data is when that room is filled with receipts, videos, and sensor logs all coming in every second.)
- **Big data vs Business intelligence:** Business intelligence (BI) focuses on analyzing historical data to make decisions, often using dashboards and reports. Big data can be a source for BI, but big data is broader and includes real-time processing, predictive analytics, and handling unstructured data. (Example: BI is like looking in your rearview mirror at past sales. Big data is like having a GPS that also predicts traffic and suggests the best route ahead.)

## Step-by-step breakdown

1. **Data Generation** — Data is created from various sources like sensors, social media, transaction logs, and mobile devices. This is the raw material for big data. The speed and diversity of generation defines the velocity and variety challenges.
2. **Data Ingestion** — Data is collected and moved to a storage system using tools like Apache Kafka or Azure Event Hubs. This step handles high-volume streams and ensures no data is lost. It is like a massive conveyor belt carrying data into the system.
3. **Data Storage** — The ingested data is stored in a distributed file system like HDFS or Azure Data Lake Storage. Data is split into blocks and replicated across multiple servers for fault tolerance. This allows petabytes of data to be stored reliably.
4. **Data Processing** — Processing engines like Apache Spark or MapReduce transform the raw data into useful information. This can be batch processing (e.g., nightly reports) or stream processing (e.g., real-time fraud detection). The work is split across many servers to run in parallel.
5. **Data Analysis and Visualization** — The processed data is analyzed using query tools like SQL, and insights are visualized using dashboards like Power BI. This step delivers the value from big data, allowing businesses to make data-driven decisions.

## Practical mini-lesson

To truly understand big data in a practical IT context, you need to go beyond definitions and see how it is implemented in real environments. Imagine you are a junior data engineer at a logistics company that tracks thousands of delivery trucks. Each truck has GPS, engine sensors, and a camera that sends video footage every few seconds. In one day, you generate about 10 terabytes of data. The first challenge is ingestion. You cannot just dump files onto a server because the network would saturate. Instead, you set up a messaging queue like Azure Event Hubs that can handle millions of events per second. The data is then streamed into a data lake. In Azure, you would likely use Azure Data Lake Storage Gen2, which is built on Blob Storage but adds a hierarchical namespace. This allows you to organize data like a file system, which is important for analytics tools to understand directory structures. Next, you need to process the data. For real-time tasks, like detecting a truck that has stopped for too long, you use Azure Stream Analytics. It runs a continuous SQL query over the streaming data. For historical analysis, like calculating average fuel efficiency over the past month, you use batch processing with Azure Synapse Analytics or Spark on HDInsight. A common mistake in practice is not accounting for data veracity. Sensor data can be noisy. A GPS might report a location that is 100 meters off due to signal interference. You must implement data cleaning steps, such as filtering outliers and correcting timestamps. Data governance is another practical concern. With big data, you have to track where data came from, who has access, and ensure compliance with regulations like GDPR. Tools like Azure Purview help with data cataloging and lineage. While the DP-900 exam does not require you to set up these services, you should understand the flow: ingest, store, process, analyze. Knowing that big data systems are distributed and fault-tolerant is key. In an interview or on the job, you might be asked to design a pipeline for a given scenario. For example, you might be asked to recommend a solution for a company that needs to store 50 TB of logs daily and run hourly reports. The correct answer would include Azure Data Lake Storage for storage and Azure Data Factory to orchestrate batch processing. Avoid the trap of picking a single all-in-one solution like Azure SQL Database, because it cannot scale horizontally for such volumes. The practical takeaway is that big data is not just about scale, it is about choosing the right combination of tools to handle speed, variety, and reliability.

## Memory tip

Remember the five Vs: Volume, Velocity, Variety, Veracity, Value. Think VVVVV, Very Valuable Data.

## FAQ

**Do I need to know how to use Hadoop for the DP-900 exam?**

No, the DP-900 exam is conceptual. You need to understand what Hadoop is and its role in big data, but you will not be asked to write any commands or configuration.

**What is the difference between batch processing and stream processing in big data?**

Batch processing processes data in large chunks at scheduled times, suitable for historical analysis. Stream processing processes data as it arrives, in real time, suitable for immediate actions like fraud detection.

**Is big data only about storage?**

No, big data includes ingestion, storage, processing, analysis, and visualization. Storage is just one component, though a critical one.

**What is a data lake and how is it related to big data?**

A data lake is a storage repository that holds raw data in its native format. It is commonly used for big data because it can handle large volumes of unstructured and semi-structured data.

**Can I use Excel for big data?**

No, Excel has a row limit (about 1 million rows) and cannot handle the volume, velocity, or variety of big data. You need distributed systems like Azure Synapse or Spark.

**What does veracity mean in big data?**

Veracity refers to the quality and trustworthiness of the data. Big data often includes messy, incomplete, or inaccurate data that needs to be cleaned before analysis.

## Summary

Big data is a foundational concept in modern IT and cloud computing, representing datasets that are too large, fast, or diverse for traditional tools. In the DP-900 exam, understanding the five Vs, the difference between structured and unstructured data, and the role of services like Azure Data Lake Storage and Azure Synapse Analytics is critical. Big data is not just about storage but encompasses the entire pipeline from ingestion to visualization. Learners often confuse data lakes with data warehouses or believe that big data always means real-time processing. By mastering the basic concepts, you can confidently answer scenario-based questions on the exam and build a strong foundation for more advanced data certifications. Remember the five Vs and the flow: ingest, store, process, analyze. This knowledge will serve you well in interviews and on the job as data continues to grow in volume, velocity, and variety.

---

Practice questions and the full interactive page: https://courseiva.com/glossary/big-data