# Data Life Cycle, Storage, and Data Quality

> Chapter 2 of the Courseiva DATA-PLUS curriculum — https://courseiva.com/learn/data-plus/data-life-cycle-and-storage

**Official objective:** 1.2 — Describe the data life cycle and data quality dimensions

## Introduction

What happens to data from the moment it's created until it's permanently deleted, and why does half the time spent on a data project go into just cleaning it up?

Every piece of data you work with has a lifespan, and if you don't manage that lifespan carefully, you end up with a mess of unreadable files, broken reports, and wrong conclusions. For the DA0-002 exam, understanding this lifecycle and the dimensions of data quality is the absolute foundation – without it, you can’t build reliable reports or make good business decisions.

## The Family Recipe Book Analogy

Have you ever helped a relative organise a massive collection of family recipes, each one scribbled on a different scrap of paper, some stained with age, others missing crucial steps?

That chaotic pile is your data before any structure. The data life cycle is the entire journey of that recipe: from the moment your grandmother first thought of blending that secret sauce (creation), to scribbling it on a napkin (storage), to you photocopying it for the family reunion (use), to you realising the ink is fading (maintenance), to finally throwing away a torn, unusable copy (destruction).

Now, data quality is whether that recipe actually works. Is the ingredient list complete? Are the measurements precise (cup vs. handful)? Is the instruction 'bake until done' clear? A 'low-quality' recipe has missing steps, vague quantities, or conflicting directions from two different relatives. A 'high-quality' recipe is accurate, complete, consistent, and timely – you could hand it to a stranger and they'd bake that perfect pie. Just like data, a recipe is only valuable if its quality is high enough for someone to trust and use it.

## Core explanation

Let's break down the data life cycle. This is a six-stage model that describes the entire journey of data within an organisation. It's a framework to help you think about where data comes from, how it's used, and what you should do with it when it's no longer needed.

**Stage 1: Create.** Data doesn't just appear. It is created or captured. This could be a point-of-sale system recording a customer's purchase, a sensor reading the temperature in a freezer, a web form collecting a user's email address, or an employee manually typing a client's name into a spreadsheet. The key thing here is that data must be captured accurately at the source. If you type a customer's name as 'Jon' instead of 'John', that error will haunt every report downstream.

**Stage 2: Store.** Once data is created, it needs a home. This is the storage stage. The home could be a simple spreadsheet, a database on a server, a cloud-based data warehouse, or even a physical filing cabinet. The choice of storage depends on the type of data, how often it will be accessed, and who needs to see it. For example, a small business might store customer orders in an Excel file, while a large retailer would use a relational database like SQL Server or a cloud data warehouse like Snowflake.

**Stage 3: Use.** This is where the data gets put to work. It is processed, analysed, queried, visualised, or fed into another application. For example, the sales data from the previous stage is used to generate a monthly sales report, a dashboard showing which products are selling best, or a recommendation engine that suggests items to a customer. This stage often involves running calculations, creating charts, or building machine learning models.

**Stage 4: Share.** Data rarely lives in isolation. It needs to be shared with other people or systems. This stage involves transmitting data to colleagues, partners, customers, or other software. For example, exporting a sales report as a PDF and emailing it to the CEO, or providing an API (Application Programming Interface) so a partner company can automatically pull your product inventory. Security is critical here – you must ensure only authorised people can access the data.

**Stage 5: Archive.** Not all data is active forever. When data is no longer needed for daily operations but must be kept for legal compliance or historical analysis, it is moved to an archive. Archival storage is cheaper and slower than active storage. For example, a bank must keep customer transaction records for seven years by law, but after one year, those records might be moved from the main database to an offline tape backup or a low-cost cloud storage tier.

**Stage 6: Destroy.** Eventually, data reaches the end of its useful life. Keeping data forever is expensive and risky (security breaches, legal liability). The destroy stage is about permanently deleting or destroying data so it cannot be recovered. This could be securely erasing files, physically shredding hard drives, or using software to overwrite data multiple times. For instance, when a customer closes an account, the company might destroy all copies of their personal data after the legal retention period expires.

Now, onto **Data Quality**. A data quality dimension is a measurable characteristic that determines how 'good' or 'fit for purpose' your data is. There are six key dimensions you need to know for DA0-002:

- **Accuracy:** Is the data correct? Does it reflect the real-world object it is supposed to represent? For example, a customer's phone number in the database should match the number they actually use. If it's one digit wrong, the data is inaccurate.

- **Completeness:** Is all the required data present? A customer record missing a postcode is incomplete. If you need to send promotional mail, that missing postcode makes the record useless.

- **Consistency:** Is the data the same across different systems? For example, a customer's name stored as 'John Smith' in the sales system but 'Jon Smith' in the support system is inconsistent. This causes confusion and errors when merging data.

- **Timeliness:** Is the data up to date and available when needed? A stock report from last week is not timely if you need to know today's inventory levels. Timeliness is about the data being current and accessible within a reasonable time frame.

- **Validity:** Does the data conform to the defined format or rules? An email address field containing 'notanemail' is invalid. A date field with '32/13/2023' is invalid. Validity ensures data conforms to business rules.

- **Uniqueness:** Is there only one record for each real-world entity? Duplicate records (e.g., two customer IDs for the same person) are a sign of poor uniqueness. This wastes storage and can lead to sending two copies of the same marketing mail.

Understanding these concepts is crucial because poor-quality data costs organisations time, money, and reputation. The DA0-002 exam will expect you to identify which dimension is broken in a given scenario.

## Real-world context

Let's step into the shoes of a junior data analyst at a medium-sized e-commerce company called 'ShopFast'. Your boss asks you to create a weekly sales report showing the top 10 best-selling products. This simple task forces you to deal with every stage of the data life cycle and multiple data quality dimensions.

**Step 1: Understanding the Data Sources (Create).** You first need to find where the sales data comes from. You discover the data is created in two places: (1) the company's online store website, which records every completed order, and (2) a separate mobile app. Both systems create transaction records with fields like order ID, product ID, quantity, and price.

**Step 2: Accessing the Storage (Store).** You log in to the company's data warehouse (a central data storage built for analysis) to find the raw sales tables. The data from both the website and the app is stored here, but you notice the website uses a field called 'Prod_ID' while the app uses 'ProductCode'. This is a consistency problem.

**Step 3: Cleaning the Data (Use and Quality).** Before you can run any analysis, you must fix the data quality issues. You perform a check:

- **Accuracy:** You spot an order with a total price of £1,000,001 for a single £20 t-shirt. This is clearly an error (fat-fingered entry). You must decide whether to correct it (if you can find the correct value) or exclude it.
- **Completeness:** Several orders are missing the 'quantity' field. You cannot include these in a 'top seller' count, so you either exclude them or use a default value (but that would reduce accuracy).
- **Consistency:** You create a mapping table to standardise the field names from the two source systems before merging the data.
- **Timeliness:** The data from the app is updated every hour, but the website data is only loaded overnight. Your report, if run at 3pm, will show incomplete data from the website. You must note this gap.
- **Validity:** A product ID field contains a text string 'N/A' instead of a number. You filter that out.
- **Uniqueness:** You find one customer who has three nearly identical orders placed within 30 seconds. It turns out the 'Buy' button was clicked three times due to a glitch. You deduplicate these.

**Step 4: Building the Report (Use and Share).** After cleaning, you calculate the top 10 products. You then share this report via a dashboard that the sales team can access. You set up data quality alerts so the team is notified if the data accuracy drops below a threshold.

**Step 5: Archival and Destruction (Archive and Destroy).** You know that raw transaction data must be kept for 3 years for tax purposes. After that, it will be automatically archived to a slower, cheaper storage. After 7 years, it will be permanently destroyed using a secure deletion tool.

In this scenario, your core job as a data professional was not just analysing numbers – it was managing the lifecycle and ensuring data quality at every step. This is exactly the kind of thinking the DA0-002 exam tests.

## Exam focus

The DA0-002 exam loves to test your ability to recognise and apply the six stages of the data life cycle and the six dimensions of data quality in specific, realistic scenarios. You will not be asked to recite definitions word-for-word, but to diagnose what is happening in a short paragraph. Here is exactly what you must master:

- **Exam Topic 1: The Six Life Cycle Stages.** You must be able to put them in the correct order (Create, Store, Use, Share, Archive, Destroy) and identify which stage a given action belongs to. For example, 'exporting a report to a PDF' is Share, not Use. 'Moving old records to tape' is Archive.

- **Exam Topic 2: Data Quality Dimensions.** The most frequently tested dimensions are Accuracy, Completeness, Consistency, and Timeliness. Validity and Uniqueness appear less often, but you must still know them. The exam will give you a data quality problem and ask you to name the dimension.

**Common traps the exam sets:**
- **Trap: Confusing 'Accuracy' with 'Validity'.** Accuracy is about truth to the real world (the phone number is correct). Validity is about meeting a format rule (the phone number has the right number of digits). A phone number like '12345' might be valid (if the rule is '5 digits') but inaccurate (it's not the customer's real number).
- **Trap: Misplacing 'Archive'.** A question might describe moving data to a cheaper storage system and ask if this is 'Archive' or 'Destroy'. The key is the intent: if data is being kept for future reference or compliance, it's Archival. If it's being erased permanently, it's Destruction.
- **Trap: Overthinking the data use stage.** 'Use' does not mean just viewing the data. It means any form of processing or analysis to derive value. Running a SQL query to calculate an average is 'Use'. Simply reading a report that someone else generated is not 'Use' – it's 'Share'.

**Key definitions to memorise:**
- **Data life cycle:** The sequence of phases data goes through from creation to destruction.
- **Data quality:** The overall utility of a dataset, measured by accuracy, completeness, consistency, timeliness, validity, and uniqueness.
- **Data governance:** The overall management of data availability, usability, integrity, and security. This is a broader concept, but the life cycle is part of it.

The exam will also test your understanding that poor data quality is the root cause of many business failures. Expect a question where you must choose the biggest risk of low-quality data – the correct answer is almost always 'incorrect business decisions' or 'loss of customer trust'.

Finally, be aware of the 'data profiling' term. The exam may ask what tool or process you use to discover the quality of data. The answer is data profiling – it's the act of examining datasets to find errors, inconsistencies, and missing values. You'll need to know it's part of the 'Use' stage, before analysis.

## Step by step

1. **1. Capture/Create Data** — Data is born at a source: a form submission, a sensor reading, or a manual entry. This is the most critical step because errors introduced here propagate throughout the entire life cycle. You must ensure data is captured accurately and in a structured format.
2. **2. Store Data in a Suitable Repository** — Once captured, data must be placed in a storage system (database, data warehouse, file system, cloud bucket). The choice of storage depends on volume, velocity, variety, and who needs access. This step directly impacts how easily the data can be used later.
3. **3. Profile and Clean Data** — Before analysis or sharing, you must examine the data for quality issues. This is data profiling. You then correct inaccuracies, fill in missing values, resolve inconsistencies, and remove duplicates. This step is often the most time-consuming but is essential for reliable results.
4. **4. Use Data for Analysis or Operations** — The cleaned data is now ready for its purpose: generating reports, feeding dashboards, training machine learning models, or powering business processes. This is where the data delivers value. The specific use case determines which quality dimensions are most important.
5. **5. Share Data with Stakeholders or Systems** — Data is distributed to authorised people or systems via dashboards, exports, APIs, or email. Security controls (like access permissions and encryption) are applied here. You must ensure shared data is still high quality and properly anonymised if needed.
6. **6. Archive or Destroy Data as Per Policy** — After its useful life, data is either moved to archival storage (for legal or historical reasons) or permanently destroyed. The retention policy defines how long data must be kept. Destruction must be secure to prevent data breaches.

## Comparisons

### Accuracy vs Validity

**Accuracy:**
- Measures if data values match the real-world truth
- Example: Customer's phone number is the number they actually have
- Failing accuracy means the data is wrong in meaning

**Validity:**
- Measures if data conforms to a defined format or rule
- Example: Phone number field contains exactly 10 digits
- Failing validity means the data is not in an acceptable format

### Archive vs Backup

**Archive:**
- Moves inactive data to long-term, low-cost storage
- Purpose is compliance, reference, or historical preservation
- Retrieval is typically slow and not intended for daily use

**Backup:**
- Creates a copy of active data for disaster recovery
- Purpose is to restore data after accidental deletion or system failure
- Retrieval is fast and intended for immediate restoration

### Use Stage vs Share Stage

**Use Stage:**
- Involves processing, querying, or analysing the data
- Example: Running a SQL query to calculate an average
- The data is being transformed or acted upon

**Share Stage:**
- Involves transmitting data to others or making it available
- Example: Exporting a report and emailing it to a colleague
- The data is not being changed, just distributed

### Completeness vs Uniqueness

**Completeness:**
- Measures if all required fields are filled in
- Example: A customer record missing a postcode
- Concerned with presence/absence of data values

**Uniqueness:**
- Measures if each real-world entity has only one record
- Example: Two customer IDs for the same person
- Concerned with multiple representations of the same thing

## Diagram

_This flowchart shows the six stages of the data life cycle, from creation to destruction, with a decision point on whether to archive or destroy._

```mermaid
flowchart TD
    A[Create/Capture] --> B[Store in Database/Warehouse]
    B --> C[Profile and Clean Data]
    C --> D[Use for Analysis/Reports]
    D --> E[Share via Dashboards/Exports]
    E --> F{Will data be needed again?}
    F -- Yes --> G[Archive to Long-Term Storage]
    F -- No --> H[Securely Destroy Data]
    G --> H
```

## Common misconceptions

- **Misconception:** Data that is complete is automatically accurate. **Reality:** Completeness only measures if all required fields are filled in, not whether the values are correct. A completely filled form could have all fields filled with wrong information. (Beginners assume that if a record has no missing fields, it must be good data. They confuse the quantity of data with the truthfulness of data.)
- **Misconception:** Archiving and backing up data are the same thing. **Reality:** A backup is a copy of active data for disaster recovery, created frequently and stored for short-term recovery. An archive is a move of inactive data to long-term, low-cost storage for compliance or reference, often with slow retrieval. (Both involve storing data in a secondary location, but their purpose, frequency, and retrieval speed are completely different. Books often use similar language for both.)
- **Misconception:** Data quality only matters for data used in reports. **Reality:** Data quality matters at every stage of the life cycle, from creation to destruction. Bad data created at the start will pollute everything downstream, including archives and even the decision to destroy data. (People often think of data quality as a 'analysis' concern, not a lifecycle-wide concern. They don't realise that even archived data can be used for audits, so it needs to be correct.)
- **Misconception:** The data life cycle stages always happen in a strict linear order with no overlap. **Reality:** In reality, stages overlap and repeat. Data is created, stored, used, modified, and then used again. Archiving can happen multiple times. The model is a framework, not a rigid step-by-step process. (Textbooks teach the stages sequentially for clarity, leading beginners to believe it's a waterfall process. The exam tests understanding that it's a cycle and data can revisit stages.)
- **Misconception:** Consistency means all data should be stored in the same format across all systems. **Reality:** Consistency means that the meaning of data is the same across systems. Different systems can use different formats (e.g., one stores dates as 'MM/DD/YYYY' and another as 'YYYY-MM-DD') as long as the value (e.g., 1st January 2023) can be reliably translated without ambiguity. (Beginners often think consistency means identical formatting. The exam tests the conceptual understanding of consistency as semantic agreement, not syntactic uniformity.)

## Key takeaways

- The data life cycle has six stages in order: Create, Store, Use, Share, Archive, Destroy.
- Data quality dimensions measure how fit data is for its intended purpose, not just whether it exists.
- Accuracy measures if data reflects the real world, while validity measures if data conforms to a defined format or rule.
- Archival stores inactive data for long-term compliance or reference, while destruction permanently erases data beyond recovery.
- Poor data quality leads directly to incorrect business decisions and loss of trust.
- Data profile is the process of examining a dataset to assess its quality across completeness, accuracy, consistency, and other dimensions.
- Timeliness means data is both current and available quickly enough for the decision it supports.
- Duplicate records violate the uniqueness dimension and waste storage and processing resources.

## FAQ

**What is the difference between a data life cycle and a data analytics lifecycle?**

The data life cycle (Create, Store, Use, Share, Archive, Destroy) covers the entire journey of data from birth to death. A data analytics lifecycle (like the CRISP-DM model) is a subset that focuses only on the 'Use' stage – how you explore, analyse, and model data to extract insights.

**Can data quality be improved after the data is already stored?**

Yes, through a process called data cleansing or data scrubbing. You can correct errors, fill in missing values (using imputation), standardise formats, and remove duplicates. However, it is much easier to prevent quality issues at the creation stage than to fix them later.

**Is 'deleting a file' the same as 'destroying data' in the data lifecycle?**

No. Simply deleting a file from a computer often only removes the pointer to the data, leaving the actual data recoverable with special software. Proper data destruction involves overwriting the data multiple times, degaussing (using a magnetic field), or physically destroying the storage media.

**What happens if I have both 'Archive' and 'Backup' – are they different?**

Yes. A backup is a temporary copy of active data to protect against loss or corruption, designed for quick restoration. An archive is a long-term retention of data that is no longer actively used, usually for compliance purposes and with slower retrieval speeds.

**What is a data repository?**

A data repository is any place where data is stored. It can be a simple file folder, a relational database, a data warehouse (for analysis), a data lake (for raw data), or a cloud storage service. The choice of repository depends on the data type and how it will be used.

**What is the biggest risk of poor data quality in a business?**

The biggest risk is making incorrect business decisions. Bad data leads to flawed analysis, which leads to wrong conclusions about customer behaviour, product performance, or market trends. This can result in wasted marketing spend, poor investments, and loss of customer trust.

## Check your understanding

1. **What is the correct order of the first three stages of the data life cycle?**

   Answer: Create, Store, Use.

2. **A sales report uses customer data from last year. The data is accurate and complete, but the report is for a meeting tomorrow. Which data quality dimension is likely violated?**

   Answer: Timeliness. The data may be out of date if it's from last year and not current enough for the decision needed.

3. **A database has two records for the same person: 'John Doe' and 'Jon Doe'. Which data quality dimension is primarily affected?**

   Answer: Consistency (and potentially Uniqueness if both records refer to the same entity). The name representation differs.

4. **What is the difference between archiving data and destroying data?**

   Answer: Archiving moves inactive data to long-term storage for preservation, while destroying permanently erases the data so it cannot be recovered.

5. **A company moves old transaction records to a tape drive for a 7-year legal retention period. Which life cycle stage is this?**

   Answer: Archive.

---

Interactive version with quiz and diagrams: https://courseiva.com/learn/data-plus/data-life-cycle-and-storage