This chapter covers data lifecycle management on Google Cloud, a core topic for the GCDL exam that appears in approximately 10-15% of questions under Domain 3 (Data Analytics AI). You will learn how to automate the movement, deletion, and archival of data across storage classes to optimize cost and performance. Mastering these concepts is essential for designing cost-effective, compliant data architectures in Google Cloud.
Jump to a section
Imagine a public library that receives thousands of new books every month. The librarian must decide which books to keep on the shelves (hot data), which to move to a basement archive (warm data), and which to donate or recycle (cold data). The library has a finite shelf space, so not every book can stay on the main floor forever. The librarian uses a policy: new books stay on the display shelf for 30 days, then move to the general stacks for 6 months, then to the basement for 2 years, after which they are reviewed for donation. If a book is frequently checked out, it gets promoted back to the display shelf. This mirrors Google Cloud's data lifecycle management: data moves through storage classes based on access frequency and age, with automated policies (like Object Lifecycle Management) that transition objects from Standard to Nearline to Coldline to Archive. Just as the library tracks checkout logs to decide promotions, Cloud Storage uses access metadata and age conditions. The librarian's goal is to minimize cost while ensuring popular books are quickly accessible, just as cloud engineers balance performance and cost by tiering data appropriately.
What is Data Lifecycle Management?
Data Lifecycle Management (DLM) on Google Cloud refers to the policies and processes that govern data from creation through deletion. It is primarily implemented using Object Lifecycle Management in Cloud Storage, which automatically transitions objects between storage classes or deletes them based on conditions like age, creation date, or storage class. DLM is critical for reducing storage costs: Standard storage costs ~$0.020/GB/month, while Archive costs ~$0.0012/GB/month — a 16x difference. By moving infrequently accessed data to colder storage, organizations can save 50-80% on storage bills.
How It Works Internally
Cloud Storage objects are stored in buckets. Each bucket can have a lifecycle configuration — a set of rules defined in JSON or XML. Each rule has an action (e.g., SetStorageClass or Delete) and a condition (e.g., age: 30 days). When an object meets all conditions of a rule, the action is applied asynchronously — typically within 24 hours. The lifecycle manager runs periodically, scanning all objects in the bucket. Conditions include:
- age: Days since object creation (must be >= 0)
- createdBefore: Date in RFC 3339 format (e.g., "2023-01-01")
- isLive: true for live objects, false for archived (noncurrent) versions
- matchesStorageClass: List of storage classes (e.g., ["STANDARD","NEARLINE"])
- numNewerVersions: For versioned buckets, number of newer versions of the object
Key Components and Defaults
Storage Classes: Standard (default), Nearline (30-day min), Coldline (90-day min), Archive (365-day min). The minimum storage durations are enforced at billing — deleting early incurs an early deletion fee equal to the remaining days' storage cost.
Lifecycle Rules: Up to 100 rules per bucket. Actions: SetStorageClass (to any class) or Delete. Conditions can be combined with AND logic.
Autoclass: A feature that automatically transitions objects between Standard, Nearline, Coldline, and Archive based on access patterns, without manual rules. It uses machine learning to predict access frequency. Autoclass cannot be combined with manual lifecycle rules on the same bucket.
Retention Policies: Bucket-level policies that prevent object deletion or modification for a specified duration (e.g., 365 days). Retention policies override lifecycle delete actions — objects under retention cannot be deleted.
Object Holds: Per-object locks that prevent deletion or modification, overriding lifecycle rules.
Configuration and Verification
Lifecycle rules are configured via gcloud CLI, REST API, or Console. Example using gcloud:
gcloud storage buckets update gs://my-bucket --lifecycle-file=lifecycle.jsonSample lifecycle JSON:
{
"rule": [
{
"action": {"type": "SetStorageClass", "storageClass": "NEARLINE"},
"condition": {"age": 30}
},
{
"action": {"type": "SetStorageClass", "storageClass": "COLDLINE"},
"condition": {"age": 90}
},
{
"action": {"type": "Delete"},
"condition": {"age": 365}
}
]
}To verify rules:
gcloud storage buckets describe gs://my-bucket --format="default(lifecycle)"Interaction with Related Technologies
BigQuery: DLM applies to external tables referencing Cloud Storage. Lifecycle transitions affect query performance — querying Archive data incurs retrieval costs and latency.
Cloud Storage FUSE: Lifecycle rules still apply to objects accessed via FUSE. Frequent transitions to colder classes can cause performance degradation.
Data Transfer Service: Transferring data into a bucket does not automatically apply lifecycle rules to existing objects; rules apply only to new objects after rule creation unless conditions like createdBefore are used.
Compliance: Retention policies and holds are essential for regulatory compliance (e.g., SEC 17a-4). Lifecycle delete actions must respect retention policies.
Exam-Relevant Details
The minimum billing period for Nearline is 30 days, Coldline 90 days, Archive 365 days. Deleting before these periods incurs a fee equal to the remaining days' storage cost.
Lifecycle actions are asynchronous and may take up to 24 hours to execute.
Autoclass cannot be used with manual lifecycle rules on the same bucket.
Conditions in a rule are ANDed — all must be true for the action to trigger.
You can have multiple rules; they are evaluated independently.
The age condition uses object creation time, not last modification time.
For versioned buckets, isLive condition distinguishes current and archived versions.
Define Storage Classes
Identify the data access patterns: hot data (accessed frequently) stays in Standard; warm data (accessed monthly) goes to Nearline; cool data (quarterly) to Coldline; cold data (yearly or less) to Archive. Each class has a minimum storage duration: Nearline 30 days, Coldline 90 days, Archive 365 days. These durations affect early deletion fees. For example, if you move an object to Nearline and delete it after 10 days, you pay the remaining 20 days of storage cost.
Create Lifecycle Rules
Write a JSON or XML file with rules. Each rule specifies an action (SetStorageClass or Delete) and conditions (age, createdBefore, isLive, matchesStorageClass, numNewerVersions). Use the gcloud command: `gcloud storage buckets update gs://BUCKET_NAME --lifecycle-file=FILE`. Rules are evaluated in order, but all applicable rules are applied (multiple actions can occur on the same object if conditions are met). For example, a rule to transition to Nearline at age 30 and another to Coldline at age 90 will both execute.
Apply Lifecycle Configuration
Apply the lifecycle configuration to the bucket. This is a bucket-level setting. Once applied, the lifecycle manager scans objects in the bucket periodically (typically once per day). New objects created after the rule is applied are subject to the rules from creation. Existing objects are also scanned and will meet conditions like `age` based on their creation date. If you want to apply rules retroactively to existing objects, use `createdBefore` conditions.
Monitor and Verify
Use `gcloud storage buckets describe` to view the lifecycle configuration. Monitor storage costs via the Google Cloud Console billing reports or export logs to BigQuery. Check the `storage.googleapis.com/storage_class_change` audit logs to see when objects transition. If objects are not transitioning as expected, verify that conditions are met (e.g., age count starts from object creation, not from rule creation). Also ensure no retention policy or hold is blocking deletion.
Optimize and Adjust
Analyze access patterns using Cloud Monitoring metrics or Storage Insights. Adjust lifecycle rules to better match actual usage. For example, if you find that objects accessed after 90 days are still being accessed monthly, consider extending the transition to Coldline to 180 days. Use Autoclass if access patterns are unpredictable — it automatically moves objects based on ML-driven access predictions. Remember Autoclass cannot coexist with manual rules.
Enterprise Scenario 1: Healthcare Compliance
A hospital stores patient imaging data (DICOM files) for 7 years as required by HIPAA. They use Cloud Storage with a retention policy of 7 years to prevent deletion. However, after 3 years, access drops to near zero. They configure lifecycle rules: after 90 days, transition from Standard to Nearline; after 365 days, to Coldline; after 1095 days (3 years), to Archive. This reduces storage costs by ~70%. The retention policy ensures no data is deleted before 7 years. Misconfiguration could occur if the retention policy is set to a shorter duration than the lifecycle delete rule — the delete rule would be blocked, but the object would remain in Archive, incurring costs. The engineer must ensure retention policy duration exceeds any lifecycle delete age.
Enterprise Scenario 2: Media and Entertainment
A video streaming company ingests raw footage (hot data) for editing. After 30 days, the footage is rarely accessed. They use lifecycle rules to transition to Nearline at day 30, Coldline at day 90, and Archive at day 365. But they also need to keep the latest 5 versions of each object (versioning enabled). They use numNewerVersions condition to delete old versions after 30 days. Common mistake: forgetting that age condition applies to each version individually, so old versions may be deleted earlier than intended. They also use Autoclass on a separate bucket for user-uploaded content where access patterns are unpredictable. Performance consideration: retrieving from Archive takes hours (not milliseconds), so critical data must be restored before access is needed.
Scenario 3: Financial Services Logs
A bank generates terabytes of transaction logs daily. They need to retain logs for 7 years for regulatory audits. They use lifecycle rules: Standard for 7 days, Nearline for 30 days, Coldline for 365 days, Archive for 2555 days (7 years), then delete. They also apply a retention policy of 7 years to prevent early deletion. A common issue: if the retention policy is set to 7 years from object creation, but the lifecycle delete rule also triggers at 7 years, the delete action will be blocked until the retention period expires. To avoid this, the delete rule should be set to a slightly later age (e.g., 2556 days) to ensure the retention policy has already expired. They use Cloud Audit Logs to monitor lifecycle transitions and set up alerts if objects are not transitioning as expected.
What the GCDL Exam Tests
Domain 3 (Data Analytics AI) Objective 3.1: "Describe data lifecycle management and how it applies to Google Cloud." The exam focuses on:
The four storage classes: Standard, Nearline, Coldline, Archive — their use cases, minimum storage durations, and costs.
Object Lifecycle Management: conditions (age, createdBefore, isLive, matchesStorageClass, numNewerVersions) and actions (SetStorageClass, Delete).
Autoclass: what it does, its limitations (cannot combine with manual rules, uses ML).
Retention policies and holds: how they override lifecycle delete actions.
Early deletion fees: when they apply and how they are calculated.
Common Wrong Answers and Traps
"Lifecycle rules are applied immediately" — Wrong. They are asynchronous and can take up to 24 hours. Candidates confuse this with bucket policy changes that are near-instant.
"Autoclass can be combined with manual lifecycle rules" — Wrong. The exam explicitly tests that Autoclass and manual rules are mutually exclusive on the same bucket.
"Age condition counts from the time the rule is created" — Wrong. Age is counted from object creation time. Candidates often think existing objects start aging from rule creation.
"Deleting an object from Nearline after 10 days incurs no extra fee" — Wrong. Early deletion fee applies: you pay the remaining 20 days of storage cost.
Specific Numbers and Terms
Nearline minimum: 30 days
Coldline minimum: 90 days
Archive minimum: 365 days
Lifecycle rules max per bucket: 100
Lifecycle action max time: 24 hours
Autoclass: uses ML, cannot be used with manual rules
Retention policy: bucket-level, minimum duration 1 second, maximum none
Edge Cases
If a retention policy and lifecycle delete rule both apply, the delete action is blocked until the retention period ends.
For versioned buckets, isLive: false targets noncurrent versions. numNewerVersions can delete old versions when newer ones exist.
Lifecycle rules apply to all objects in the bucket, including those in folders (prefixes).
Conditions are ANDed within a single rule; multiple rules are ORed (each rule independently evaluated).
Eliminating Wrong Answers
When you see a question about lifecycle timing, remember: asynchronous, up to 24 hours. If the question says "immediately" or "within minutes," it's likely wrong. For cost questions, recall the minimum durations. If an option says "no early deletion fee" for Nearline deletion at 10 days, it's wrong. For Autoclass, any option suggesting manual rules can be added is incorrect. For retention, any option saying lifecycle can override retention is wrong.
Cloud Storage offers four storage classes: Standard (hot), Nearline (30-day min), Coldline (90-day min), Archive (365-day min).
Object Lifecycle Management automates transitions and deletions based on conditions like age, creation date, and storage class.
Lifecycle actions are asynchronous and typically execute within 24 hours.
Autoclass uses machine learning to automatically move objects between storage classes; it cannot be used with manual lifecycle rules.
Retention policies and object holds prevent deletion, overriding lifecycle delete actions.
Early deletion fees apply if an object is deleted before its minimum storage duration (30/90/365 days).
Lifecycle conditions within a single rule are ANDed; multiple rules are evaluated independently (ORed).
Age condition counts from object creation time, not rule creation time.
These come up on the exam all the time. Here's how to tell them apart.
Manual Lifecycle Rules
Requires explicit rule definition in JSON/XML.
Supports conditions like age, createdBefore, isLive, matchesStorageClass, numNewerVersions.
Actions include SetStorageClass and Delete.
Can be combined with retention policies and holds.
Provides deterministic, predictable transitions.
Autoclass
No manual configuration; uses ML to predict access.
Automatically transitions between Standard, Nearline, Coldline, Archive.
Cannot set custom conditions or delete actions.
Cannot be combined with manual lifecycle rules on same bucket.
Adapts to changing access patterns without manual updates.
Mistake
Lifecycle rules apply instantly after configuration.
Correct
Lifecycle actions are asynchronous and typically execute within 24 hours. There is no guarantee of immediate application. The lifecycle manager runs periodic scans.
Mistake
Autoclass can be used alongside manual lifecycle rules on the same bucket.
Correct
Autoclass and manual lifecycle rules are mutually exclusive. If you enable Autoclass, you cannot add manual lifecycle rules, and vice versa.
Mistake
The age condition in lifecycle rules counts from the date the rule was created.
Correct
Age is always counted from the object's creation time, not from the rule's creation date. Existing objects will meet the age condition based on their original creation timestamp.
Mistake
Deleting an object from Nearline after 10 days incurs no extra cost.
Correct
Nearline has a 30-day minimum storage duration. Deleting before 30 days incurs an early deletion fee equal to the remaining days' storage cost. For 10 days, you pay for 20 extra days.
Mistake
Retention policies and lifecycle delete rules can both delete objects at the same time.
Correct
Retention policies block deletion. If a lifecycle delete rule triggers while a retention policy is active, the delete action is suppressed until the retention period expires. The object remains.
Reveal each answer, then mark whether you got it right. Score 60%+ to unlock the next chapter.
Nearline is for data accessed less than once a month, with a 30-day minimum storage duration. Coldline is for data accessed less than once a quarter, with a 90-day minimum. Coldline has lower storage cost ($0.007/GB vs $0.010/GB) but higher retrieval cost ($0.02/GB vs $0.01/GB). Use Coldline for data that is accessed very infrequently but must be retained.
Yes. For versioned buckets, you can use the `numNewerVersions` condition to delete older versions when a specified number of newer versions exist. For example, `numNewerVersions: 3` will delete versions that have 3 or more newer versions. You can also use `isLive: false` to target noncurrent versions.
Retention policies prevent object deletion for a specified duration. If a lifecycle delete rule triggers during the retention period, the delete action is blocked. The object remains until the retention period expires. After expiration, the lifecycle delete rule may execute on the next scan. To ensure deletion, set the lifecycle delete age to be greater than the retention duration.
You incur an early deletion fee equal to the remaining days' storage cost. For example, if you delete after 100 days, you pay for 265 extra days. The fee is calculated based on the storage class rate. Always consider the minimum duration when planning transitions.
Yes. Lifecycle rules apply to all objects in the bucket, including existing ones. The `age` condition counts from the object's creation time, so existing objects will meet the condition when their age reaches the specified value. You can also use `createdBefore` to target objects created before a specific date.
Yes, the maximum is 100 rules per bucket. Each rule can have multiple conditions (ANDed). If you need more complex logic, consider combining conditions or using multiple buckets.
Autoclass uses machine learning models trained on access patterns across Google Cloud. It monitors read and write timestamps and predicts future access frequency. Objects are moved to colder classes when they are predicted to have low access, and promoted to warmer classes if access increases. The exact algorithm is proprietary.
You've just covered Data Lifecycle Management on Google Cloud — now see how well it sticks with free GCDL practice questions. Full explanations included, no account needed.
Done with this chapter?