Cloud Storage Lifecycle Rules

Cloud Storage Lifecycle Rules are automated policies that manage the lifecycle of objects in Google Cloud Storage (GCS) buckets. They allow you to automatically transition objects to lower-cost storage classes (e.g., from Standard to Nearline, Coldline, or Archive) or delete them when they meet certain conditions. This helps reduce storage costs and enforce data retention policies without manual intervention.

In enterprise environments, data accumulates rapidly. Many objects become less frequently accessed over time but still need to be retained for compliance or future analysis. Manually moving or deleting objects is impractical at scale. Lifecycle rules automate this process, ensuring that data is stored in the most cost-effective class based on its access patterns and age. The ACE exam tests your ability to configure these rules correctly, especially regarding conditions and actions.

Lifecycle rules are defined at the bucket level. Each rule consists of a condition and an action. The condition specifies when the action should be triggered, such as after a certain number of days from object creation, or when the object's storage class is a specific class. The action defines what happens: either transitioning the object to another storage class or deleting it.

When you set a lifecycle rule, GCS evaluates it asynchronously. There is no guarantee of immediate execution; actions typically occur within 24 hours, but can take up to 48 hours in some cases. The evaluation is done on a best-effort basis. Objects that match the conditions are queued for the action. For transition actions, the object is moved to the target storage class. For delete actions, the object is permanently removed.

- Conditions: - age: Number of days from object creation to trigger action. Default: no value (required for age-based rules). - createdBefore: Date (e.g., '2023-01-01') – triggers action for objects created before that date. - isLive: Boolean (true/false) – applies to live or noncurrent (versioned) objects. Default: true. - matchesStorageClass: List of storage classes (e.g., 'STANDARD', 'NEARLINE', 'COLDLINE', 'ARCHIVE') – triggers action only for objects in those classes. - numNewerVersions: Number of newer versions of an object (for versioned buckets). - daysSinceNoncurrentTime: Days since object became noncurrent (versioned buckets). - daysSinceCustomTime: Days since custom time was set on object. - customTimeBefore: Date before which custom time was set. - noncurrentTimeBefore: Date before which object became noncurrent.

- Actions: - SetStorageClass: Transition to 'NEARLINE', 'COLDLINE', 'ARCHIVE', or 'STANDARD'. - Delete: Permanently delete the object. - AbortIncompleteMultipartUpload: Abort incomplete uploads after a specified number of days.

- Defaults: - No default rules exist; you must explicitly set them. - Lifecycle rules are evaluated approximately once per day. - Minimum storage duration before transitioning: 30 days for Nearline, 90 days for Coldline, 365 days for Archive. If you attempt to transition before the minimum, the transition will be delayed until the minimum period is met. - There is no cost for setting lifecycle rules, but each transition action incurs a charge (Class A operation for transition to a colder class, Class B for transition to a warmer class). Deletion is also a Class A operation.

Lifecycle rules can be configured via the Google Cloud Console, gsutil, or REST API.

Using gsutil:

To set a lifecycle rule from a JSON file:

gsutil lifecycle set lifecycle-config.json gs://my-bucket

Example JSON file:

{
  "rule": [
    {
      "action": {"type": "SetStorageClass", "storageClass": "NEARLINE"},
      "condition": {"age": 30, "isLive": true}
    },
    {
      "action": {"type": "Delete"},
      "condition": {"age": 365, "isLive": true}
    }
  ]
}

To view current lifecycle configuration:

gsutil lifecycle get gs://my-bucket

Using Console:

To check if lifecycle rules are working, you can use gsutil ls -L to see the storage class of objects over time. Also, audit logs (Cloud Audit Logs) record lifecycle actions for monitoring.

Example:

gsutil ls -L gs://my-bucket/object.txt | grep "Storage class"

This shows the current storage class, which should change after a lifecycle transition.

A large e-commerce company generates terabytes of server logs daily. They need to retain logs for 90 days for debugging, then for 7 years for compliance. Initially, logs are stored in Standard class for fast access. After 30 days, access frequency drops significantly. They configure a lifecycle rule to transition logs from Standard to Nearline after 30 days, then from Nearline to Coldline after 90 days, and finally delete after 7 years. This reduces storage costs by ~70%. In production, they use versioning to keep multiple log versions and add a rule to delete noncurrent versions after 30 days. A common misconfiguration is setting the transition to Coldline at 30 days, which violates the 90-day minimum for Coldline, causing the rule to be delayed. The team monitors using Cloud Monitoring alerts on lifecycle action failures.

A financial institution backs up databases daily and stores backups in Standard class. After 1 year, backups are rarely needed but must be retained for 10 years. They implement a lifecycle rule to transition backups older than 365 days to Archive class. They also set a rule to delete backups older than 10 years. Because Archive has a 365-day minimum, they ensure backups stay in Standard for at least 365 days before transitioning. A pitfall: they initially set the transition to Archive at 180 days, but GCS delayed the action until day 365. After correcting, they saved costs. They also use bucket lock to prevent deletion of backups during the retention period, which overrides lifecycle delete actions.

A media streaming service stores video files. New releases are in Standard for fast streaming. After 6 months, they become less popular. They transition to Nearline after 180 days, then to Coldline after 2 years, and delete after 5 years. They also abort incomplete multipart uploads after 7 days to clean up failed uploads. A challenge: the lifecycle rule for aborting incomplete uploads must have a condition with age or daysSinceCustomTime? Actually, for incomplete uploads, the condition uses age based on the upload initiation time. They set age: 7 and action AbortIncompleteMultipartUpload. This works well. They also use numNewerVersions to keep only the latest 3 versions of each video. The ACE exam often tests these real-world patterns.