High Availability and Disaster Recovery for Cloud Databases

High availability (HA) and disaster recovery (DR) are two strategies to keep your database working when things go wrong, but they solve different problems. HA is about surviving small, local failures quickly. DR is about surviving big, catastrophic failures that destroy an entire location.

Let us start with high availability. The goal of HA is to minimise downtime — the time your database is unavailable. Downtime is measured in nines. Three nines means 99.9% uptime, which allows about 8.7 hours of downtime per year. Five nines means 99.999% uptime, which allows only about 5.26 minutes of downtime per year. To achieve HA, you need redundancy: multiple copies of the same data running on different servers.

In Google Cloud SQL, HA is built using a primary instance and a standby instance. The primary instance handles all your database reads and writes. The standby instance sits idle in a different zone within the same region. A zone is one datacentre. A region is a geographic area with multiple zones, like us-central1. When the primary fails — for example, its server crashes — Google Cloud automatically switches traffic to the standby. This process is called a failover. The standby becomes the new primary immediately. You do not need to change your application code because the connection string (the address your app uses to talk to the database) stays the same. The small gap during failover is usually under 60 seconds.

Cloud Spanner, Google’s globally distributed database, gives HA by design. It automatically replicates your data across multiple zones within a region, or even across regions. If one zone fails, Spanner uses a consensus protocol called Paxos to choose a new leader replica. Your application keeps working with no downtime. You do not need to configure a separate standby because Spanner manages its own redundancy.

Now for disaster recovery. DR handles events that take out an entire region, such as a major earthquake, a power grid failure, or a software bug that corrupts data across all zones in a region. DR requires you to have a copy of your data in a different region. In Cloud SQL, you set up a cross-region replica. This replica asynchronously receives changes from the primary. That means there is a small delay — usually a few seconds — between a write on the primary and that write appearing on the replica. This delay is called replication lag. If a disaster happens, you can promote the replica to become a new primary. You will lose any data that had not yet been replicated (the lagged data). This data loss is measured by the recovery point objective (RPO). You also have a target for how long it takes to get the new database running: the recovery time objective (RTO). For example, you might set an RPO of 15 minutes (you accept losing up to 15 minutes of recent data) and an RTO of 2 hours (you want the database back within 2 hours).

Cloud Spanner handles DR differently. You can configure a multi-region configuration, such as nam3 which uses regions in North America. Spanner automatically keeps your data consistent across all regions. If one region fails, Spanner continues serving from the other regions. There is no failover time and no data loss because the data is synchronously replicated — every write is written to multiple regions before the database confirms success. Synchronous replication means zero lag, but it adds latency because your application must wait for the write to reach the distant region.

Other services also have HA and DR options. Bigtable, Google’s NoSQL database, offers replication within a single cluster (for HA) and across clusters (for DR). Firestore uses multi-region replication automatically. Memorystore, Google’s managed Redis, provides a standard tier (single server) and a high-availability tier (with a replica).

In summary, HA keeps your database running through local failures with fast failover. DR keeps your business alive through region-wide disasters by maintaining separate copies in another location. A complete database design uses both, because HA alone cannot survive a regional disaster, and DR alone cannot give you instant failover for a simple server crash.

You work as a junior database engineer at a retail company called ShopNow. ShopNow runs an e-commerce website that processes 10,000 orders per hour. The database is a Cloud SQL PostgreSQL instance. Your manager tells you that last month, a single zone in us-central1 lost power for 45 minutes. ShopNow was down for the whole 45 minutes because it had only a single database instance. The company lost USD$600,000 in sales. Now you have been asked to design a high availability and disaster recovery solution.

You start by enabling HA on the existing Cloud SQL instance. In the Google Cloud Console, you edit the instance and choose High Availability. Cloud SQL automatically creates a standby instance in a different zone within us-central1. The failover happens in under 60 seconds. Now if one zone fails, the website keeps running. But you know this does not protect against a disaster that affects the entire us-central1 region, such as a widespread tornado.

For disaster recovery, you create a cross-region replica in us-west2. The replica asynchronously replicates data from the primary. You test the replication lag and find it is typically 3-5 seconds. You decide on an RPO of 5 seconds — you accept losing no more than 5 seconds of orders if a disaster happens. You set an RTO of 1 hour: you want the new database running within 60 minutes. You document a disaster recovery runbook that includes these steps:

You also practise this failover quarterly. In a test, you force a simulated disaster by deleting the primary instance's data. Your team promotes the replica. The first time, it takes 2 hours because the DNS cache had not been cleared. You update the runbook to flush DNS first. The second drill completes in 45 minutes. Your manager approves your design.

For Cloud Spanner, you would take a different approach. If ShopNow used Spanner, you would configure a multi-region instance. You would select the nam3 configuration, which includes us-central1, us-east1, and us-west1. Spanner automatically routes reads to the nearest region and handles regional failures without downtime. You do not need a separate HA standby because Spanner has built-in redundancy. You still practise disaster scenarios: you simulate a region outage by blocking network traffic from one region using VPC firewall rules, then verify that the application continues to serve from the other regions without any manual action.