Cloud Spanner Replication and TrueTime

Cloud Spanner is Google's fully managed, horizontally scalable, globally distributed relational database service. It is the first and only database that combines the benefits of relational database structure (ACID transactions, SQL) with non-relational horizontal scalability, all while providing strong consistency across regions. Traditional relational databases can scale vertically (by adding more power to a single machine) but hit limits. NoSQL databases scale horizontally but sacrifice ACID transactions and strong consistency. Spanner solves this by using a novel combination of synchronous replication, Paxos-based consensus, and a global time service called TrueTime.

For the ACE exam, you need to know that Spanner is designed for mission-critical applications that require both strong consistency and global scale, such as financial trading platforms, global supply chain management, and large-scale advertising systems.

TrueTime is the key innovation that makes Spanner's global consistency possible. It is a globally distributed time service that provides a monotonically increasing timestamp with a bounded uncertainty interval. Each Spanner server is equipped with a combination of GPS receivers and atomic clocks. The GPS provides absolute time with high accuracy, while atomic clocks maintain time locally between GPS syncs. The uncertainty interval, known as the "TrueTime error bound" (ε), is typically 1-7 milliseconds. The system guarantees that the actual time of an event is within the interval [t_earliest, t_latest] returned by the TrueTime API.

The TrueTime API exposes two core functions: - TT.now(): Returns an interval [earliest, latest] representing the current time uncertainty. - TT.after(t) and TT.before(t): Return true if the time t is definitely in the past or future, respectively.

When a write transaction occurs, Spanner assigns a commit timestamp s that is guaranteed to be greater than all previously assigned timestamps. To ensure this, Spanner uses the following mechanism: 1. The leader replica for a Paxos group calls TT.now() to get an interval [e, l]. 2. It selects a commit timestamp s that is greater than l (the latest possible current time). This ensures that s is in the future relative to the TrueTime uncertainty. 3. Before committing, the leader waits until TT.after(s) returns true—meaning the actual time has passed s by at least the uncertainty bound. This wait is called the "commit wait" and is typically equal to ε (1-7 ms). 4. Once the wait completes, the leader knows that any other transaction that might have started before s will see a timestamp earlier than s, ensuring a consistent global order.

To create a Spanner instance with replication across three regions:

gcloud spanner instances create test-instance \
    --config=nam3 \
    --description="Test Instance" \
    --nodes=2

The --config flag specifies the regional or multi-regional configuration. For example, nam3 is a multi-region configuration covering US regions.

To view instance configuration details:

gcloud spanner instances describe test-instance

To list available configurations:

gcloud spanner instance-configs list

To create a database with optional leader options:

gcloud spanner databases create mydb --instance=test-instance

To run a query with strong consistency:

SELECT * FROM MyTable WHERE Key = 'value';

By default, reads are strong. To use stale reads, you set a timestamp bound:

# Using gcloud with stale reads (not directly supported via CLI, but in client libraries)
# Example in Java:
// DatabaseClient client = ...;
// TimestampBound bound = TimestampBound.ofExactStaleness(10, TimeUnit.SECONDS);
// try (ResultSet rs = client.singleUse(bound).executeQuery(statement)) { ... }

TrueTime is not just for Spanner; it is part of Google's global infrastructure. However, for the ACE exam, focus on Spanner's use of TrueTime for: - Global Snapshot Isolation: Spanner provides externally consistent reads and writes. A read at a timestamp t sees all transactions that committed before t and none that committed after. TrueTime ensures that the timestamp t is globally meaningful. - Paxos Leader Lease: The Paxos leader uses TrueTime to manage leases. The leader holds a lease that expires at a certain time; it uses TrueTime to ensure it does not act as leader after the lease expires, preventing split-brain scenarios. - Distributed Transactions: For transactions spanning multiple Paxos groups (e.g., across different tables or partitions), Spanner uses two-phase commit (2PC) with TrueTime timestamps to ensure atomicity and consistency across groups.

A large investment bank uses Spanner to maintain a real-time order book for equities traded across multiple exchanges in New York, London, and Tokyo. The bank requires ACID transactions to ensure that buy and sell orders are matched without double-spending or lost updates. They deploy a Spanner multi-region instance with the nam3 configuration (US regions) and eur3 (Europe) and asia1 (Asia) as separate instances, but they need global consistency across all regions. To achieve this, they use a single Spanner instance spanning all three regions with 9 replicas (3 per region). The write latency is approximately 50-100 ms due to cross-continent Paxos communication. They use strong reads for order matching and stale reads for reporting (with 5-second staleness). The bank monitors the TrueTime uncertainty metric in Cloud Monitoring to ensure it stays below 7 ms. When misconfigured—e.g., using multiple single-region instances instead of a multi-region instance—they experienced inconsistent order books because writes in one region were not visible in another until replication lag caught up, violating the requirement for strong consistency.

A major ad tech company uses Spanner to store ad impressions and clicks for real-time bidding. They need to deduplicate events and calculate budgets across multiple data centers. They deploy a Spanner multi-region instance with 5 replicas (2 in us-east1, 2 in us-west1, 1 in us-central1). They use stale reads for ad serving (to minimize latency) with a staleness of 10 seconds, because showing an ad that is up to 10 seconds behind is acceptable. For budget updates, they use strong reads to ensure they don't overspend. They scale nodes based on peak QPS: during Black Friday, they increase from 10 nodes to 50 nodes to handle 500,000 QPS. The common pitfall is underestimating the node count: each node handles only 10,000 QPS reads, so they must calculate nodes = (peak QPS) / 10,000. Failure to scale leads to throttling and increased latency.

A gaming company uses Spanner to maintain a global leaderboard for a multiplayer game. They use Spanner's interleaved tables to store player profiles and scores. They need strong consistency to ensure that the leaderboard accurately reflects the latest scores. They deploy a single-region Spanner instance in us-central1 because most players are in North America. For players in other regions, they use Cloud CDN to cache leaderboard results with a Time-to-Live (TTL) of 30 seconds. They use stale reads for the cached data and strong reads only when a player submits a new score. The problem they encountered: when they initially used stale reads for score submission, they sometimes overwrote a higher score with a lower one because the stale read returned an older score. They fixed this by using strong reads for the update operation. The exam tests this scenario: always use strong reads for read-modify-write operations.