Storing NoSQL Data with Cloud Bigtable and Firestore

Before we talk about Cloud Bigtable and Firestore, you need to understand what a NoSQL database is. The term 'NoSQL' stands for 'Not Only SQL'. Traditional databases, called relational databases (like Cloud SQL or MySQL), store data in tables with strict rows and columns. Every row must have the same columns, and you must define the columns ahead of time. This is great for things like a bank account system where every account has an account number, balance, and owner name. But it is terrible for storing data that changes shape over time, like a social media post where one post has text, another has an image, and a third has a video.

NoSQL databases throw away the strict table structure. They allow each piece of data (called a 'record' or 'document') to have different fields. This makes them incredibly flexible and fast for certain workloads. Google Cloud offers two main NoSQL databases: Cloud Bigtable and Firestore. They solve different problems.

Cloud Bigtable is a 'wide-column' NoSQL database. Imagine a giant spreadsheet where each row has many, many columns (sometimes thousands), but each row does not need to have all the columns. The most important thing in Bigtable is the 'row key'. This is a unique identifier for each row, like a customer ID. Bigtable can only look up data by this row key. It cannot do complex searches like 'find all customers who are male and over 30 years old'. It can only say 'give me the row for customer ID 529'. If you want to scan many rows, you can ask for all rows where the row key starts with '529', but that is a limited scan. Bigtable is built for very high throughput (thousands of reads and writes per second) and very low latency (under 10 milliseconds). It stores data on Solid State Drives (SSDs) or Hard Disk Drives (HDDs). It is excellent for time-series data (like sensor readings every second), financial trading data, and recommendation engine data. It was inspired by Google's internal Bigtable system, which powers Google Search and Gmail.

Firestore is a 'document-oriented' NoSQL database. Imagine a collection of file folders. Each folder is a 'document', and documents are stored in 'collections'. A document can contain any fields you want: text, numbers, lists, or even sub-documents. For example, a document for a user might have fields like 'name', 'email', 'favourite_colours' (a list), and 'address' (a sub-document with 'street' and 'city'). Firestore can do complex queries. You can say 'find all users where age is greater than 25 and favourite_colour includes blue'. Firestore maintains indexes (like a book index) to make these searches fast. It is designed for mobile and web applications that need real-time updates. If a user updates a document, every other user watching that document gets the change within seconds. Firestore also has strong consistency guarantees — when a write completes, every subsequent read will see the new data. Bigtable, by contrast, has 'eventual consistency' for some operations, meaning a read immediately after a write might not see the latest data.

What do these databases replace? They replace the need for a relational database with complex joins (combining data from different tables) when your data does not have many relationships. They also replace the need for manual scaling. With a traditional database, if you suddenly get 100 times more users, you have to buy a bigger server or reconfigure your system. Bigtable and Firestore are 'managed services' on Google Cloud — Google handles the scaling for you. Bigtable can grow from 1 node to hundreds of nodes automatically. Firestore scales automatically based on traffic.

Why do they exist? Because modern applications generate data at a speed and volume that traditional relational databases cannot handle. A single IoT (Internet of Things) device might send a temperature reading every second. If you have 10 million devices, that is 10 million writes per second. A relational database would collapse. Bigtable handles this effortlessly. Similarly, a chat app where every message triggers a notification to dozens of users needs real-time speed and flexible data structures — Firestore handles that.

When deciding which to use, think about the query patterns. If you only ever need to look up data by a single key (like a device ID or user ID) and you need millions of writes per second, choose Cloud Bigtable. If you need to do complex searches ('find all orders in the last hour that cost over $50') and you want real-time updates, choose Firestore. For the PDE exam, you must memorise that Bigtable is for analytical, high-throughput workloads (think 'data science and analytics') and Firestore is for transactional, user-facing applications (think 'mobile apps and websites').

Imagine you are building the backend for a mobile game called 'Galaxy Racers'. Players compete in real-time races across the galaxy. Each player has a profile, a list of owned spaceships, and a race history. You have 5 million active players. You need to store every race result so that later you can analyse which spaceships are most popular. You also need to show a player their friend list and update it in real time when a friend changes their profile picture.

Step 1: You identify that the player profile and friend list need complex queries. A player might search for 'all friends who have played in the last week' or 'all ships that are faster than 200 mph'. This is a perfect fit for Firestore. You create a collection called 'players'. Each document in this collection has a unique player ID as the document ID. The document holds fields like 'username', 'email', 'spaceships_owned' (a list of strings), and 'friends' (a list of player IDs). Firestore handles real-time updates easily — when a player changes their username, the app updates the Firestore document, and all friends see the change instantly.

Step 2: The race results, however, are a different beast. Every race generates a record containing the race ID, the participating player IDs, the start time, the finish time, and the spaceship used. With 5 million players, there could be 500,000 races per day. You need to be able to look up the history for a single player quickly (by player ID) and you need to perform massive analytical queries later, like 'what was the average race time for spaceship X in region Y last month'. For this, you choose Cloud Bigtable. The row key for each race result is a combination of player ID and race start time, like 'player123#20241005T143022'. This allows fast lookups for a specific player's races, and you can scan all rows that start with a particular player ID.

Step 3: You configure Bigtable with a cluster of 5 nodes initially, with SSDs for low latency. You set up automatic scaling so that when a new game update drives 10x more races, Bigtable adds nodes automatically. You also enable 'replication' across two Google Cloud regions (e.g., us-east1 and europe-west1) so that players on different continents get similar latency. Bigtable automatically replicates data between the regions with eventual consistency.

Step 4: For Firestore, you enable 'Firestore in Native mode' (rather than Datastore mode). You use Firestore Security Rules to control who can read and write which documents. You also set up a 'Cloud Function' that triggers every time a new race result document is added to a 'race_results' collection in Firestore (note: for this scenario, you might actually use Bigtable for race results, so this Cloud Function might be for an auxiliary task). The Cloud Function sends a push notification to the player saying 'Your race result is saved!'.

What does an IT professional actually do? They do not build the database from scratch. They use the Google Cloud Console (a web interface) or the gcloud command-line tool to create a Bigtable instance or a Firestore database. They write code (usually in Python, Go, or Java) that uses the client libraries to read and write data. For Bigtable, they install the 'hbase-client' library and write commands like 'table.put(row_key, column_family, column_qualifier, value)'. For Firestore, they use client libraries and write commands like 'db.collection("players").document("player123").set({"username": "racer99"})'. They monitor the databases using Cloud Monitoring to check latency, throughput, and error rates. They also manage backups. For Bigtable, they configure 'backup schedules' that create copies of the table data to Cloud Storage. For Firestore, they use 'Firestore Export' to export all data to a Cloud Storage bucket periodically.