This chapter covers Google Cloud Firestore and Datastore, two NoSQL document databases on Google Cloud. For the ACE exam, understanding their differences, use cases, and migration paths is critical—approximately 5-8% of exam questions touch on these services. You must know when to use each, their consistency models, pricing, and how they relate to Firebase and App Engine. This chapter provides a deep dive into their architectures, features, and exam traps.
Jump to a section
Imagine you run a growing company that needs to store employee records. Datastore is like a massive filing cabinet with labeled folders. Each folder is a 'kind' (e.g., 'Employee'), and inside you have individual files (entities) with a unique ID. You can only file documents in a specific folder, and to find a file, you must know the folder and the ID or run a query that scans folders. The cabinet has strong consistency for reads after writes, but you cannot have nested folders inside folders—only flat kinds. Firestore, on the other hand, is like a modern digital library with a card catalog system. You have 'collections' (like shelves) and 'documents' (books). Each book can contain subcollections—like a book having a 'chapters' sub-shelf. This allows deep nesting. The library also provides real-time updates: if someone borrows a book, the librarian immediately notifies everyone watching that shelf. Firestore also offers multi-region replication with eventual consistency for queries, but strong consistency for single-document reads. The library is newer, faster for mobile apps, and has a more flexible querying system, but the filing cabinet is simpler and cheaper for large-scale batch operations. Both store data, but the library is designed for interactive, real-time applications, while the cabinet is for back-office batch processing.
What Are Firestore and Datastore?
Cloud Datastore and Cloud Firestore are NoSQL document databases provided by Google Cloud. Datastore originated as the storage layer for App Engine (High-Replication Datastore), while Firestore is the next-generation version, also serving as Firebase's primary database. Both are schemaless, horizontally scalable, and offer ACID transactions, but they differ in consistency models, query capabilities, and pricing.
Datastore Architecture
Datastore stores data as entities, each belonging to a kind (like a table in SQL). Entities have properties (key-value pairs) and a unique key. Keys can be arranged in ancestor relationships, forming entity groups. Writes within a single entity group are strongly consistent; cross-group writes are eventually consistent. Queries over a single entity group are strongly consistent; global queries are eventually consistent. Datastore uses indexes for queries—composite indexes must be defined in advance via a index.yaml file. It supports transactions (up to 25 entity groups per transaction with cross-group transactions enabled). Datastore is ideal for large-scale batch processing, analytics, and applications requiring high write throughput.
Firestore Architecture
Firestore stores data in documents (similar to JSON objects) organized into collections. Documents can contain subcollections, allowing hierarchical data modeling. Firestore offers two consistency modes: strong consistency for single-document reads and writes, and eventual consistency for queries and multi-document transactions (unless using strong consistency mode via a single-location database). It supports real-time listeners (via WebSocket), automatic multi-region replication, and composite indexes (automatic for single-field queries, manual for multi-field). Firestore pricing is based on reads, writes, deletes, and storage, with a free tier (10 GiB storage, 50K reads/day, 20K writes/day). It is designed for mobile and web app development with Firebase integration.
Key Differences
Consistency: Datastore offers strong consistency within entity groups; Firestore offers strong consistency for single-document reads, eventual for queries (unless using strong consistency mode).
Data Model: Datastore uses kinds and entity groups; Firestore uses collections and documents with subcollections.
Queries: Datastore requires composite index pre-definition; Firestore automatically indexes single fields, composite indexes need manual creation.
Transactions: Datastore supports cross-group transactions (up to 25 groups); Firestore supports multi-document transactions (up to 500 documents).
Real-time: Firestore has built-in real-time listeners; Datastore does not.
Pricing: Datastore charges per entity read/write; Firestore charges per document read/write/delete.
Region: Datastore is available in all regions; Firestore is available in most regions but with multi-region options (nam5, eur3).
Migration from Datastore to Firestore
Google recommends migrating from Datastore to Firestore for new projects. The migration can be done in-place (Datastore mode) or by exporting/importing data. Datastore mode Firestore uses the Datastore API but runs on Firestore infrastructure, offering better performance and consistency. The ACE exam may ask about migration steps: export Datastore entities, import into Firestore, and update application code to use Firestore client libraries.
Configuration and Commands
gcloud datastore export — exports entities to Cloud Storage.
gcloud firestore import — imports entities from Cloud Storage.
gcloud firestore indexes composite create — creates composite indexes.
gcloud firestore databases create — creates a Firestore database in Native mode or Datastore mode.
gcloud datastore indexes create — creates composite indexes for Datastore.
Example: Export Datastore data:
gcloud datastore export --namespaces='(default)' gs://my-bucket/Example: Import into Firestore:
gcloud firestore import gs://my-bucket/2019-05-26T11:49:00_12345/Performance Considerations
Datastore: Write throughput scales with entity groups; up to 1 write per second per entity group. Use sharding to increase throughput.
Firestore: Scales automatically with no per-entity-group limits. Maximum write rate is 10,000 writes per second per database (can be increased via quota request).
Firestore real-time listeners consume resources; limit number of listeners per client.
Both services use indexes; avoid over-indexing to reduce write costs and latency.
Security
Both integrate with IAM. Fine-grained access control using Firestore Security Rules (for Firebase) or IAM roles (for server-side). Datastore uses IAM only. Firestore supports App Check for mobile app protection.
Exam Tips
Know that Firestore is the recommended database for new projects.
Understand that Datastore mode Firestore is for legacy App Engine apps.
Remember that Firestore has a free tier; Datastore does not.
Be aware that Firestore real-time listeners are not available in Datastore mode.
Know that cross-group transactions in Datastore are limited to 25 groups; Firestore can handle up to 500 documents.
Choose Firestore or Datastore
Determine whether to use Firestore or Datastore based on application requirements. For new projects, especially mobile/web apps requiring real-time sync, choose Firestore Native mode. For existing App Engine apps using Datastore, use Datastore mode Firestore. Consider consistency needs: Firestore offers strong consistency for single-document reads; Datastore offers strong consistency within entity groups. Evaluate pricing: Firestore has a free tier; Datastore charges per entity operation. Also consider query complexity—Firestore automatically indexes single fields, while Datastore requires manual composite index creation.
Create a Firestore Database
Use the Google Cloud Console, gcloud CLI, or API to create a Firestore database. Choose between Native mode (Firestore API) or Datastore mode (Datastore API). For Native mode, specify a database ID (e.g., '(default)') and location (e.g., 'nam5' for multi-region). Use: `gcloud firestore databases create --location=nam5`. For Datastore mode, use: `gcloud app create --region=us-central` (creates App Engine with Datastore mode). The database is created with a default index on document IDs and property names.
Define Data Model
Design the data model using collections and documents (Firestore) or kinds and entities (Datastore). In Firestore, organize data into collections (e.g., 'users') containing documents (e.g., 'user123'). Documents can have subcollections (e.g., 'orders'). In Datastore, define kinds (e.g., 'User') with entity keys and properties. Use ancestor relationships for entity groups. Avoid deep nesting in Firestore (max depth 20). Consider that Firestore queries are shallow (only one collection level) unless using collection group queries.
Set Up Indexes
Firestore automatically indexes single-field queries. For composite queries (e.g., `WHERE age > 30 ORDER BY name`), create composite indexes via Console or CLI: `gcloud firestore indexes composite create --collection-group=users --field-config=field-path=age,order=ASCENDING --field-config=field-path=name,order=ASCENDING`. Datastore requires composite indexes defined in `index.yaml` and deployed via `gcloud datastore indexes create index.yaml`. Without proper indexes, queries will fail. Over-indexing increases write cost and latency.
Write and Read Data
Use client libraries (Node.js, Python, Java, etc.) to perform CRUD operations. In Firestore, use `db.collection('users').doc('user123').set({name: 'Alice'})`. Real-time listeners attach via `onSnapshot()`. In Datastore, use `datastore.save({key: datastore.key(['User', 'user123']), data: {name: 'Alice'}})`. Transactions: Firestore uses `db.runTransaction()`; Datastore uses `datastore.transaction()`. Firestore transactions can operate on up to 500 documents; Datastore transactions are limited to 25 entity groups (with cross-group enabled).
Monitor and Optimize
Monitor usage via Cloud Monitoring and Logging. Firestore provides dashboards for read/write operations, latency, and errors. Optimize by reducing document size (max 1 MiB), using subcollections for scalability, and limiting real-time listeners. For Datastore, monitor entity group write throughput and consider sharding. Use composite indexes sparingly. Review pricing: Firestore charges per read/write/delete; Datastore charges per entity read/write. Use Cloud Scheduler for periodic exports to Cloud Storage for backup.
Enterprise Scenario 1: Mobile App Backend with Real-Time Sync
A social media startup uses Firestore as its primary database for a mobile app. The app requires real-time updates when users post messages or like content. Firestore's real-time listeners allow the app to instantly reflect changes across all clients without polling. The data model uses collections: 'posts', 'users', and 'likes' as subcollections under each post. The startup benefits from Firestore's free tier during early development. As the app grows, they enable multi-region replication for disaster recovery. Challenges include managing composite indexes for complex queries (e.g., posts sorted by popularity within a date range) and controlling costs as read/write volume increases. They use Firestore security rules to restrict access based on user authentication. Misconfiguration of security rules can lead to data breaches; they learned to test rules in the Firebase console simulator.
Enterprise Scenario 2: Legacy App Engine Migration
A large e-commerce company has a monolithic App Engine application using Datastore. They need to migrate to Firestore for better scalability and real-time features. They use the in-place migration to Datastore mode Firestore, which requires no code changes. After migration, they gradually update code to use Firestore client libraries for real-time inventory updates. They export Datastore entities to Cloud Storage as a backup before migration. The migration is done during low-traffic hours to minimize downtime. They discover that Datastore mode Firestore still requires composite indexes for existing queries, which they define in index.yaml. The migration reduces write latency by 30% due to Firestore's improved infrastructure.
Enterprise Scenario 3: IoT Data Ingestion
An industrial IoT company ingests sensor data from thousands of devices. They choose Datastore because of its high write throughput and lower cost per entity operation compared to Firestore. Each device writes a sensor reading every second. They use entity groups by device ID to maintain strong consistency for each device's data. To scale writes, they shard entity groups by time (e.g., one entity group per hour). They use composite indexes for queries like 'average temperature per device over last hour'. Challenges include managing index bloat and ensuring consistent backups. They use Cloud Scheduler to export Datastore data to BigQuery for analytics. A misconfiguration of entity groups (too many writes per group) causes contention and high latency; they resolve by increasing sharding granularity.
ACE Exam Focus
This topic falls under Objective 2.3: Planning Solutions for Data Storage. The exam tests your ability to choose between Firestore and Datastore based on requirements. Key areas: - Consistency models: Know that Datastore provides strong consistency within entity groups, eventual across groups. Firestore provides strong consistency for single-document reads, eventual for queries (unless using strong consistency mode). The exam may ask: 'Which database ensures strong consistency for all queries?' Answer: Neither—Firestore queries are eventually consistent by default. - Data modeling: Understand that Firestore uses collections and documents with subcollections; Datastore uses kinds and entity groups. The exam may present a scenario requiring hierarchical data and ask which service supports subcollections (Firestore). - Pricing: Firestore has a free tier; Datastore does not. The exam may ask about cost optimization for a mobile app with low traffic—Firestore free tier is the answer. - Real-time: Firestore supports real-time listeners; Datastore does not. A question about live updates points to Firestore. - Migration: Know that Datastore mode Firestore allows existing Datastore apps to run without code changes. The exam may ask about migration steps: export/import or in-place upgrade. - Transactions: Datastore transactions limited to 25 entity groups (with cross-group enabled). Firestore transactions up to 500 documents. The exam may ask about maximum transaction size.
Common Wrong Answers
'Firestore provides strong consistency for all queries.' This is false; only single-document reads are strongly consistent. Candidates confuse Firestore's strong consistency claim with Datastore's entity group consistency.
'Datastore supports real-time listeners.' It does not; candidates may assume all Google databases have real-time features.
'Firestore is only available through Firebase.' Firestore is a standalone Google Cloud service; Firebase integration is optional.
'Datastore is recommended for new projects.' Google recommends Firestore for new projects; Datastore is legacy.
Specific Numbers
Firestore max document size: 1 MiB.
Firestore max write rate: 10,000 writes/second per database (default).
Datastore max write rate: 1 write/second per entity group.
Firestore free tier: 10 GiB storage, 50K reads/day, 20K writes/day.
Datastore cross-group transaction limit: 25 entity groups.
Firestore transaction limit: 500 documents.
Edge Cases
If an application requires strong consistency for all queries, you must use Datastore with entity groups or Firestore with single-document reads only. Firestore does not support strongly consistent multi-document queries.
For multi-region disaster recovery, Firestore offers multi-region locations (nam5, eur3); Datastore does not have multi-region replication.
When migrating from Datastore to Firestore, composite indexes must be recreated; the exam may ask about index.yaml conversion.
Eliminating Wrong Answers
If a question asks for a database with real-time capabilities, eliminate Datastore immediately. If it asks for a free tier, eliminate Datastore. If it asks for strong consistency across all operations, consider Datastore with entity groups, but note that even Datastore is eventually consistent across groups. The correct answer often depends on the specific requirement: 'real-time' → Firestore, 'legacy App Engine' → Datastore mode, 'low cost for small app' → Firestore free tier.
Firestore is the recommended NoSQL database for new projects; Datastore is legacy.
Firestore supports real-time listeners; Datastore does not.
Firestore has a free tier; Datastore does not.
Datastore provides strong consistency within entity groups; Firestore provides strong consistency for single-document reads.
Firestore uses collections/documents with subcollections; Datastore uses kinds/entities with entity groups.
Migration from Datastore to Firestore can be done via export/import or in-place Datastore mode.
Firestore transactions can involve up to 500 documents; Datastore cross-group transactions are limited to 25 entity groups.
Composite indexes are required for multi-field queries in both services; Firestore auto-indexes single fields.
These come up on the exam all the time. Here's how to tell them apart.
Cloud Firestore
Uses collections and documents with subcollections (hierarchical).
Real-time listeners for live updates.
Free tier: 10 GiB storage, 50K reads/day.
Strong consistency for single-document reads; eventual for queries.
Recommended for new projects, especially mobile/web apps.
Cloud Datastore
Uses kinds and entity groups (flat hierarchy).
No real-time listeners.
No free tier; pay per entity operation.
Strong consistency within entity groups; eventual across groups.
Legacy; suitable for existing App Engine apps and batch processing.
Mistake
Firestore and Datastore are completely different databases with no migration path.
Correct
Datastore mode Firestore allows existing Datastore applications to run on Firestore infrastructure without code changes. Data can be exported from Datastore and imported into Firestore Native mode.
Mistake
Firestore provides strong consistency for all reads and queries.
Correct
Firestore offers strong consistency only for single-document reads. Queries (including multi-document reads) are eventually consistent by default. Strong consistency for queries can be achieved in single-location databases but not in multi-region mode.
Mistake
Datastore supports real-time listeners like Firestore.
Correct
Datastore does not have built-in real-time listeners. Real-time functionality must be implemented via polling or third-party tools.
Mistake
Firestore is only available through Firebase and cannot be used with Google Cloud services.
Correct
Firestore is a fully integrated Google Cloud service accessible via the Cloud Console, gcloud CLI, and client libraries. Firebase integration is optional for mobile/web apps.
Mistake
Datastore is cheaper than Firestore for all workloads.
Correct
Firestore has a free tier that can make it cheaper for low-volume applications. For high-volume batch operations, Datastore may be cheaper due to lower per-operation costs. Pricing depends on read/write patterns.
Reveal each answer, then mark whether you got it right. Score 60%+ to unlock the next chapter.
Firestore Native mode uses the Firestore API with collections, documents, subcollections, and real-time listeners. Datastore mode uses the Datastore API with kinds, entities, and entity groups, but runs on Firestore infrastructure. Datastore mode is for legacy App Engine apps that want to migrate without code changes. Native mode is for new projects requiring Firestore features.
Yes, Firestore can be used with App Engine (standard and flexible) via client libraries. However, App Engine standard environment has a built-in Datastore API (Datastore mode) for legacy apps. For new App Engine apps, Firestore Native mode is recommended.
Use `gcloud datastore export` to export entities to Cloud Storage. Then use `gcloud firestore import` to import into Firestore. Ensure the export includes all namespaces and kinds. The import will create corresponding collections and documents in Firestore.
Yes, Firestore supports multi-document ACID transactions. A transaction can read and write up to 500 documents across multiple collections. Datastore also supports ACID transactions, but limited to 25 entity groups (with cross-group enabled).
The maximum size of a Firestore document is 1 MiB (1,048,576 bytes). This includes the document name, fields, and subcollection references. For larger data, use subcollections or consider Cloud Storage.
You can create composite indexes via the Google Cloud Console (Firestore > Indexes) or using the gcloud CLI: `gcloud firestore indexes composite create --collection-group=COLLECTION --field-config=field-path=FIELD,order=ORDER`. For Datastore, define indexes in `index.yaml` and deploy with `gcloud datastore indexes create index.yaml`.
Firestore provides strong consistency for single-document reads and writes. Queries (e.g., collection queries) are eventually consistent by default. To achieve strong consistency for queries, you can use a single-location database (not multi-region) and ensure all reads are within a transaction. Datastore provides strong consistency within an entity group.
You've just covered Cloud Firestore and Datastore — now see how well it sticks with free ACE practice questions. Full explanations included, no account needed.
Done with this chapter?