devShakib

Firestore Data Modelling for Read-Heavy Sites: Denormalisation, Projections, and Indexes

Firestore schema design for content driven sites: denormalise for reads, slug as document ID, field projections when a listing blows the 2MB response cap.

Firestore bills you per document read, and a content site is roughly 99% reads. That one sentence should be doing more work in your schema design than it usually does. The shape you pick is not just an ergonomics question about how the data "should" be organised — it is a cost model, a latency model, and, once your documents get fat enough, a correctness model too.

I learned the last part the hard way. My portfolio runs on Firestore: 114 blog posts, 98 tools, 32 games, plus projects, apps and leads, served to a Flutter web client with no backend in between. A listing query that had worked fine for a year started hanging and then failing outright, and the cause was not an index, a rule or a network. It was that every post document carries its full Markdown body, and I was asking for all of them at once. The fix was a field projection — fetch four fields instead of the whole document — and it turned a failing query into a 40KB response.

This post is the data model I would start from now for any read-heavy, content-driven site on Firestore: denormalise for the read path, put the derived values on the document, use the slug as the document ID, keep documents small enough that a list query is cheap, and be deliberate about arrays versus subcollections, composite indexes, cursors, listeners, and the reflex to count things.

Model for the read, not for the entity

The instinct you bring from SQL is to model the entity — a post has an author, an author is a row, so store an author reference. Firestore punishes that instinct immediately, because there is no join and there is no server to hide one behind. Resolving ten author references on a ten-item list is ten extra document reads and ten extra round trips, executed on the user's device, on the user's network, while they stare at a spinner.

So denormalise aggressively. My posts document stores author as a plain string. If I ever rename myself, that is a one-off batch update over 114 documents — a job I will run once, versus a join every list render forever.

The same logic applies to anything derived. readingMinutes is stored, not computed on read. excerpt is stored, not sliced from content at render time. coverImageUrl is a resolved absolute URL, not a storage path the client has to turn into one. All three are computed once, in the admin editor, at write time:

Map<String, dynamic> toMap() => {      'title': title,      'slug': slug,      'excerpt': excerpt,      'readingMinutes': readingMinutes,      'coverImageUrl': coverImageUrl,      // ...      'updatedAt': FieldValue.serverTimestamp(),    };

The rule I use: if a value is displayed more often than it changes, store it. A blog post is written once and read thousands of times. Any work you can shift to the write is work you are doing once instead of once per visitor, and on a per-read billing model that is not a micro-optimisation — it is the whole game.

A second, less obvious form of denormalisation is the status flag. Every public collection carries status: 'published' | 'draft' on the document itself rather than in a separate collection or a subcollection of drafts. That is what lets the security rules say "anyone may read documents where status == 'published'" in one line, and it is what lets the public query be a single equality filter.

Slug as the document ID

Use the slug as the document ID. posts/firestore-data-modelling-for-read-heavy-sites, not posts/aB3xK9pQ2mZ.

I know this because I shipped both patterns on the same site and only one of them is good. Tools, games and projects are keyed by slug — the seed script writes documents/projects?documentId=<slug> over REST, and the built-in sync writes col.doc(slug). Posts, historically, were not, so looking one up costs a query:

Future<Post?> getPostBySlug(String slug) async {  final snap = await _col(Collections.posts)      .where('slug', isEqualTo: slug)      .where('status', isEqualTo: PublishStatus.published)      .limit(1)      .get();  return snap.docs.isEmpty ? null : Post.fromDoc(snap.docs.first);}

Both cost one document read. They are not equivalent otherwise. The direct doc(slug).get() needs no index and no query planning, returns a DocumentSnapshot you can use directly instead of a QuerySnapshot you have to unwrap and check, is trivially cacheable, and — the part that matters most — makes the URL and the primary key the same string. There is exactly one canonical location for a post, and it is derivable from the address bar without a lookup table.

It also makes uniqueness free. Two posts cannot share a slug, because two documents cannot share an ID. With the query approach, nothing stops you saving a duplicate slug and quietly shadowing a live URL; limit(1) will just pick one.

The cost is real and worth naming: document IDs are immutable, so changing a slug means creating a new document and deleting the old one. That is a copy, a delete, and ideally a small redirects document mapping old slug to new so the URL does not 404. That friction is a feature. Stable URLs are the single most valuable thing you can give a content site, and a schema that makes renaming slightly annoying is a schema that stops you doing it casually.

Keep the slug in the document body too, even though it duplicates the ID. Firestore does not put the document name into the deserialised data, and having slug as an ordinary field means your model class works identically whether it came from a doc() get, a query, a REST response or a fixture in a test.

The 2MB response cap and the listing that timed out

Here is the failure. A blog post document contains its full Markdown content. My posts run 2,600 to 3,300 words, which lands each document around 20–25KB. Fine individually — the documented hard ceiling is 1 MiB per document and I am nowhere near it.

The problem is the list. An admin listing that fetched the whole posts collection was pulling 114 × ~22KB ≈ 2.5MB in a single response. It did not fail cleanly. It hung, then eventually errored, and the error said nothing useful about size. The per-document limit is the one that is documented and the one you will never hit by accident; the one that actually bites is the ceiling on a single query response, which in practice you start feeling around 2MB.

There are two fixes and you want both.

Fix one: project the fields you actually need

A listing shows a title, a slug, a date and a status. It does not show 3,000 words of Markdown. Firestore can return a subset of fields, and the saving is proportional — four small fields instead of one 22KB blob is a ~50× reduction in payload for the same number of billed reads.

The catch that cost me an afternoon: select() is not in the Firestore client SDKs. It exists in the server SDKs and in the REST API, and it is simply absent from the Dart/Flutter, Web and mobile clients. If you are calling from a Flutter app there is no projection to reach for.

Over REST it is a structured query with a select:

{ "structuredQuery": {    "from": [{ "collectionId": "posts" }],    "select": { "fields": [      { "fieldPath": "title" }, { "fieldPath": "slug" },      { "fieldPath": "status" }, { "fieldPath": "publishedAt" }    ]},    "orderBy": [{ "field": { "fieldPath": "publishedAt" }, "direction": "DESCENDING" }],    "limit": 50 } }

For a single document it is a field mask on the URL:

curl "https://firestore.googleapis.com/v1/projects/$PROJECT/databases/(default)/documents/posts/$SLUG?mask.fieldPaths=title&mask.fieldPaths=excerpt&mask.fieldPaths=updatedAt"

I use exactly this in the Python scripts that seed content and regenerate the sitemap. They need every post's slug and updatedAt and nothing else, and projecting turns a multi-megabyte pull into a few tens of kilobytes.

Fix two: stop putting the body in the list document

Projection is a patch. The structural answer is that a document should be small enough that fetching a page of them is never a question, which means the big field does not belong on the document you list.

Two shapes work. Split the body into a sibling document — posts/<slug> for metadata, postBodies/<slug> for the Markdown — so the list reads only small documents and the detail page does one extra get it was going to do anyway. Or put it in a subcollection, posts/<slug>/body/main, which keeps the two together and lets rules inherit naturally.

Either way, the list query never touches the heavy field. On the client, where you have no select(), this is the only real fix. The rough threshold I use: if the field is over about 5KB and is not rendered in any list, it belongs somewhere else.

Subcollection or array field?

This is the modelling decision people agonise over, and it has a short rule: an array field when the collection is bounded, small, and always needed with its parent. A subcollection when it is unbounded, independently queryable, or big enough to threaten the document.

tags is an array. Every post has exactly nine, they are always rendered with the post, and array-contains filters them directly. Making tags a subcollection would mean a second read per post to render a card, for zero benefit.

Comments, revisions, or per-user progress would be subcollections. They grow without bound, you want to query and paginate them on their own, and — critically — writing one element of an array rewrites the entire document. arrayUnion is convenient but it is still a full document write, it still re-derives every index entry, and it still runs into the one-write-per-second-per-document guidance. A hundred people appending to an array on one document is a hotspot; a hundred people writing separate subcollection documents is not.

The limits that decide the boundary in practice:

There is a third shape people forget: a map field. When keys are known and few — social links, download URLs per platform, a stats block — a map is better than an array of objects, because you can index and query links.github as a real field path, and you can update one key with dotted-path notation without rewriting siblings.

The composite indexes you didn't know you needed

Firestore indexes every field automatically, which is exactly generous enough to lull you. Those single-field indexes cover an equality filter, or a range, or an orderBy — on one field. Combine two and you need a composite index that does not exist yet.

My public blog feed does precisely that:

Query<Map<String, dynamic>> query = _col(Collections.posts)    .where('status', isEqualTo: PublishStatus.published);if (tag != null && tag.isNotEmpty) {  query = query.where('tags', arrayContains: tag);}query = query.orderBy('publishedAt', descending: true).limit(limit);

That is two composites, not one, and both are checked into firestore.indexes.json:

{ "collectionGroup": "posts", "queryScope": "COLLECTION",  "fields": [ { "fieldPath": "status", "order": "ASCENDING" },              { "fieldPath": "publishedAt", "order": "DESCENDING" } ] },{ "collectionGroup": "posts", "queryScope": "COLLECTION",  "fields": [ { "fieldPath": "status", "order": "ASCENDING" },              { "fieldPath": "tags", "arrayConfig": "CONTAINS" },              { "fieldPath": "publishedAt", "order": "DESCENDING" } ] }

The tag-filtered variant is the one you forget. It is a different query with a different field combination, so it is a different index, and it only breaks when a user clicks a tag chip — a path that is easy to miss in testing.

Three things I now treat as non-negotiable:

There is an escape hatch worth knowing about, and I used it for a while. If you filter with a single equality and sort client-side, you only touch the automatic single-field index and never need a composite at all. That is a legitimate trade for a collection of 32 games, where you fetch all of them anyway. It is the wrong trade for 114 posts, because sorting client-side means downloading everything to sort it — which is how you get back to the 2MB problem.

Exempt the fields you never query

The counterpart to adding indexes is removing them. Every field is indexed by default, including a 22KB Markdown blob that no query will ever filter or sort on. That costs storage and it costs write throughput on every save.

Single-field index exemptions fix it: exempt content (and excerpt, and any long text) from single-field indexing. Nothing about your reads changes, writes get cheaper, and you claw back headroom against the 40,000-entries-per-document limit. It is the least glamorous line in firestore.indexes.json and one of the highest-leverage.

Reading lists without paying twice

Cursors, not offsets

Firestore has offset(), and it is a billing trap: you are charged for every document the offset skips. Page 10 of a 20-item list bills you for 200 reads to show 20. Offsets look like SQL and cost like a full scan.

Cursors are the real mechanism. Keep the last DocumentSnapshot of the page and pass it back:

class PostsPage {  final List<Post> posts;  final DocumentSnapshot<Map<String, dynamic>>? cursor;  final bool hasMore;}Query<Map<String, dynamic>> q = /* ... */ .limit(limit);if (startAfter != null) q = q.startAfterDocument(startAfter);

Two details that matter. hasMore is docs.length == limit — a full page means there may be more, and it costs nothing to compute, whereas asking "how many are left" costs a whole extra query. And the cursor must come from a query with the same orderBy; a snapshot from a differently-ordered query will position wrongly rather than error. If you cannot hold a snapshot — across a page reload, say — startAfter with the raw sort values works too, as long as you include enough fields to be unique.

Listeners versus one-shot gets

snapshots() is the default in every Firestore tutorial and the wrong default for a content site. A listener bills the full result set on attach, then one read per changed document, and it holds a connection open for the lifetime of the widget. There is also a rule that catches people out: a listener disconnected for more than about half an hour re-reads its entire result set on resume. Background a tab for lunch, come back, pay for the whole list again.

My blog publishes twice a week. A listener on that feed would spend its entire life waiting for something that is not going to happen. So the public read paths are one-shot get() calls returning a Future, and the app refetches when the user navigates. Listeners are reserved for where live updates are actually the point: the admin panel, the leads inbox, the profile document that the whole shell rebuilds from.

The tiebreaker question is not "is this data dynamic?" — everything is dynamic eventually. It is "does a change need to reach a screen someone is currently looking at, without them acting?" For a blog post, no. For an inbox, yes.

Why "count the collection" is usually the wrong instinct

Aggregation queries mean count() no longer downloads documents, and it is cheap: billed as one document read per 1,000 index entries matched, minimum one. It is genuinely fine — I use it in the admin dashboard.

But reach for it and you have usually already made a modelling mistake, because a count is a derived value, and derived values belong on a document. If a number is shown on every page load, keep a stats/site document with postCount, toolCount, gameCount, updated with FieldValue.increment(1) in the same batch that creates the entity. One read for the whole dashboard, always consistent because it is written in the same atomic operation.

Also worth knowing before you lean on it: a filtered count() still needs the same composite index the equivalent query would, it has its own execution timeout on very large result sets, and it counts index entries rather than documents — which for array fields is not the same number.

Most of the time the honest answer is that nobody needed the count. "114 posts" in a header is decoration. Pagination is better served by hasMore than by a total, and "showing 9 of 114" is a design choice you can simply not make.

Key takeaways

The through-line is that Firestore is not a relational database with a different syntax; it is a key-value store with indexes bolted on, and it rewards schemas that make every read a direct lookup of something already in the shape you need. Denormalise, key by slug, keep the heavy fields out of the list, index deliberately, paginate with cursors, and the read path stops being something you have to think about at all.