Untested backups fail when it counts. How I run quarterly restore drills, set honest RPO/RTO, back up Firestore, Auth and Storage, and beat ransomware on a budget.
A backup you have never restored is not a backup. It is a rumor. It is a comforting story you tell yourself at 2am when everything is fine, and a story that falls apart the one time you actually need it. I learned this the way most people learn it: not from a blog post, but from a cold sweat.
So now I do something that makes new engineers on my team visibly nervous the first time they see it. Once a quarter, I schedule a window, and I delete things on purpose. Real data. In a real environment. Then I sit there and prove I can get it back. If I can't, that's not a disaster — that's a Tuesday, and I've just found the bug in my recovery instead of finding it during an actual outage. The whole point is to move the panic to a day when nothing is on fire. This is what disaster recovery testing actually looks like when you take it seriously: a controlled, deliberate rehearsal instead of a live improvisation under maximum stress.
Here is the uncomfortable truth that every backup vendor's marketing page politely avoids: taking a backup and restoring a backup are two completely different systems, and only one of them is tested by default.
Your backup job runs every night. It succeeds. You get a green checkmark, maybe a Slack message. That checkmark tells you a file was written somewhere. It tells you nothing about whether that file:
On one project I inherited a nightly export that had a perfect green record going back fourteen months. When I finally ran a restore drill, the export was missing an entire subcollection — a security rules change months earlier had quietly stripped the service account's read access to it, the export tool treated "no permission" as "no documents," and wrote a clean, complete-looking backup of the wrong shape. Fourteen months of green checkmarks. The subcollection held delivery addresses. Nobody noticed until I went looking, on purpose, on a calm afternoon. If a real incident had gone looking first, that's the kind of gap you find at the worst possible moment.
The mental model I want you to adopt: a backup is a hypothesis. "I believe I can reconstruct this system from this artifact." Every untested backup is an unfalsified hypothesis. The restore drill is the experiment. Until you run the experiment, you don't have a backup — you have a belief, and beliefs are not a recovery strategy. This is why backup validation, not backup creation, is the metric that actually correlates with surviving an outage.
Before you touch a single tool, answer two questions honestly. Not aspirationally. Honestly. These two numbers — RPO and RTO — are the foundation of every disaster recovery plan, and getting them right is what separates a cheap working strategy from an expensive fake one.
Most teams pick these numbers by vibes, and the vibes are always "zero and zero." Then they price out the infrastructure to achieve zero-and-zero — synchronous multi-region replication, hot standbys, point-in-time recovery — and quietly go back to no backups at all because it's expensive and complicated. That's the worst outcome — perfect being the enemy of existing.
Do the opposite. Start honest. For most of what my team ships, the truthful answer is: losing an hour of data is annoying but not fatal, and being down for two hours is embarrassing but survivable. That is a wildly cheaper problem to solve than "zero seconds, zero bytes," and being clear about it unlocks a zero-budget solution instead of a five-figure one.
A quick way to force the conversation with a client or your own team is to classify your data by how much it actually matters:
| Data class | Example | RPO | RTO |
| --- | --- | --- | --- |
| Critical / transactional | orders, payments, user auth | 1 hour | 1 hour |
| Important / operational | app content, user profiles | 24 hours | 4 hours |
| Reproducible | caches, derived indexes, thumbnails | N/A | rebuild |
The third row is the one people miss. A surprising amount of your "data" is derived from other data. You don't back up a cache; you back up the source and rebuild. Knowing which is which shrinks the problem enormously — often the truly irreplaceable, must-restore-to-the-minute slice is a small fraction of your total storage, and you can spend your effort and money there instead of spreading it thin over thumbnails you can regenerate in a loop.
A worked example: on one delivery app, "the database" looked like tens of gigabytes. Once we separated it, the transactional core that needed a one-hour RPO was under two gigabytes — orders, payment references, user records. Everything else was either operational content on a lazy 24-hour cadence or reproducible derived data we never backed up at all. That reframing turned an intimidating backup problem into a nightly export of one small, well-defined set of collections.
Most of my stack sits on Firebase — Firestore, Firebase Authentication, Cloud Storage for user uploads, a bit of Cloud Storage for binaries. The whole appeal of serverless is that there's no server to babysit. The flip side nobody mentions: there's also no server you can just rsync off at 3am. You back up through APIs, not filesystems.
The good news is you don't need to buy a "backup product." You need a scheduled export and somewhere durable to put it. On the free tier, that "somewhere" is a second bucket in a different location, and my rule is that the backup destination must be a resource an attacker who owns your app credentials can't casually reach.
For Firestore, the managed export is the boring correct answer. It writes a consistent snapshot to a Cloud Storage bucket:
# Nightly Firestore export to a dedicated backup bucketgcloud firestore export "gs://myapp-backups/$(date +%Y-%m-%d)" \ --project=myapp-prod \ --collection-ids=orders,users,content
Naming the collection IDs is deliberate. It documents exactly what's in the snapshot, and it stops a new collection from silently joining (or missing) the backup without anyone deciding it should. If you omit --collection-ids, you export everything — which sounds safer until a teammate adds a collection full of huge derived blobs and your export cost and duration quietly balloon.
Because I refuse to pay for Cloud Functions where a free runner will do, the schedule lives in GitHub Actions with a service account that has export permission and nothing else:
name: firestore-backupon: schedule: - cron: "0 2 * * *" # 02:00 UTC daily workflow_dispatch: # so I can trigger a drill by handjobs: export: runs-on: ubuntu-latest steps: - uses: google-github-actions/auth@v2 with: credentials_json: ${{ secrets.BACKUP_SA_KEY }} - uses: google-github-actions/setup-gcloud@v2 - name: Export Firestore run: | STAMP=$(date +%Y-%m-%d) gcloud firestore export "gs://myapp-backups/$STAMP" \ --project=myapp-prod \ --collection-ids=orders,users,content - name: Prune exports older than 30 days run: | CUTOFF=$(date -d '30 days ago' +%Y-%m-%d) gsutil ls gs://myapp-backups/ | while read p; do d=$(basename "$p") [[ "$d" < "$CUTOFF" ]] && gsutil -m rm -r "$p" doneOne quiet failure mode with any scheduled backup: the job breaks and nobody notices, because a job that doesn't run also doesn't send a failure email. Wire the workflow to alert on failure — a Slack message, an email, anything — and separately alert if no fresh backup object has appeared in the bucket for more than a day. "Did last night's backup actually run?" should be a question your monitoring answers, not one you ask after an incident.
Two more things that aren't in the happy-path tutorials, and each one is a way to think you have a backup when you don't:
firebase auth:export, and treat the resulting file as a secret — it contains password hashes.gsutil -m rsync to a versioned backup bucket covers it.The pattern generalizes far past Firebase. Whether you're on Firebase, Supabase, AWS with DynamoDB plus S3 plus Cognito, or any other serverless stack, "serverless" means your backup is an orchestration problem across three or four separate services, not one dump of one database. Write down every service that holds state. The one you forget is the one that ends your weekend.
This is the part people skip, and it's the only part that actually matters. Everything above is setup; the restore drill is where you find out whether any of it works.
A restore drill is simple to describe and uncomfortable to do. You provision a throwaway project — myapp-restoretest — and you rebuild production inside it from last night's artifacts. No shortcuts, no reaching into the real database for the missing piece. If the runbook doesn't produce a working system, the runbook is wrong, and better to learn that now.
# Restore last night's export into an isolated projectgcloud firestore import "gs://myapp-backups/2026-06-30" \ --project=myapp-restoretest
Then I check three things, in order. Each one is a strictly higher bar than the last, and skipping ahead is how "the restore worked" turns into an outage:
The deleting-on-purpose part fits here. In the isolated project, I delete a collection, a user, a storage object — then restore just that slice. Selective restore is a different, harder skill than full restore, and it's the one you'll actually use, because real incidents are usually "someone nuked one collection" or "a migration corrupted one field on every user," not "the datacenter is on fire." Practicing only the full-restore path leaves you improvising the surgical one when it counts.
Put the drill on the calendar as a recurring event with a named owner. "We should test restores sometime" is how you end up never testing restores. A scheduled 90-minute block every quarter, owned by a person, is how it actually happens. I also fail the drill on purpose sometimes — hand it to a teammate who didn't write the runbook and watch where they get stuck. Those stuck points are the bugs, and they're free to fix on a calm afternoon and brutally expensive to fix at 3am.
There are two shapes of disaster, and they need different defenses. Confusing them is how teams build a backup that survives the common case and evaporates in the case that actually threatens the business.
The fat-finger is accidental and one-directional. Someone runs a delete against prod thinking it's staging. A migration script has an off-by-one and wipes half a collection. Yesterday's backup fixes this completely, because the "attacker" is a well-meaning colleague who is not also deleting your backups.
Ransomware and a compromised credential are adversarial. Whoever's in there wants your recovery to fail. The first thing capable attackers do is find and destroy backups, because a company that can restore doesn't pay. If your backup bucket is writable by the same service account that runs your app, and that account leaks, your backups are just more things the attacker gets to delete. Any backup strategy that assumes the attacker won't touch the backups is not a ransomware defense at all.
This is where immutability and blast-radius separation earn their place, even on a budget:
You don't need all of it on day one. But the moment you hold anything an attacker would ransom, "the backup is in the same account with the same keys" is not a backup strategy. It's a single point of failure with extra steps.
Here's the failure I didn't see coming the first time. We restored the data perfectly, pointed a build at it, and the app still wouldn't start — it sat spinning on a config value that lived only in a dashboard nobody had ever exported. Twenty minutes of a "successful" restore, and the product was still dark. A database is maybe half of a running system. The rest is scattered in places nobody backs up, and no amount of Firestore export covers it.
Things that are not your database and will absolutely take you down:
My fix is a boring principle: everything that isn't user data should be reproducible from git. Infrastructure as code, security rules in the repo, config committed (secrets referenced, not committed). If rebuilding the environment means clicking through a console from memory, you don't have a recovery plan — you have a memory test, and you'll take it under maximum stress. The restore drill is also where you catch this, because a drill in a clean project forces you to produce every config value and rule from source rather than borrowing them from the still-running production you're pretending is gone.
The last piece is a document written for one specific reader: you, at 3am, adrenaline-soaked, having just realized production is gone. That person is not clever. That person cannot improvise. That person needs a checklist with exact commands they can paste.
A recovery runbook that works has these sections, and as little prose as you can get away with:
Keep it somewhere reachable when your infrastructure is down — which means not only in the wiki that's hosted on the thing that just died. A copy in git, a copy in a doc on a different provider, a printout in a drawer if you're paranoid. I am a little paranoid.
The test of a good runbook is simple: hand it to the newest person on the team and have them run a restore drill from it, with you silently watching. Every question they ask is a gap. Fix the gaps while it's a drill.