Rotate Keys Without the 2 A.M. Outage

Rotate secrets without downtime: use short lived identity based credentials, a TTL cached secret accessor, and an additive overlap window runbook for GCP and AWS.

The honest reason most teams don't rotate secrets is that rotation feels like defusing a bomb. Somewhere in the codebase a key is baked into an environment variable at boot, three services read it, and nobody is confident which one falls over if you change it. So the key from two years ago is still live, still in someone's shell history, still in a Slack thread. I've shipped enough production systems to know: you won't rotate what you're afraid of rotating. The fix isn't discipline — it's architecture.

This post is about making secret rotation so cheap and boring that you'll actually do it on a schedule instead of only after a breach forces your hand. There are three pieces, and they build on each other: prefer credentials that expire on their own, structure the app so a secret is read fresh instead of frozen at startup, and keep a rotation runbook so the swap is muscle memory instead of a 2 a.m. improvisation. Get all three right and rotation stops being an incident and becomes a config edit.

Why secret rotation causes outages in the first place

It helps to name the failure precisely, because the fix follows directly from it. Rotation goes wrong for two structural reasons, not because someone was careless:

• The secret is captured at process boot. The value gets read once — usually from an environment variable — and frozen into a client object for the process lifetime. Changing it now requires a redeploy, and a redeploy is a moving target while traffic is live.
• Rotation is treated as an atomic swap. Old key off, new key on. In the gap between those two actions, in-flight requests authenticate with a credential that no longer works.

Every technique below is aimed at one of those two problems: either make the secret refreshable at runtime, or make the swap additive so there's never a gap. Keep that framing in mind and the rest is just mechanics.

The best rotation is the one you don't do

Before we talk about rotating long-lived keys, kill as many of them as you can. A secret you never issued is a secret you never have to rotate, leak, or audit. Reducing the count of long-lived credentials is the highest-leverage secrets-management move there is, and it happens at design time, not incident time.

Prefer identity over keys. On GCP and AWS, the biggest win is workload identity. If your service runs on Cloud Run, GKE, or an EC2 instance, it can assume a role or service-account identity and get short-lived tokens minted automatically — typically valid for an hour and refreshed under the hood by the SDK. There's no static key file to leak, no key to rotate, and no "last used two years ago" credential rotting in a bucket. In my Firebase-backed projects this matters a lot: server-side code touching Firestore or Storage should run as a service identity, not carry a downloaded service-account JSON around. The moment you download that JSON, you own a long-lived credential forever — it's exactly the kind of key that ends up committed to a repo or pasted into a CI variable and forgotten.

The same instinct applies to third-party APIs where you have a choice: OAuth access tokens over static API keys, because access tokens are short-lived by design and a refresh token is far easier to revoke than to hunt down every copy of a static key. A useful mental model: a short-lived token rotates itself every hour, so a leak has an expiry date baked in. A static key leaks forever until you notice.

A few concrete substitutions worth making before you write any rotation code:

• Downloaded service-account JSON → workload identity / attached service account. Let the runtime mint credentials for you.
• Long-lived personal access token → short-lived OAuth token with a refresh token. Revocation becomes one API call instead of a manhunt.
• Shared database password → IAM database authentication where the engine supports it, so the connection token is issued per-session and expires on its own.

When you genuinely can't avoid a long-lived secret — a partner API that only issues static keys, a legacy database password, a webhook signing secret — that's the case the rest of this post is built for.

Structure the app so the key is a config change, not a redeploy

Here's the design failure that turns rotation into an outage. This is the pattern I want you to stop writing:

// At startup — the secret is now frozen for the process lifetime.final apiKey = Platform.environment['PARTNER_API_KEY']!;final client = PartnerClient(apiKey: apiKey);

Once the process boots, that value is immutable. Rotating means a redeploy, and a redeploy while both keys are briefly valid means racing your own rollout. Instead, read the secret through an accessor that can refresh, and fetch it from a secret store (GCP Secret Manager, AWS Secrets Manager, Vault) rather than an env var:

class SecretRef {  SecretRef(this._loader, {this.ttl = const Duration(minutes: 5)});  final Future<String> Function() _loader;  final Duration ttl;  String? _cached;  DateTime _fetchedAt = DateTime.fromMillisecondsSinceEpoch(0);  Future<String> value() async {    final age = DateTime.now().difference(_fetchedAt);    if (_cached == null || age > ttl) {      _cached = await _loader();      _fetchedAt = DateTime.now();    }    return _cached!;  }  void invalidate() => _cached = null; // force refetch on next read}

Now the secret has a short cache TTL. When you rotate the value in the secret store, every instance picks up the new one within the TTL window — no deploy, no restart, no coordinated rollout. The invalidate() hook lets you force an immediate refetch: if a call returns 401/403, invalidate and retry once before giving up. That single retry converts "rotation caused a spike of errors" into "a handful of requests took one extra round trip."

Here's what wiring that retry into a request looks like in practice, so the accessor actually earns its keep:

Future<http.Response> callPartner(SecretRef key, Uri url) async {  Future<http.Response> send() async =>      http.get(url, headers: {'Authorization': 'Bearer ${await key.value()}'});  var res = await send();  if (res.statusCode == 401 || res.statusCode == 403) {    key.invalidate();   // maybe we're holding a just-rotated-out value    res = await send(); // one fresh attempt with the latest secret  }  return res;}

Two rules make this reliable:

• Never bake the secret into a long-lived client. Build the auth header per request (or per short-lived client) from SecretRef.value(). A PartnerClient that captured the key in its constructor defeats the whole thing — you're back to the frozen-at-boot problem, just hidden one layer deeper.
• Cache with a TTL, don't fetch on every call. Hitting the secret store on every request adds latency and cost, and some managed stores bill per access and enforce rate limits. A few minutes of staleness is the price of not DDoSing your own secret store. Tune the TTL to the shortest propagation window you're comfortable waiting during a rotation — shorter TTL means faster propagation but more fetches.

One subtlety worth calling out: this pattern assumes each instance holds the secret in memory only, never writes it to disk or logs. If your accessor caches to a file for "resilience," you've just created a new long-lived copy to leak. In-memory, TTL-bounded, refetched from the source of truth — that's the whole contract.

The overlap window is the whole trick

The reason rotation causes outages is people treat it as a swap: turn off old, turn on new. Between those two actions, requests fail. Correct rotation is additive first, subtractive later. You want an overlap window where both the old and new credential are valid simultaneously, so no in-flight request can land in a gap.

The generic sequence, whatever the provider:

• Create a new credential alongside the old one. Both are live. This is the additive step — nothing breaks because nothing was removed.
• Publish the new value to your secret store as a new version.
• Propagate — wait at least one cache TTL so every instance is serving the new value. Watch your 401/403 rate; it should stay flat.
• Verify the old credential is no longer in use. Most providers expose a "last used" timestamp on a key — this is your ground truth.
• Revoke/delete the old credential only after "last used" has gone quiet for a safe margin.

Skip step 4 and you're guessing. That "last used" field is the single most useful thing for rotation confidence, and I check it before deleting anything. The failure mode it protects you from is the quiet one: a batch job, a cron worker, or a rarely-hit region that only reads the secret every few hours and is still holding the old value long after your dashboards look calm.

A note on stateless vs. stateful secrets. The additive window is easy for credentials the provider validates independently — API keys, service-account keys, OAuth clients — because two of them can coexist. It's trickier for a shared symmetric signing secret, where both sides must agree on one value. For those, prefer a scheme that accepts multiple valid keys at once (a key set with an active signer plus still-accepted verifiers), so you get the same overlap window instead of a hard cutover.

A rotation runbook you can hand to anyone

Rotation should be a checklist, not a puzzle you re-solve each time. I keep one per secret, in the repo, next to the code that uses it. Here's the shape, as a bash runbook against GCP Secret Manager — the same skeleton works for AWS with aws secretsmanager and for Vault with vault kv:

#!/usr/bin/env bashset -euo pipefailSECRET="partner-api-key"# 1. Mint the new credential (provider-specific) and capture it into $NEW_VALUE.#    Do NOT revoke the old one yet — both must be valid (the overlap window).# 2. Add it as a new version. The app's SecretRef reads "latest".printf '%s' "$NEW_VALUE" | gcloud secrets versions add "$SECRET" --data-file=-# 3. Propagate: wait one cache TTL, then watch error rate stay flat.sleep 360echo ">> Check the 401/403 dashboard now. Abort if it moved."# 4. Confirm the old credential is idle in the provider console#    (its 'last used' timestamp should be older than the propagation window).# 5. Only then, disable the previous secret version and revoke the old key.gcloud secrets versions disable "$OLD_VERSION" --secret="$SECRET"# provider-specific: delete/revoke the old key

The AWS shape is nearly identical — aws secretsmanager put-secret-value to publish the new version, then aws secretsmanager update-secret-version-stage to move the AWSCURRENT stage, with AWSPREVIOUS giving you a built-in one-step rollback. The point isn't the exact CLI; it's that every provider gives you a versioned publish and a reversible disable, and your runbook should lean on both.

Three details that make this safe in practice:

• Disable before delete. Disabling a version is instantly reversible if something screams; deletion isn't. Deletion is the last thing you do, and only after the "last used" check is quiet.
• Rotation and rollback are the same procedure run in opposite directions. If step 3 shows errors, you re-enable the old version and you're back where you started, because you never destroyed anything. That symmetry is exactly why the additive-first ordering matters — it keeps rollback trivial.
• Log the rotation, not the secret. Record who rotated what and when, and the version numbers involved. Never log the value itself. An audit trail of events is what makes the next rotation faster; a logged secret is just a fresh leak.

Finally, treat leaked-secret rotation as the emergency path of this same runbook. The steps are identical — you just compress the propagation window and accept some request failures, which is exactly the tradeoff you want to make consciously, not discover mid-incident. Because you rehearse the boring scheduled version, the emergency version is the same muscle memory under pressure instead of a first attempt at 2 a.m.

Automating rotation once the fundamentals hold

Once the app reads secrets through a TTL-cached accessor and the runbook is additive, automation becomes safe rather than terrifying. Managed rotation — AWS Secrets Manager rotation schedules, or a scheduled job that runs your runbook — works precisely because every step is reversible and observable. I don't reach for automation first, though. I get one manual rotation to be genuinely boring, prove the overlap window and the "last used" check behave, and only then hand the checklist to a scheduler. Automation should encode a procedure you already trust, not paper over one you don't.

Key takeaways

• Rotation is a design property, not a chore. The two root causes of rotation outages are secrets frozen at process boot and swaps treated as atomic. Fix those structurally and the fear goes away.
• Eliminate long-lived keys first. Prefer workload identity and short-lived tokens (OAuth, IAM auth) so most credentials expire and refresh themselves — the cheapest rotation is the one you never perform.
• Read secrets through a TTL-cached accessor. Never capture a key in a long-lived client; build auth per request and add an invalidate()-then-retry on 401/403 so rotation is a config change your running fleet picks up on its own.
• Make the swap additive. Create, publish, propagate one TTL, verify the old key is idle via its "last used" timestamp, then revoke. Both credentials valid at once means no gap for requests to fall into.
• Keep a per-secret runbook in the repo. Disable before delete, treat rollback as the same procedure reversed, log the event but never the value, and reuse the same checklist as your emergency path.

Do that once per secret and the scary part disappears: you're no longer defusing a bomb, you're editing config. That's the version of secret rotation you'll actually keep doing — on a schedule, without the 2 a.m. outage.