Cut cloud observability cost to near zero: structured logs, low cardinality metrics, tail sampling, retention design, and free tier Loki/Prometheus/Grafana.
A client once forwarded me their observability invoice with a two-word subject: "is this real?" Just under $2,100 for one month, on a product with maybe four thousand daily users — roughly triple what the app itself cost to run. They were paying more to watch the system than to run it, and they had no idea how it got that high.
That is the trap, and almost everyone walks into it the same way: by treating observability as a thing you buy instead of a thing you design.
Here is my actual position after six years of shipping Flutter apps and the Firebase and Google Cloud backends behind them: you can have real logs, real metrics, and real distributed traces for a side project or an early-stage startup and pay something that rounds to zero. Not a toy version. The real thing — enough to debug a production incident at 3am. The catch is that you don't get there by finding a cheaper monitoring vendor. You get there by being ruthless about two things: cardinality and sampling. Everything below is downstream of those two words.
The default observability stack is priced against enterprises and sold to everyone. The pricing models are almost all variations on the same theme: you pay per gigabyte ingested, per unique time series, per event, per seat, per host. None of those meters scale with how useful the data is. They scale with how much of it you throw at the wall.
The failure mode is boring and predictable. You ship, you add a logging SDK, you accept the defaults, and the defaults are generous because the vendor's revenue is your volume. Every request logs a fat JSON blob. Every metric carries a user_id label. Nobody sets retention, so 90 days of debug logs sit in hot storage forever. Six months later the bill has a comma in it and nobody can explain which line item is load-bearing.
The uncomfortable truth: most of that telemetry is never read. On that client's project we pulled the query logs from their observability tool — the actual dashboard and search access patterns — and something like 95% of ingested log volume was never queried by a human or an alert. They were paying to store evidence for a trial that never happened.
So the first move isn't technical. It's a mindset shift: telemetry is a cost center you are actively managing, not a safety blanket you accumulate. Every field, every label, every log line should justify its existence by answering a question you will actually ask.
When something is broken in production and you're half awake, you are not browsing dashboards for fun. You are trying to answer three questions, in order:
Everything you collect should earn its place by helping answer one of those three. That's the filter. A metric that doesn't help you decide is it broken is decoration. A log field that doesn't help you locate where or what changed is noise you're paying to store.
I keep this list literally pinned in my notes, because in the calm of a Tuesday afternoon it's tempting to instrument everything. Then a real incident hits and you realize you have ten thousand data points and none of them tell you whether checkout is down. Design backward from the 3am questions and the volume problem mostly solves itself. This is also the cleanest way to think about the difference between monitoring and observability: monitoring answers question one on dimensions you predicted in advance, and observability is having enough high-quality signal to answer questions two and three that you didn't predict.
If you do one thing for cheaper observability, make your logs structured. Not prose. JSON with consistent keys. This single change is the highest-leverage, lowest-cost move in the entire discipline, because structured logs are simultaneously your logs, a decent chunk of your metrics, and the raw material for cheap traces.
A log line like this is nearly useless at scale:
[2026-06-14 09:12:44] Payment failed for user in checkout
You cannot aggregate it, alert on it, or correlate it. Compare:
{ "ts": "2026-06-14T09:12:44Z", "level": "error", "event": "payment_failed", "trace_id": "8f3c2a1b", "route": "/checkout", "gateway": "stripe", "error_code": "card_declined", "latency_ms": 812, "region": "me-central1"}Now it's queryable. "Show me payment_failed grouped by error_code in the last hour" is a one-liner. And critically, you can derive metrics from logs instead of emitting a separate metric for every dimension — which is exactly where metric bills explode.
The rules I hold to:
event: "payment_failed", not free text. Events are cheap to count and easy to alert on.user_id is a value in the payload, never a metric label. The distinction matters and I'll get to why in the metrics section.trace_id on everything. It costs almost nothing and it's the thread that stitches a request together across services.error and warn are cheap because they're rare. info is your steady heartbeat. debug is off in production by default and toggled per-request or per-tenant when you're chasing something.Here's the pattern I use in Dart backends and Cloud Run services — one place that owns the log shape, so nothing downstream has to think about it:
void logEvent( String event, { String level = 'info', Map<String, Object?> fields = const {},}) { final record = <String, Object?>{ 'ts': DateTime.now().toUtc().toIso8601String(), 'level': level, 'event': event, 'trace_id': Zone.current[#traceId], ...fields, }; stdout.writeln(jsonEncode(record));}That's it. Write JSON to stdout, let the platform ship it. On Cloud Run and most container hosts, stdout is already collected for free up to a generous quota, and Google Cloud Logging automatically parses each line into a structured entry you can filter on. You didn't buy anything.
One edge case worth planning for: don't log secrets, tokens, or full request bodies into these events. A structured log is easy to query, which means a leaked field is easy to find — by you and by anyone who gets read access. Whitelist the fields you emit rather than dumping whole objects, and you sidestep both a security problem and a size problem at once.
Cardinality is the word that should be tattooed on the inside of every engineer's eyelids before they touch a metrics system. Cardinality is the number of unique combinations of label values on a metric. It is the single variable that turns a $0 setup into a $2k one.
A metric like http_requests_total with labels {route, status} is fine. If you have 20 routes and 5 status classes, that's 100 time series. Cheap. Now someone adds user_id as a label "for debugging." You have 4,000 users. That's 400,000 time series — and it grows every time a new user signs up. Prometheus, or any time-series database, stores each series independently in memory and on disk. You just multiplied your storage and memory footprint by 4,000, and you did it with one well-intentioned line. This is the classic cardinality explosion, and it is responsible for more surprise observability bills than any other single mistake.
The rule is simple and non-negotiable:
Labels are for bounded, low-cardinality dimensions. Everything unbounded goes in a log, not a label.
Bounded and safe: route, method, status_class, region, gateway, deploy_version. These have a small, known set of values.
Unbounded and forbidden as labels: user_id, email, request_id, trace_id, session_id, raw URLs with IDs in them, full error messages. Anything user-generated. Anything with an ID. And watch the sneaky ones — a path label that includes /orders/98213 is unbounded in disguise, because every order id mints a new series. Normalize it to the route template /orders/{id} before it becomes a label.
When you genuinely need to slice by a high-cardinality thing during an incident, that's what your logs are for. You query the logs for that one user's trace_id. You do not build a permanent time series for every user who has ever existed.
For each service, I emit exactly three families of metrics — the RED method:
That's it per service. Three metrics, a handful of bounded labels each. From those you can build almost every alert and dashboard that matters, and the whole thing fits inside a free-tier Prometheus or a tiny self-hosted instance without breaking a sweat. Latency histograms are the one place I'll spend a little more cardinality, because bucketed p95/p99 is worth it — but even then the buckets are bounded and defined up front.
If you run infrastructure with queues and workers rather than plain HTTP, the sibling of RED is the USE method (Utilization, Saturation, Errors) for resources. Same spirit: a tiny, fixed set of signals per thing, chosen deliberately, rather than a metric for every field you can reach.
Tracing is where teams either get enormous value or burn the most money, and the difference is entirely about sampling. A distributed trace is a record of one request's journey across your services. Detailed, correlated, and expensive at 100% — because you're storing a full call tree for every request, including the millions that are boring and identical.
Nobody needs a trace of every successful health check. So don't keep them.
The two sampling strategies, and when to use each:
Tail sampling is the whole game for cost-effective tracing. You keep every trace you'd actually want at 3am — the slow ones, the failed ones, the weird ones — and throw away the overwhelming majority that are just proof the system works. The volume drops by 90%+ and you lose essentially nothing you'd ever query.
The OpenTelemetry Collector does tail sampling out of the box, and it's the piece I'd install first. A minimal policy:
processors: tail_sampling: decision_wait: 10s policies: - name: keep-errors type: status_code status_code: { status_codes: [ERROR] } - name: keep-slow type: latency latency: { threshold_ms: 1000 } - name: sample-the-rest type: probabilistic probabilistic: { sampling_percentage: 1 }Errors and slow requests: kept, always. Everything else: 1%. Your trace storage bill is now a rounding error, and the traces you kept are the only ones you'd ever open. One operational caveat: tail sampling needs to buffer all spans of a request until it can make a decision, so with multiple Collector replicas you either route by trace id to a single instance or run a small dedicated sampling tier. On a project this size a single Collector handles it comfortably.
Standardizing on OpenTelemetry here matters beyond the sampling. It's vendor-neutral instrumentation, so the same traces, metrics, and logs can point at a self-hosted backend today and a managed one later without touching application code. You avoid getting locked into the pricing model that started this whole mess.
One more thing that costs nothing: propagate the trace_id into your structured logs (you saw it in the log schema above). Now a single ID connects the metric that alerted, the trace that shows the path, and the log line with the actual error. That correlation is what expensive platforms sell as a premium feature. You just built it with a shared string.
You don't need a vendor to get all three signals. Here's what I actually reach for, depending on how allergic to ops the project is.
The fully self-hosted, near-zero-cost stack:
| Signal | Tool | Why it's cheap |
|---------|-----------------------------|---------------------------------------------------|
| Logs | Grafana Loki | Indexes labels, not log content — storage is tiny |
| Metrics | Prometheus + VictoriaMetrics | RED-method metrics fit in a small instance |
| Traces | Grafana Tempo or Jaeger | Object-storage backed, tail-sampled upstream |
| Views | Grafana | One pane over all three, free and open source |
Grafana Loki deserves a specific callout because its design is the cost strategy. Traditional log systems index the full text of every log — expensive. Loki indexes only a small set of labels and stores the raw log content compressed in object storage (S3, GCS, Cloudflare R2, whatever). Object storage is roughly two cents per gigabyte per month. You can keep a lot of logs for the price of a coffee, as long as you keep your label cardinality low — the same discipline as metrics, applied again. Loki will happily let you blow up its index with a high-cardinality label too, so the rule from the metrics section travels with you.
For a small project I'll run this whole stack on one modest VM plus a cheap object-storage bucket. Under real load — a few thousand users, a few hundred requests a second in bursts — it holds up fine, because the tail sampling and low cardinality mean the actual data volume is modest even when traffic isn't.
The I-don't-want-to-run-anything option: lean on the free tiers.
info-level noise you'll never read.My honest default for a bootstrapped product: cloud-native free tier for logs and metrics because they're already there and already free, plus a self-hosted or free-tier tracing backend fed by an OpenTelemetry Collector doing tail sampling. Total monthly cost: the object-storage bucket and a small VM if you self-host tracing. Single-digit dollars, often less.
The most expensive word in observability is "forever." Almost nobody sets retention deliberately, so everything defaults to keeping data far longer than anyone will ever look at it. Retention is where you claw back the last big chunk of cost, and it's free to configure.
Different signals have different useful lifespans. I set them on purpose:
The pattern: the rawer the data, the shorter it lives; the more aggregated, the longer. Raw logs are voluminous and perishable. Aggregates are tiny and durable. Match retention to that shape and your storage bill stops being a growing liability and becomes a flat, predictable line. A tiered object-storage lifecycle policy (hot bucket for recent data, cold/archive class for the rest, auto-delete past the window) automates the whole thing so you're not remembering to prune by hand.
Two hard-won opinions to finish.
On alerts: the only alerts I keep are ones that are symptom-based and actionable. An alert should fire on something a user feels — error rate above X, p99 latency above Y, a queue backing up, a job that didn't run — and it should mean a human needs to do something now. If an alert fires and the honest response is "yeah, that happens sometimes," it's not an alert, it's a notification, and it's training you to ignore the real ones. I deleted more than half of one project's alerts in an afternoon and the on-call rotation immediately got quieter and more responsive, because the remaining pages actually meant something.
Alert on symptoms, not causes. "CPU is at 80%" is a cause and often a fine one; it's not a reason to wake someone. "Checkout error rate is 12%" is a symptom users feel, and it is. Cause-based alerts are how you end up with 40 alerts and alert fatigue. A handful of good symptom alerts, tied straight to the "is it broken" question, beat a wall of them. If you formalize this, it's the SLO/error-budget approach: define what "good" means for the user, alert when you're burning the budget too fast, and ignore everything that doesn't move that number.
On dashboards: I've deleted more dashboards than I've built, and every deletion was an improvement. The failure mode is the "wall of graphs" — 30 panels nobody looks at until an incident, and during the incident they're worse than useless because you're scanning noise. I keep two kinds:
Everything else — the vanity panels, the "might be interesting someday" graphs — I delete. A dashboard you don't trust during an incident is a dashboard that's costing you attention when you have the least of it to spare.
trace_id.Do this and your observability bill rounds to zero while your ability to debug production actually goes up — because you're drowning in less noise and standing on signal you chose on purpose.