Treat your CI/CD pipeline like a product: defend build time, tier checks pre merge vs post merge, kill flaky tests, and track DORA metrics in GitHub Actions.
"Is CI broken or just slow again? I've been waiting 22 minutes to find out if my one-line fix passes." A senior engineer pinged me that on a Thursday afternoon, and I read it three times before I understood what I was looking at. He wasn't reporting a bug. He was filing a support ticket. My CI pipeline had users, and they were miserable, and I had never once thought of them that way.
For years I'd treated the pipeline as plumbing — fix it when it clogs, ignore it when it flows, never think about it otherwise. That's exactly how you end up with the software equivalent of a government office: technically functional, universally dreaded. A CI/CD system has users. They have expectations, a tolerance for latency, a nose for flakiness, and a talent for routing around anything that slows them down. If you never talk to them or measure their pain, they suffer in silence and then quietly stop trusting the thing. This post is about treating the pipeline like what it actually is — a product I own, with an SLA, a roadmap, and a backlog — instead of a pile of YAML I only touch when it screams.
The users of your CI system are your own engineers, plus a few you might forget: the release manager waiting on a green build, the reviewer who needs a preview deploy, the on-call engineer bisecting a bad commit at 2am, and future-you trying to understand why a check exists. Every one of them interacts with the pipeline the way a customer interacts with a product. They form opinions. They complain in Slack. They develop workarounds.
Here's the tell that your continuous integration pipeline is an unowned chore: nobody can answer "how long should a build take?" because nobody decided. It's just however long it happens to be today. On a Flutter project last year, our main-branch build had crept to 19 minutes because every few months someone bolted on another job — a new lint, another integration suite, a coverage upload — and nobody ever subtracted anything. Each addition was individually reasonable. The sum was a product nobody would have shipped on purpose.
So I started running the pipeline like a product. That means three habits I stole directly from building apps:
That reframe sounds soft, but it changes every downstream decision. A chore gets the minimum effort that keeps it from failing loudly. A product gets a budget, a quality bar, and someone who cares whether its users are happy. The rest of this post is what that ownership actually looks like in practice.
The single number that shapes how engineers feel about CI is time-to-signal: how long from git push until you know pass or fail. Treat it exactly like you'd treat p95 latency on an API endpoint. Set a budget, defend it, and treat regressions as bugs.
My rule of thumb, learned the hard way:
The goal isn't a heroic 90-second build. The goal is to know your budget and refuse to blow it silently. When our Flutter build crossed 15 minutes, I did what I'd do with a slow endpoint — I profiled it instead of guessing. The wins were boring and enormous:
flutter pub get plus a Gradle dependency download was eating four minutes on every run for zero reason. Caching is the single highest-leverage CI optimization most teams never bother to measure.Here's the shape of that in a GitHub Actions workflow — nothing clever, just ordered and cached:
jobs: quick-checks: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: subosito/flutter-action@v2 with: { channel: stable, cache: true } - run: flutter pub get - run: dart format --set-exit-if-changed . - run: flutter analyze test: needs: quick-checks # don't test code that won't lint runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: subosito/flutter-action@v2 with: { channel: stable, cache: true } - run: flutter pub get - run: flutter test --concurrency=4 --coverageThe needs: quick-checks dependency is a product decision, not a technical one. I'm saying: your fastest feedback should come first, and I won't spend six minutes of your time confirming a mistake a linter caught in twenty seconds. cache: true on the Flutter action is doing quiet, load-bearing work here — it's the difference between reinstalling the SDK on every run and reusing it.
One more thing I've learned to measure: queue time is part of time-to-signal. If your build runs in 6 minutes but sits in a runner queue for 8, your users experience a 14-minute pipeline. When I profile CI latency, I profile from git push, not from "job started." The engineer waiting on the result doesn't care where the delay lives.
A big reason pipelines feel slow is that everything gets dumped into one bucket. The fix is to decide, deliberately, where each check earns its place. I think about three tiers, cheapest and fastest first.
Formatting, import sorting, obvious lint. Fast, deterministic, no network. I keep this thin on purpose — a slow pre-commit hook is one engineers disable, and a disabled check protects nothing.
# .pre-commit-config.yamlrepos: - repo: local hooks: - id: dart-format name: dart format entry: dart format --set-exit-if-changed language: system types: [dart]
The real gate. Analyzer, unit and widget tests, build verification. This is what blocks merge, so it has to be both trustworthy and fast. Every minute here is a minute multiplied across every PR every engineer opens all year. A 2-minute regression in the merge gate isn't a 2-minute problem — it's 2 minutes times hundreds of runs a month.
The expensive, slow, or flaky-prone stuff that shouldn't hold a PR hostage: full integration suites, end-to-end tests against a real device farm, performance benchmarks, deploys to staging. If one of these breaks, it's a fast follow, not a blocked engineer.
The mistake I made for years was treating every check as pre-merge. It felt safer. It was actually worse — it made the merge gate so slow that people batched huge PRs to amortize the wait, which made reviews harder and bugs bigger. Moving the 8-minute integration suite to post-merge cut our merge-gate time nearly in half, and we caught the rare integration break minutes after landing instead of before. That tradeoff was worth it every single time.
The mental model that unlocks this: not every check needs to run at the same moment, and "block the merge" is the most expensive slot you have. Spend it only on checks that are fast, trustworthy, and catch bugs a human review plausibly wouldn't. Everything else moves left (to the laptop) or right (to post-merge).
A flaky test — one that passes and fails on identical code — is not a minor annoyance. It is an outage of your feedback system. It teaches engineers the most dangerous lesson in software: that red doesn't mean broken. Once "just hit rerun" becomes muscle memory, a real failure sails straight through, because nobody believes the pipeline anymore.
I treat flakiness as a first-class incident:
main, where the code is known-good. If main goes red for any reason other than a genuine regression, that's a flake, and I log it. A pipeline that's green 99.5% of the time on known-good code has a 0.5% flake rate. If that number climbs past 2 to 3%, I treat it like a rising error rate on a production service.Those two Flutter-specific culprits are worth naming because they're so common. Shared emulator state is a test-isolation problem — the fix is a fresh emulator or a cleared collection per test, not a rerun. Unsettled animations are a timing problem — await tester.pumpAndSettle() instead of a bare pump() resolves most of them. Both are ordinary bugs. Retrying them just means the bug fails half as often and hides twice as well.
The payoff is cultural, and it's the whole reason I bother. When red reliably means broken, engineers respond to red. When it doesn't, a green pipeline is just a random number generator that occasionally blocks your merge. Trust in the signal is the actual thing I'm shipping. The tests are just how I manufacture it.
Here's an uncomfortable truth: most CI config is write-only. One person wrote it, it works, and everyone else is terrified to touch it. That's a bus-factor-of-one dressed up as automation. If your pipeline is a product, its source has to be as readable as your app's source, because the whole team maintains it.
A few rules that have paid off for me:
test and build-android beat job1 and job2. When a step fails at 2am, the name is the error message the on-call engineer reads first.Makefile the pipeline calls. My favorite trick: the CI config should mostly call the same commands a developer runs locally. If make test is what runs on your laptop and in CI, there's one thing to debug, not two — no more "works on my machine, red in CI" mysteries born of two subtly different command invocations.--concurrency=4 is there because higher numbers OOM the runner. Write that down.# One source of truth. Local and CI run the exact same commands.format: dart format --set-exit-if-changed .analyze: flutter analyzetest: flutter test --concurrency=4 --coverageci: format analyze test
When the pipeline calls make ci, the YAML shrinks to orchestration — checkout, cache, call the target — and the actual logic lives somewhere a developer can run and read. The day a junior engineer fixed a CI break himself, without pinging me, I knew the readability work had paid for itself. That's the real test of pipeline-as-code: can someone who didn't write it debug it, locally, without you in the room?
Most of the value of a pipeline comes from making the common case effortless. Open a PR, checks run, it goes green, you merge. That's the golden path, and 95% of changes should never leave it. If the happy path is smooth, nobody resents the guardrails.
But real projects have weird cases, and a product that pretends they don't just gets bypassed. A hotfix at midnight when a non-critical check is timing out. A docs-only change that doesn't need the full test matrix. A dependency bump you need to land to unblock everyone. If there's no legitimate escape hatch, engineers invent illegitimate ones — force-pushes, admin-merge, disabling checks in a panic and forgetting to re-enable them.
So I build the escape hatches deliberately, and I make them loud:
README.md doesn't trigger a device-farm run. The golden path itself should be smart about what a change actually needs:on: pull_request: paths-ignore: - '**/*.md' - 'docs/**'
The difference between an escape hatch and a hole in the fence is whether it's designed. Designed ones keep you honest. Undesigned ones become the new normal, and six months later your "required" checks are theater.
You can't run a product on vibes, and you can't run a pipeline on them either. I ignore most CI dashboards and watch four numbers. These map cleanly onto the DORA metrics — deployment frequency, lead time, change failure rate, and mean time to recovery — but I frame them as questions about my users' experience.
| Metric | The question it answers | My target |
|---|---|---|
| Time-to-signal (p95) | How long until an engineer knows pass/fail? | Under 10 min on PRs |
| Pipeline success rate on main | Does green actually mean green? | Above 97% (flake budget) |
| Change failure rate | How often does a merged change break something? | Under 15% |
| Time-to-recovery | When main goes red, how fast is it green again? | Under 30 min |
The one people skip is time-to-recovery, and it's the most revealing. A pipeline that fails occasionally but recovers in ten minutes is healthy. One that's usually green but stays broken for three hours when it breaks is quietly rotting — it means nobody owns the red, and everyone's blocked while they wait for someone to notice. I'd rather have a pipeline that breaks twice as often and heals five times as fast.
I review these monthly, the same way I'd review product metrics. Not to hit arbitrary numbers, but to catch trends. A slow, steady climb in time-to-signal is the pipeline telling you it needs maintenance before an engineer has to tell you in Slack. A rising change failure rate says the pre-merge gate is letting real bugs through — either it's too thin, or it's too flaky to trust. The numbers are a conversation with your users that happens before the angry ping does.
A product has scope, which means it also has an explicit "not doing this" list. Automation has a cost — to build, and worse, to maintain — and some things aren't worth paying it. Restraint is a feature.
git push, queue time included — or the feedback loop is broken and bugs land late.make targets locally and in CI, name jobs like functions, and comment the why.The pipeline isn't the chore. Neglecting it until it breaks — that's the chore. Ship it like you mean it.