The dashboard says we ship faster now. More deploys per week, less time from ticket to production.
What it doesn’t say is what we stopped doing: tests, reviews, alerts that were “noisy”. And a “limited” release that has been limited for three months.
If you measure speed up to the deploy and leave out what happens after, you can present work pushed into the future as an improvement.
#.Not every deleted test is a mistake
Removing unnecessary tests can improve the suite. What matters is distinguishing what we’re removing.
- Repeats coverage or checks removed behavior
- It may be unnecessary
Check that it adds no necessary coverage.
- Fails intermittently
- Diagnose first
It may be badly written or catching a real race condition.
- Breaks when an internal function is renamed
- Check what it tests
It may be coupled to the implementation. Rewrite it while keeping the coverage.
Some tests are the scar of an incident. Delete them without keeping that coverage, and you can lose what the team learned.
Before removing one, ask: what failure would it let through, and where would that be covered instead? If nobody can answer, it’s not time to delete it yet.
#.Work that leaves the pipeline can come back another way
Removing a useful check without replacing it may save time on the deploy and cost it later in manual review, support or incidents.
The senior who “always checks before we ship”. Friday’s manual test list. The support person who already knows what to ask the customer.
That work doesn’t show up on the dashboard. It shows up on someone’s calendar. You can also accept the risk without adding reviews. But accepting it isn’t the same as removing it.
#.Without observability, “it works” is an opinion
Turning off a noisy alert can be right. If it was the only way to detect a failure, you need another signal.
Without one, you find out from the customer. By then you’re measuring your users’ patience.
Not having seen an error is not the same as not having one.
#.Limited release: to learn, or to hide?
Releasing to a small group can help you learn and contain risk. It can also be a complete delivery if that was the agreed scope.
Make three things clear:
- What you want to learn, or what risk you want to contain.
- What signals will let you evaluate it.
- Whether the limit is permanent; if not, when you’ll revisit it.
There’s another version: with no agreed small scope or plan to learn or manage the risk, the feature ships to a few because it’s held together with tape and nobody trusts what’s being delivered.
That’s not an experiment. It’s fragility with a guest list.
The problem is presenting it as the fulfillment of a larger scope without saying what’s still pending.
#.Measure all the way to the consequences
DORA, the research program on software teams, measures speed alongside stability: deployment frequency and time from commit to production, but also the share of deployments requiring immediate intervention and recovery time after a failed deployment.
In 2024 they added a fifth metric: deployment rework rate. Of all your deployments, what share were unplanned responses to an incident in production?
If a change comes back three times, those three fixes count too. They may still be a good trade compared with waiting for a much slower release. But they aren’t free.
My starting point for looking at delivery and its consequences.
| What to look at | How to read it |
|---|---|
| Delivery | What we delivered and its scope. |
| Rework | Unplanned deployments caused by incidents, out of all deployments. |
| Support | Tickets related to those releases, per exposed user. |
| Incidents | Those caused by the releases: count, work hours and severity. |
Each figure has its own denominator. A lower rate doesn't, on its own, make up for a much more severe failure.
And give it time: an incident from today’s release may show up next month. If these figures get worse, there’s a cost to explain.
Compare work of equivalent scope. Add up the hours of everyone involved.
If the extra hours match or exceed the savings, we didn't save effort. We moved the work somewhere else.
Shipping sooner may justify it. Then say so: a cost we accepted, not a saving.
#.Look at the incentives too
Before blaming the team that deleted the tests, I’d look at what we were asking of them.
Rewarding only deployments can make cutting what we don’t measure look like a good decision. You can also gain time by automating, making changes smaller or removing pointless waits. The deploy count doesn’t tell those apart.
The question isn’t who cut the corner. It’s who decided that corner didn’t count.
Think about your team’s last speed report. What’s missing from it?
If the saving depends on someone else doing that work later, we didn’t remove work. We left a debt with a due date.