The Interim We Specified and Threw Away
There is a decision in this project that I am more pleased with than any of the code, and it produced nothing.
The situation
After the Subscribe outage was stabilised, one problem was left standing. Subscribing to a match cost one rate-limited external API call per notification kind per device, and those calls happened inside the user's HTTP request. Production had this in it:
{"msg":"match_subscribed","status":"partial_failure",
"subscription_count":34,"subscription_failed":2,"duration_ms":9347}
A user waited 9.3 seconds and two of their 36 subscriptions never got created. Nothing told them.
The obvious mitigation is to stop the bulk backfill from starving the interactive path. Reserve some fraction of the rate budget for user-initiated subscribes, let the backfill have the rest, and a person tapping a button beats a scheduled job draining a backlog. It is a well-understood pattern, it is maybe two days of work, and it would measurably improve the thing users complain about.
So I specified it. priority-aware-subscribe-budget: stories, acceptance criteria, assumptions, the lot. It went through spec and design and arrived at its approval checkpoint on 1 September.
Then I killed it. The run manifest says why, in the terse way those things do:
aborted at checkpoint: interim mitigation rejected in favour of
going straight at ADR-0001 Option B
Why
Two reasons, and only the second one is interesting.
The first is that it does not fix the problem. Reserving capacity makes a user's subscribe less likely to fail. It cannot make it impossible, because the operation is still N external calls that can each fail on their own, inside a request someone is waiting on. Priority scheduling changes the odds. The failure mode stays in the design. That is triage, and triage is worth buying when you need the time.
The second reason is the one I want to write down. I went and checked what the interim would have protected, against what the real fix removes.
The priority budget would have guarded four backfill functions. Three of them, GeneratePendingMatchSubscriptions, AddMissingHeartTeamSubscriptions and BackfillExtraTimeSubscriptions, are deleted outright by moving fan-out into our own service. They exist only to keep an SNS subscription estate in sync, and there is no estate afterwards.
So the interim would have shipped, worked, and then been deleted along with the functions it was written to protect. Not deprecated slowly. Deleted in the same quarter, by the change that was already decided.
The rule I would keep
An interim is worth building when the real fix is far away, and worth skipping when it protects code the real fix removes.
Both halves matter. I have talked myself into interim work plenty of times on the strength of the first half alone, because "the real fix is months out" is almost always true when you say it and often false in practice. The second half is checkable today. Open the plan for the real fix, list the functions the interim touches, and see how many survive. If most of them do not, you are not buying time. You are buying a diff that has to be reverted.
The distinction is not about how good the interim is. The priority budget was a decent piece of design. It is about whether the thing it protects has a future.
The spec was not wasted
Worth being precise here, because "we threw it away" invites the conclusion that writing it was a mistake.
The spec is what made the abort defensible. Before writing it I had a vague sense that a priority budget would help. Writing the stories and criteria forced me to enumerate exactly which code paths were involved, which is what surfaced the overlap with the deletions. The argument for not building it came out of the work of specifying it.
I would rather spend a morning on a spec that ends in an abort than spend two days on an implementation that ends in a revert. That trade is only available if specifying is cheap and there is a gate between specifying and building.
Which is the other half of why this happened at all. My workflow puts an explicit approval checkpoint between the plan and the execution. If the pipeline ran straight from spec to implementation, this would have shipped, because everything about it was individually reasonable and nothing in the spec itself said "check whether the real fix deletes this". The gate is not there to catch bad specs. It is there to give you one moment to ask whether the thing you have just carefully described is still worth having.
Recording an abandoned option
The last thing I did was write it into the ADR as a section of its own: what was considered, that it reached a checkpoint, and why it was rejected.
Abandoned work usually leaves no trace. The branch gets deleted, the ticket gets closed as won't-do, and six weeks later somebody proposes the same thing, reasonably, because the argument against it lives in one person's memory. Writing the rejection down where the decision lives costs a paragraph and it means the next person to have this idea gets the counter-argument attached.
It also keeps the record honest about how the decision was actually reached. The ADR would read as though the architecture was decided cleanly and executed in order. It was not. Something was specified, taken most of the way, and dropped, and that is worth being able to see.
Part of Deleting a Bottleneck, on the SNS Subscribe outage and the migration to direct FCM delivery.