The ADR modelled the fan-out tail and production disagreed in both directions. The largest fan-out was 1,023 recipients rather than 308, and the two slowest were not the largest: a handful of stalled requests dragged a 3-second fan-out to 18. Why the timeout was left alone, and what a clean shadow verdict does not tell you.
Six dated corrections sit inline in this ADR, above the reasoning they overturn, including one headed 'this ADR overstated what we own'. On why a record edited to look right is worth less than one that shows its working, and the planned stage that would have been dead code.
Eight posts on a 48-hour push notification outage caused by a quota AWS will not raise, and the migration that removed it from the architecture instead of pacing beneath it. What each one covers, and the order that makes them make sense.
The cutover stopped publishing to SNS and never touched provisioning, so the service spent a day creating an entire topic estate for notifications SNS no longer carried. Stopping creation dropped throttling ~500x at a cliff. On deriving suppression from the predicate that already governs delivery, and a dedup key that referenced a thing about to stop existing.
Running a new fan-out beside the old one and comparing looks like a symmetric test. The two sides read different tables written at different times, so the divergence is the measurement rather than noise. On an asymmetric verdict, why an instrument that never runs validates nothing, and making every miss explain itself.
Four options for a push architecture that had run into a quota AWS will not raise. Why a broker cannot replace SNS, why Kafka is the wrong answer at 1.2 messages per second, and why moving cost from subscribe to publish was correct here: a user who subscribes one second before a goal now receives that goal.
A priority-aware rate budget was specified, taken to its approval checkpoint, and then deliberately abandoned. Three of the four functions it would have protected were deleted by the real fix, so it would have been dead code the day it landed. On when an interim earns its cost and when it does not.
Push notifications stopped reaching users for 48 hours because every bit of growth in the design ran through the one AWS API with a hard, non-adjustable quota. On a throttle spiral that cannot recover on its own, a run that logged success while 50,000 subscriptions failed, and 25,297 pieces of cloud routing state serving 4,976 devices.
A comment claimed 50 parallel workers left headroom under a 100 calls-per-second quota. Fifty concurrent calls at 30ms each is about 1,600 per second. On the difference between a count and a rate, why a limiter that sleeps per call cannot cap anything, and where a rate limiter has to be charged so the retries inside your SDK are counted too.
Seventeen posts on pulling the subscriptions and payments half of a Django monolith into a Go service — measured before it was designed, phased so every step could be undone, and proved against 243,325 real purchases before it answered a single user. What each one adds, and the order that makes them make sense.
We gate every phase of this extraction on a mechanical check: each acceptance criterion needs a passing test, and the verdict is an exit code. It has caught real defects. It has also produced three lessons entirely about itself — because a test whose subject is the shape of your codebase is invalidated by refactoring by design, and every false failure is an invitation to weaken it.
Porting a service means every scoping decision rests on a fact about production. A log query matched 0 of 292,932,577 records and read as a definitive answer — the format it searched for was 100% of live traffic. A 500-row sample reported 96% and the full corpus reported 80%. On the difference between 'none exist' and 'my query can't see them', and the positive control that separates them.
Three times in one project, code that was correct, reviewed, merged and deployed did nothing at all — because the value it needed never crossed one of the nine boundaries between where it's declared and where it's read. On a symptom four independent bugs produced, a permission error that arrived disguised as absence, and four endpoints that passed every test and answered 404.
Granting a paid tier sounds like setting a boolean. It's a predicate over subscription states nobody agrees on, applied to four systems with no transaction between them. On ordering writes by what each partial failure leaves behind, refusing to compensate, a product decision that granted access to precisely nobody, and one enum asked two questions that look like one.
When you extract a service and both copies write the same database, the success condition stops being 'it works' and becomes 'it agrees'. On a comment that would have made us refuse paying subscribers, two ugly quirks I ported on purpose, and the one place we diverged deliberately — plus the auth change that looked like an optimisation and was actually a security decision.
Three sources — the board, my workflow's own run records, and git — merged into one line per working day. Getting the data was easy; the naive version produced 58 activities for a single Tuesday. On why each filter had to earn its place, and the four ways duplicate work arrives wearing different clothes.
Specs gated on a mechanical audit, progress mirrored onto a team board, a timesheet filled from what actually happened. An honest account of a day spent building and debugging that setup — including the three work items I created on a live board by accident, and the one failure that showed up in every single component.
A complete Boards client — fetch, create, transition, link, reparent, WIQL — in one bash script with no SDK. Most of the post is the four failures that shaped it: a permissions problem that arrives as HTTP 500, parents that mark themselves as your work, a flag that silently orphans cards, and two tokens of identical length where one was revoked.
A guard compared whole JSON payloads to avoid needless re-renders. The payload included a generated timestamp, so it never matched once — the UI rebuilt every 2.5 seconds and threw you back to the top of any document you were reading. On volatile fields, and why gating an event on 'did it change' can lose that event permanently.
The rebuild succeeded, the restart silently didn't, and the health endpoint kept answering — from the old binary. I reported a change as live when it wasn't. On verifying that the process you started is the one serving, and why the lsof invocation everyone copy-pastes is wrong in two independent ways.
Every one of these produced no error, no warning, and a plausible-looking wrong result. Collected from a single day of writing bash, jq and git glue: heredocs that steal stdin, tab as IFS whitespace, NUL bytes vanishing in command substitution, jq's dot rebinding after a pipe, and four more.
A tool that keys per-project state by absolute working directory. Copy the files to a second machine with a different home directory and you don't merge history — you accumulate parallel copies of it. One repository had three. On identifiers derived from environment facts, and why making the environments agree beats translating forever.
Three times in one day a step that was clearly documented in my workflow got skipped anyway. The fix was never better wording — it was folding the step into the command that already ran at that moment. On why written procedure decays, why 'same as before' is a documentation smell, and what it means to make a step unskippable.
If the definition of done is a sentence, it will be negotiated. On making acceptance criteria executable, letting a process exit code be the verdict, refusing to record a lesson without evidence behind it — and the gap I found between ten criteria that had tests and zero that had passed.
The output of an evaluation shouldn't be a verdict someone has to trust — it should be a document they can disagree with. On ranking gaps by impact instead of tabulating features, establishing the minimum tier per capability against the catalogue you actually serve, and making the cost of "yes" explicit.