A Decision Record That Argues With Itself
The architecture decision record for this migration is 924 lines long, which is far more than an ADR should be. Most of the excess is one thing: six corrections, dated, sitting inline above the reasoning they overturn.
I did not plan that. It happened because the document was written before the work and consulted during it, so every time production contradicted it I had a choice between editing the wrong part away and writing why it was wrong. I picked the second one enough times that it became the format, and it is now the property of the document I would most want to keep.
The correction I found hardest to write
This one has its own heading in the file: this ADR overstated what we own.
The whole document describes the chosen option as direct FCM and APNs delivery, on the assumption that this service would own both legs. Apple for iOS devices, Google for Android. That framing runs through the decision drivers, the pros and cons, and the migration sequence, and it is what justified building an APNs sender as a prerequisite stage.
It is wrong. Every SNS platform application in the production account is a GCM application. There is no APNS platform application at all. Every device, iOS included, is registered as an endpoint on one Firebase application, and Firebase owns the leg to Apple. The tokens in r10_device.device_token are Firebase registration tokens, not APNs device tokens.
So direct FCM reaches exactly the same devices SNS reached, and no platform router is needed. A router had been planned as a migration stage, with acceptance criteria written. It would have been dead code on the day it landed: a dispatcher choosing between two senders where every input goes to the same one.
There is a second, independent reason it could not have worked. The column it would have branched on, r10_device.operating_system, is never assigned anywhere in non-test code. It exists, it is queried nowhere, and it holds nothing.
Two facts, either of which invalidates a planned stage, both discoverable by reading the production account and the schema. I had written the plan from the design rather than from the system, which is the same mistake as assuming the repository is the running system.
Corrections that changed what got built
Three more, briefly, because each one redirected work.
The comparison harness was supposed to wait for AWS to lift their protective rate limit before measuring, so the two sides would be comparable. That reasoning was backwards and would have discarded the most informative window available, for reasons that are the whole of another post.
The document claimed FCM offered a multicast endpoint accepting 500 tokens per request, and used that to argue the fan-out tail would be modest. That endpoint was deprecated and stopped working in June 2024. Neither Firebase nor Apple has multicast, so a 2,400-device fan-out is 2,400 HTTP requests on either, and the thing that governs the tail is bounded concurrency instead of batching.
And the shadowed notification kind changed, because the first choice fired at most once per match and would have produced no data for days.
A prediction that was wrong in the useful direction
After provisioning was switched off I wrote that the throttling would decay slowly as the existing topic estate aged out.
It stopped at a cliff, in the minute the scheduler picked up the flag: about 44 rejections a minute before, 0.07 after. The throttling had been driven almost entirely by new provisioning rather than by the million subscriptions already standing, which means the deletions I had scheduled as the following stage are housekeeping and not the fix at all.
Keeping the wrong prediction next to the measurement is worth more to me than deleting it, because the wrongness is diagnostic. I had a mental model in which a large standing estate exerts pressure. It does not. Only churn does. That is a general fact about control planes that I now hold with some confidence and would not have noticed if the document had simply been updated to say what turned out to be true.
A comment about another system, again
One small correction in the same family as something I wrote about during the subscriptions extraction.
A comment in internal/models/notification.go explained that the mobile app parses a particular wrapper key in the notification payload. It does not. SNS parses that key, unwraps it, and forwards the inner object with a device token injected. The Flutter client reads the flat fields inside.
Getting that wrong in the direction the comment suggested would have meant preserving a wrapper the app never sees, or worse, deciding the payload had to change. What settled it is a test asserting byte equality between what the direct sender produces and what the SNS path produced, across the silent, visible and image-less message shapes. The payload is provably identical, so the cutover cannot have changed what any handset receives.
A comment describing another system is a snapshot with no owner and no test. This is the second one of those to cost me an afternoon in two months.
A stage built out of order, on purpose
The migration sequence in the document is numbered, and stage 2c was inserted between 2 and 3 after the fact.
Stage 3's premise is that the shadow has validated the recipient query. The shadow only reached production on 2 September, so for a day there was no verdict to read and no way to honestly start the stage that depends on one. Rather than wait, I built the APNs alert sender, which is work the migration needs and which does not depend on the shadow's answer at all.
That sender is still dark. It is not a cutover prerequisite, for the reason in the first correction above. What it is a prerequisite for is one day removing the Firebase hop for iOS, which is a separate decision with its own latency justification and no urgency behind it.
Numbering a sequence implies the order is load-bearing. Sometimes only the dependencies are, and finding the piece of work that has no dependency on the thing you are waiting for is more useful than respecting the numbers.
A bug preserved deliberately, and written down as a question
The last thing worth pulling out is not a correction but an open question the document refuses to close.
Turning off the best-player-survey notification does not stop it arriving. It is carried on the team topic, and the subscription backfill subscribes devices to that topic without consulting the notification preference table at all.
The new team-scoped fan-out query reproduces that faithfully. It reads the same users, including the ones who switched the notification off, and sends to them.
Fixing it during the cutover was the obvious thing and would have been wrong. A migration that also changes who receives a notification makes every subsequent delivery difference impossible to attribute: any change in the numbers could be the transport or the audience, and there is no way to separate them afterwards. So the behaviour is preserved and the defect is recorded as a question for whoever owns notification preferences, which is a product decision rather than a transport one.
That is the same rule as the payload equality test, from the other direction. During a migration, fidelity to the existing behaviour is what makes the migration measurable.
What makes this work
The mechanism is cheap. Correct in place, leave the original visible, date both, say which measurement or which line of code settled it.
The value is that the document records reasoning rather than conclusions. A record edited to look right is indistinguishable from a record that was right, and the difference matters the next time somebody has to decide whether to trust its estimates. Mine now says, in six places, exactly how wrong my estimates were and in which direction, which is the only honest basis for reading the estimates that have not yet been tested.
It also stops the ADR becoming a document that agrees with whatever the code currently does, which is the usual end state and makes it useless as a record of anything.
The cost is that it is long, and reads in places like an argument between two people who happen to be the same person a week apart. I would take that over a tidy document I could not audit.
Part of Deleting a Bottleneck, on the SNS Subscribe outage and the migration to direct FCM delivery.