The Half of a Cutover Nobody Schedules

Twenty-one notification kinds were being delivered directly to Firebase, with SNS entirely out of the path. The delivery numbers were good. And SNS Subscribe was still being throttled at roughly 44 rejections a minute, which is the number the whole migration existed to remove.

The cutover had stopped publishing. It had never touched provisioning. For about 24 hours the service was faithfully building the complete SNS topic estate for notifications it had no intention of ever publishing there.

Two sites, both invisible from the delivery code

The publish path checks a predicate, deliversDirectly, and diverts. Provisioning happened in two places that never consulted it.

EventProcessor.prepareEvent resolved a topic ARN before the divert happened in deliverEvent. So the act of preparing a directly-delivered event created the SNS topic it would then not publish to.

And GenerateMatchTopics ran every five minutes, pre-creating a topic for every notification kind times every language, for every match from two hours before kickoff to three hours after. Then GeneratePendingSubscriptions came along and subscribed devices to all of it, against the 20 calls-per-second ceiling AWS had declined to raise.

Neither is a bug in the cutover. Both are the original design working exactly as written. The cutover moved delivery and left the machinery that exists to make delivery possible running at full tilt, which in hindsight is the obvious failure mode of a staged migration and was not obvious to me at all while staging it.

Suppression is derived, never configured

The tempting implementation is a new environment variable: a list of kinds not to provision. Two days of work becomes twenty minutes.

I did not do that, and this is the design decision in the change I would defend hardest. Suppression is derived from the same deliversDirectly predicate that governs delivery. One parse of one variable feeds four consumers: the event processor, the scheduler, device registration and match subscription.

A separate switch is a thing that can be set wrong. Specifically, it can be set in production, which at that point had no kinds configured for direct delivery and had never sent a direct notification. A production operator enabling "don't provision GOAL" without enabling "deliver GOAL directly" produces a service that provisions nothing and delivers nothing, and the symptom is silence, which is the failure signature this entire project started with.

Deriving it means production keeps provisioning unchanged with no configuration at all, and the two facts cannot disagree.

There is a second guard underneath that one. The parsed set is nil until an FCM deliverer has been successfully built. So a bad credential leaves everything on SNS rather than leaving a kind suppressed with nothing able to deliver it.

The risk was the deduplicator

Deleting topic creation sounds like the dangerous part. It was not. The dangerous part was a cache key.

Deduplication was scoped by topic ARN: dedup:<uniqueKey>:<topicARN>. That is a sensible key while every event resolves to a topic. It stops being available the moment events stop resolving to topics.

The new scope is the canonical topic name, {stage}_{matchID}_{kind}_{language}, which is the identity the ARN carried minus its arn:aws:sns:{region}:{account}: prefix. Same discriminating power, no dependency on a resource existing.

Both ways of getting this wrong are invisible in production, which is what made it the thing to be careful about. Too loose, and two matches of the same kind collapse into one key: a real goal is classified as a duplicate and never sent, and the log line says duplicate rather than error. Unstable, and every event inside the six-hour deduplication window re-delivers, so users get everything twice and nothing looks broken on our side.

The key is now computed from configuration rather than from whether a topic happens to still exist. That distinction matters during exactly the transition this change performs: a leftover cached topic row would otherwise move the key mid-flight, which is a duplicate storm with no code change to blame it on.

The alternative I rejected was to look up an existing topic without creating one. It keeps the coupling the change exists to remove, and it defers the same key change to the next stage. The cost of doing it properly now is one narrow window: at the moment a kind cuts over, an event whose unique key was already delivered inside the six-hour window is not recognised as a duplicate. That needs an upstream duplicate re-send in the minutes straddling the deploy, and only the staging environment had cut-over kinds when it shipped.

The result was a cliff, not a decay

I had written into the ADR that throttling would decay slowly as the standing estate aged out. That was wrong in the best available way.

A reduction of about 500 times, and it happened at 17:19:42 UTC, which is the minute the scheduler task carrying the flag actually started. Not the merge time, which is half an hour earlier and misleading.

Two things that corrects. The throttling was driven almost entirely by new provisioning rather than by the million subscriptions already standing. Stopping creation was sufficient, and the deletions I had scheduled as the next stage are housekeeping rather than the fix.

And my measurement had been wrong by an order of magnitude the whole time. I had been counting log lines containing the literal text Rate exceeded, which peaked at 278 in a 30-minute bucket. Most of the damage was surfacing as context deadline exceeded on the Subscribe call instead, because a throttled call that the SDK retries three times runs out of context before it runs out of attempts. Counting failed to subscribe to SNS gives the real figure. The residual seven are the two kinds that still provision by design.

The estate drains without new tooling

With nothing replacing them, the existing topics started disappearing on their own. PruneAll runs hourly and already deletes match topics older than 24 hours, and subscriptions fell from 1,032,981 to 1,016,095 in a single hour.

The reason that is feasible is a detail in deleteTopic I had not appreciated: it removes the database subscription rows, then the SNS topic, then its own row, and it never calls Unsubscribe. Deleting a topic removes its subscriptions implicitly. So draining a million subscriptions is about 23,000 topic deletions rather than a million rate-limited API calls, which would have taken a fortnight at the rate we were allowed.

What will not drain by itself is the heart-team topics. Pruning them requires both that they have not been updated in 30 days and that they have no remaining subscriptions, and a team with supporters keeps its subscriptions for ever. Since the only thing publishing to them was moved to a team-scoped query that reads the user table directly, those are now permanently dead weight: one per team, so tidiness rather than pressure.

A reason not to hurry the deletion

Every deletion makes reverting more expensive, and I would not have thought of that in this order.

While the estate stands, reverting the whole migration is a flag flip. Once it is drained, reverting means rebuilding 1,016,095 subscriptions through the backfill at a ceiling of 20 per second, which is roughly 14 hours. A flag becomes a working day.

Which suggests leaving the estate standing for as long as it costs nothing to keep, and it costs nothing: it is not generating Subscribe calls any more, only occupying AWS-side state we no longer read.

The uncomfortable part is that production's number is unknown. The diagnostic that reports the subscription count was querying a column that does not exist, so it had never emitted a figure and nobody had noticed the absence. Worth having that number before the same change reaches production.

Two pre-existing wastes it exposed

Neither was introduced by the cutover and both had been running for a long time.

The heart-team topics described above, whose only publisher had already moved.

And three vote-result kinds with no publish site anywhere in the codebase. Every topic and every device subscription ever created for them was pure waste, from the day they were added. They were being provisioned every five minutes, subscribed against a rate-limited API, and pruned hourly, for events that no line of code emits.

Stopping the provisioning is what surfaced them, because the filter had to enumerate which kinds SNS still carries, and enumerating forced someone to ask what publishes each one.

The trap in the obvious filter

The obvious place to filter is models.MatchTopicKinds(), the function that lists which kinds get match topics. Remove the directly-delivered kinds there and every provisioning site inherits the change for free.

That function is also what internal/rest/notification.go uses to build the set of notification preferences the REST API accepts. Dropping a kind from it would start rejecting mobile clients writing their own preferences, for kinds those clients are correctly still allowed to set. A change scoped to AWS provisioning would have broken the app's settings screen.

Every filter is applied at the provisioning call sites instead, which is more code and less elegant, and correct.

The second route to SNS

One more thing this change found, which had been latent for months.

deliversDirectly had exactly one call site, inside deliverEvent. But processWorkerBatch reached SNS by a second route that never consulted it: any group of two or more events sharing a topic ARN went to the SNS batch publish API. So a cut-over kind could silently go back to SNS purely because two of its events arrived close enough together to be grouped.

It had never once fired. Zero batch publishes on staging in 12 hours against 1,174 direct deliveries, and zero in production across seven days and 1,892,754 log records, because the worker drain is non-blocking and the measured 0.04 to 0.07 events per second never queues anything. It fires at the 5,000 events per second the system was designed for.

Prepared events are now partitioned by the divert before being grouped by ARN, so the grouping is structurally incapable of carrying a cut-over kind. I prefer that to adding the check to the second site, because a predicate with two call sites will eventually have three.

Two kinds stay on SNS deliberately

A user campaign publishes to one topic per audience, so one API call reaches roughly 500,000 devices. Doing that as direct fan-out is 500,000 HTTP requests to replace one, for a notification with no latency requirement at all. Bet results are per-user, and entitlement to them is currently implicit in whether the subscription exists, which is a product question rather than a transport one.

Both are pinned in code rather than left to configuration, and configuring either one for direct delivery now emits a startup warning naming it and keeps it on SNS. A cutover that silently declines to happen is worse than one that refuses out loud.

What I took from it

When you move a workload, list everything that exists to support that workload and check each one separately. Delivery moved and provisioning stayed, and the two are only coupled in my head.

Derive a switch from an existing predicate rather than adding a parallel one, wherever the two must agree. The cost is a slightly awkward function signature. The benefit is that a production operator cannot create a state your development environment has never seen.

Before removing a resource, grep for what is keyed on it. A cache key naming a thing about to disappear is the highest-risk line in the change, and it does not look like a risk.

And when a metric refuses to move after a change that should have moved it, suspect the metric. Mine had been counting the wrong string for a week.

---

Part of Deleting a Bottleneck, on the SNS Subscribe outage and the migration to direct FCM delivery.