What a Shadow Measures

Before sending a single notification through the new path, I ran it in shadow: on every publish, keep going through SNS as normal, and also compute who our own fan-out query would have sent to, then log the comparison. Cheap, obvious, and the standard move before a cutover.

It took three attempts to get the comparison right, and each mistake was more interesting than the code.

The two sides do not measure the same thing

My first version compared our recipient set against what SNS carried, and treated any difference as something to drive to zero. That is what a shadow is normally for: the old path is the reference, the new path should agree with it, divergence is a bug in the new thing.

Then I looked at where each side comes from.

The user's intent is recorded immediately, in the request. The SNS subscription is recorded after an external, rate-limited call that had been failing for two days.

So the two sides are not two implementations of one question. One holds what people asked for. The other holds what we managed to register with AWS. The gap between them is not noise to be minimised: it is the count of users who asked for a notification and are not receiving it, which is a number this platform had never had.

That reframing killed a plan I had written into the ADR, which was to wait for AWS to lift their protective rate limit before running the comparison, on the grounds that the limit distorts which subscriptions exist. Backwards. The distortion was the most informative thing available, and waiting would have thrown away the window in which it was measurable.

An asymmetric verdict

Once the two directions mean different things, they cannot share a threshold. So the comparison has two counters that are never added together.

A recipient SNS reaches that our fan-out would miss is fatal. That is a bug in the query, and it is the only unacceptable outcome, because it means cutting over would silently stop notifying somebody.

A recipient our fan-out adds that SNS lacks is counted separately, and expected. Those are the users whose subscriptions failed during the outage.

Summing them would have produced a single "divergence" number that went up when things got better. Worse, the sign would have been dominated by the second category, so a real bug in the first would have been invisible inside it.

One implementation detail carries most of the safety here. The clean verdict is derived from the fatal set on every call rather than stored as a flag when the comparison is written. A comparison holding a missed device cannot report success, because success is not a field anybody sets.

An instrument that never runs validates nothing

The first kind of notification I chose to shadow was HIGHLIGHT_VIDEO. The reasoning was decent: it resolves recipients through the match-scoped path, which is the shape the later stages need, and it publishes from a scheduler tick every five minutes, so the shadow would run on a predictable cadence away from any hot path.

It fires at most once per match, only for matches that have a highlight video, and only within 72 hours. In practice the instrument would have sat idle for days.

I had optimised for a comfortable place to put the measurement instead of for getting measurements. Changed to GOAL and CORNER, which happen many times per match, so a defect in the query surfaces within one matchday rather than one week.

The cost of that is real and worth stating. Those two publish from EventProcessor.deliverEvent, which is the gRPC hot path, not a scheduler tick. So the shadow runs fire-and-forget in a goroutine tracked by its own wait group that graceful shutdown drains, mirroring what the Live Activity fan-out in the same function already does. Delivery returns without waiting for it, and there is a test that fails if somebody makes the call synchronous.

I deliberately added no sampling and no throttle. A match has about ten corners and three goals across 90 minutes, so even 100 concurrent matches is roughly 1,300 shadowed events over an hour and a half, about a quarter of one per second, two queries each. A throttle would be guarding a load that does not exist. If real volume contradicts that, the fix is a throttle then, with the measurement in hand.

The HIGHLIGHT_VIDEO call site stayed, inert unless configured, because it is the only coverage of the scheduler path.

What the shadow could not prove

Worth being explicit, because it is the reason for shadowing at all rather than a footnote.

No test exercised that SQL against a real database. This repository has no database-backed repository test and CI provisions no Postgres, so the query's table usage is asserted against its text and its semantics against a fake. A reversed join would pass every test in the suite.

That is precisely the gap production closes. Shadowing is not extra confidence on top of a tested query. It is the test, for the part that cannot be tested anywhere else.

Then it found misses, and had to explain them

Four hours of production data with the shadow on GOAL and CORNER:

Two things in that table. Fan-outs of 218 and 260 recipients, where 17 hours on the staging environment had never exceeded one, and 81% of staging comparisons had both sides empty. Staging proved the plumbing worked. Only production exercised the multi-device path.

And it failed the fatal criterion. Six misses, which were two distinct devices seen repeatedly. One of them was still missing twelve minutes after the first observation, which rules out a race between the publish and the asynchronous shadow query. It was a persistent state difference.

Reading the code eliminated two explanations. It was not an is_enabled semantics mismatch, because the live path applies the identical rule. It was not a heart-team topic leaking in, because the SNS side filters on match id. The leading hypothesis was a stale SNS subscription: a user's intent deleted without the subscription being cleaned up, which would mean our fan-out is correct and SNS has been over-delivering to people who turned the notification off.

I did not encode that hypothesis. The production database is not reachable from my development environment, and guessing was the thing this stage existed to stop.

What went in instead was attribution. Every miss now reports its own reason: no subscribed-match row, kind disabled, no preference row at all, or unexplained. Four reasons, not the five I first sketched, because a device whose row is gone cannot appear on the SNS side either: that query inner-joins the device table.

The verdict got harder, not easier

This is the part I would have been suspicious of if I had read it in someone else's post.

Before attribution, any miss was fatal. Afterwards, the clean verdict means no unexplained miss. On the face of it that is a definition being relaxed to let a failing thing pass, which is the oldest move in the book.

It is strictly harder to satisfy, and the reason is that it requires attribution to have returned a known reason for every miss. A comparison that skipped attribution is not clean. One whose attribution errored is not clean. Before, a miss with no explanation and a miss with a good explanation were the same thing; now the second is allowed and the first is still fatal, and both have to be positively established.

The evidence that the definition moved in the safe direction is that the original criterion's tests still pass unchanged.

After that, 499 attributed comparisons with miss_unexplained = 0. Every miss was a stale subscription or a switched-off preference, both states the fan-out is right to exclude. Which made the cutover an improvement rather than a risk: it stops notifying users who explicitly asked not to be, and reaches roughly 136 per hour who had asked and were not being reached.

What I took from it

Check whether the two sides of a comparison are written at the same moment by the same code path. If they are not, the divergence carries information and averaging it into one number destroys it.

Count the two directions of a difference separately when they mean different things, and derive the verdict from the fatal set rather than storing it, so a bad comparison cannot claim to be good.

Pick the instrument for how often it will produce data, not for where it is convenient to put. A clean signal from something that fires twice a week is not a signal yet.

And a shadow validates the logic, not the load. That distinction cost me later, and it gets its own post.

---

Part of Deleting a Bottleneck, on the SNS Subscribe outage and the migration to direct FCM delivery.