The Ceiling You Cannot Raise

In December I wrote a post about building this notifications service: events in over gRPC, deduplicated, published to an SNS topic per match, fanned out to devices subscribed to that topic. I still think the reasoning in it was sound. On 29 August it stopped delivering push notifications for about 48 hours, and no amount of tuning could have prevented that, because the thing it ran into was a number AWS will not change.

The service was not down. Events arrived, were deduplicated, and were published to the right topics for the entire outage. Every log line about publishing said what it always said — a healthy service, delivering nothing into topics that had no subscribers at all.

Two quotas, four orders of magnitude apart

We create an SNS topic per (match_id, kind, language) and subscribe device endpoints to it. A match starting means topics created and thousands of subscriptions; a match ending means the reverse.

That puts two SNS APIs on the critical path, and they are not comparable:

Publishing is the thing that looks like the load. It runs at about 1.2 events per second at peak, which is four hundredths of one percent of its quota. Subscribing is the thing nobody thinks of as load, because it feels like setup. It is where all the scaling pressure actually went, and its ceiling is 300 times lower.

The word that matters is non-adjustable. Subscribe TPS is not a soft limit you file a ticket about. I checked, because the first instinct in an outage like this is to ask for more room. There is no more room.

The spiral

On 29 August an upstream bulk import created roughly 32,000 match topics in nine hours. Each of them wanted around 2,400 device subscriptions.

The scheduler's backfill went at it with 50 parallel workers and no rate limiting, which produced about 123 Subscribe calls per second against a ceiling of 100. So essentially every call came back Throttling: Rate exceeded.

Here is the part that turns a bad afternoon into a two-day outage. A throttled call writes no r10_subscription row. The next run queries for subscriptions that ought to exist and do not, finds the same (topic, device) pair still missing, and tries again. The failure feeds the work queue that caused it.

That is a spiral with no damping term in it. It does not recover when the burst passes, because the burst is no longer what is driving it. At the point I started reading logs, live match topics had zero subscribers while events were being published into them successfully, and delivery to real devices had fallen from about 66,000 an hour to about 400 an hour at match peak.

AWS eventually applied a protective rate limit to the account, which took the accepted Subscribe rate down to roughly 7 to 9 per second. Their position, reasonably, was that the pattern was causing resource contention beyond our own service. So the effective ceiling during recovery was not 100. It was single digits, and it was imposed rather than chosen.

Why nobody noticed for two days

This is the part I find hardest to write, and the most useful.

The backfill task finished each run by logging its outcome. Throughout the outage it logged this:

{"msg":"task_completed","status":"success"}

While roughly 50,000 subscriptions failed per 20-minute window.

Per-device failures never incremented TaskResult.Errors and never cleared TaskResult.Success. The counter treated the run as a unit: the run completed, therefore the run succeeded. Nothing in that sentence is false, and it is completely useless.

There was no alarm to fire, because there was nothing to alarm on. A dashboard built from those log lines showed a healthy service for 48 hours. The outage was discovered by a person noticing that notifications had stopped arriving on their own phone.

Partial failure is the normal case for anything that fans out — a counter that reduces two thousand independent sends to one boolean will eventually report success while devices fail silently. I had written that counter myself and it had looked fine in review, because at the time I wrote it the fan-out never partially failed.

Three defects, one symptom

Reading the task path afterwards, the outage needed three separate mistakes, all in the same file, and each of them was individually defensible when it was written.

The first was the missing rate limit. There is a comment in internal/tasks/subscriptions.go claiming that 50 concurrent workers leave headroom under the 100 TPS quota. That conflates concurrency with throughput, and it is wrong by more than an order of magnitude. It deserves its own post, which is the next one.

The second was the budget. Each run had a MaxSubscriptions cap, which is exactly the right instinct. It counted successes. When every call fails, the budget is never consumed, so the run keeps going: up to 500 topics times 1,000 devices of attempts. A limit that only counts work that succeeded stops being a limit under precisely the condition you wrote it for.

The third was the ordering. Pending topics came back ORDER BY match_start_time ASC, which serves the oldest match first. Under a backlog that means the attempt budget goes to matches that finished hours ago before matches kicking off now. The users watching a game right then were last in the queue.

None of these is exotic. Each of them is the kind of thing that passes review, because reviewing them requires holding a quota number and a latency estimate in your head at the same time.

The proportions

Once the immediate fires were out I went looking for the size of what we were operating, and this is the number that decided what happened next:

We were maintaining 25,297 pieces of AWS-side routing state, roughly five per device, for 4,976 devices and about one event per second.

And the subscription data was already in our own Postgres. The r10_subscription table is written by the same code that calls Subscribe, it is queried constantly, and it is the source the backfill reads to decide what is missing. The SNS subscription was a second copy of data we already owned, held in a system where writing it cost a rate-limited API call.

That reframing is what made the fix obvious in hindsight and invisible in advance. The rate limiting, the honest accounting and the urgency ordering all had to ship, and they did, over seven releases in about a day. But they are all ways of living within a ceiling. None of them questions why the ceiling is on the critical path at all.

What I took from it

The load that breaks you is not always the load you measure. We watched publish throughput because publishing is what the service does. The API that fell over was the one we thought of as configuration.

A quota with no ticket path is an architectural constraint, not an operational one. If a limit cannot be raised, then designing to sit just under it means the design has a hard capacity that scales with churn rather than with traffic. Twice the devices is twice the subscribe load, with no headroom to buy.

A failure that recreates its own input will not recover on its own. Worth checking, for anything that retries: does a failed attempt leave the system in a state where the next pass sees the same work? If yes, the retry is not a retry. It is a loop.

And a run-level success flag on a fan-out is a lie waiting for the right conditions. The next post in this series is about how the pacing was wrong. The one after is about deciding to remove the ceiling instead of respecting it.

---

Part of Deleting a Bottleneck, on the SNS Subscribe outage and the migration to direct FCM delivery.