Deleting a Bottleneck: A Reading Order

Between 29 August and 4 September I took a push notification service through an outage, a rejected mitigation, an architecture decision, a shadow deployment and a production cutover. These eight posts came out of it. They were written as the work happened, so each one starts wherever things had got to.

This is the beginning.

The service ingests sports events over gRPC, deduplicates them, and sends push notifications to about 5,000 devices. It did that by creating an AWS SNS topic per match, per notification kind, per language, and subscribing device endpoints to those topics. I wrote about building it in December, and about rewriting its Python predecessor before that.

The design put two SNS APIs on the critical path. Publish has an account quota of 30,000 per second and carried about 1.2 events per second. Subscribe has a quota of 100 per second, documented as non-adjustable, and carried every bit of growth and churn in the system. On 29 August an upstream bulk import created roughly 32,000 topics in nine hours, each wanting about 2,400 device subscriptions, and that 300-fold asymmetry became a 48-hour outage.

The decision that followed was to stop pacing underneath the ceiling and remove it from the design: query our own Postgres at publish time and send to Firebase directly, so Subscribe stops appearing in the architecture. That shipped in stages between 1 and 4 September, and the throttling it existed to eliminate dropped by a factor of about 500.

The outage

The Ceiling You Cannot Raise: how a throttled call that writes no row turns into a spiral that cannot recover on its own, and why nobody noticed for two days while the task logged status: success through 50,000 failed subscriptions per 20 minutes. Start here.
Concurrency Is Not Throughput: a comment claiming 50 workers left headroom under a 100 per second quota, when 50 concurrent calls at 30ms each is about 1,600. Also why the rate limiter already in the codebase could not have capped anything, and where a limiter has to be charged so that SDK-internal retries are counted.

The decision

Delete the Bottleneck, Don't Pace Under It: four options, and why the question I brought (Kafka or RabbitMQ) was the wrong one: SNS was doing three separable jobs and a broker replaces exactly one. Includes why Kafka is the wrong answer at one message per second.
The Interim We Specified and Threw Away: a priority-aware rate budget, specified, taken to its approval checkpoint and deliberately abandoned, because three of the four functions it protected are deleted by the real fix.

The migration

What a Shadow Measures: running the new fan-out beside the old one is not a symmetric test: the two sides read different tables written at different times, so the divergence is the measurement. On an asymmetric verdict, an instrument that never runs, and making every miss explain itself.
The Tail Is One Stalled Request: production disagreed with the model in both directions. The largest fan-out was 1,023 recipients rather than 308, and the two slowest fan-outs were not the largest ones. Why the timeout was left alone.
The Half of a Cutover Nobody Schedules: delivery moved and provisioning stayed, so the service spent a day building an SNS topic estate for notifications it would never publish there. Also a deduplication key that referenced a resource about to stop existing.

The record

A Decision Record That Argues With Itself: six dated corrections sitting inline above the reasoning they overturn, including a planned migration stage that would have been dead code, and a prediction that was wrong in a useful direction.

Background

Two earlier posts cover the system this arc replaces, and one covers the concepts underneath it:

There is an adjacent series about a different service at the same company, extracting subscriptions and payments out of a Django monolith: The Subscriptions Extraction. The two projects share a shape, which is that both are migrations where the old and new implementations run at once and the interesting engineering is in making the comparison trustworthy.