Delete the Bottleneck, Don't Pace Under It

After the SNS Subscribe outage was stabilised across seven releases, the service was healthy and the design was unchanged. Everything we had shipped was a way of living politely underneath a quota of 100 calls per second that AWS documents as non-adjustable, and which they had temporarily reduced to single digits because our usage pattern was affecting other customers in the region.

So the question stopped being how to pace better. I wrote an ADR, and the first useful thing it did was refuse the question I brought to it.

The question I brought was the wrong one

What I actually wanted to ask was whether we should move to Kafka or RabbitMQ. That is the shape the problem presents: a fan-out system under load, and brokers are what you buy for fan-out.

Writing it down forced me to notice that SNS was doing three separable jobs for us, and a broker replaces exactly one of them:

"Migrate to Kafka" therefore cannot mean "replace SNS". It means build our own fan-out and our own delivery to Apple and Google, and then optionally put a broker in the middle. The broker is the smallest and most optional component of the change I thought I was proposing.

That reframing settled the decision more or less on its own, because once the broker is separated out, the interesting question is whether the fan-out belongs to us. And we already had the data.

The proportions

We were operating 25,297 pieces of AWS-side routing state, about five per device, to serve 4,976 devices and roughly one event per second. The routing data was already in our Postgres: r10_subscription is written by the same code that calls Subscribe, and the backfill reads it to work out what is missing. The SNS subscription was a second copy of a table we already owned, kept in a system where writing a row cost a rate-limited API call.

Four options

The options were: harden the client and keep the topology (what we had just shipped); build our own fan-out with direct FCM and APNs; that plus a broker; or Kafka specifically as a platform event backbone.

The second one was chosen, in stages, with the broker deferred and Kafka rejected.

It is the only option that removes the driver rather than accommodating it. Subscribe stops appearing in the architecture, so the control-plane ceiling stops existing as a concept rather than becoming a number we tune. It also needs no new infrastructure, which matters more than it would elsewhere: the team is two backend engineers already running ECS, Postgres, Valkey and SQS. Acquiring a fifth stateful dependency to fix a quota problem is a trade I would have to justify for years.

The cost model flips, and that is the point

Moving cost from subscribe to publish looks like a lateral move. It is correct here because of where the capacity actually sits. Subscribes are bursty, tied to matches starting, rate-limited, and they are the thing that broke. Publishes run at 1.2 per second, which is four hundredths of a percent of the publish quota. A fan-out query at that rate is nothing.

There is a second-order effect I did not anticipate and now think is the strongest argument in the document. The change does not make the failure less likely, it removes the failure. A user who subscribes one second before a goal receives that goal, because the fan-out query at publish time reads the same table the subscribe just wrote. Under the old design that user had to wait for their subscription to be accepted by an external, rate-limited, individually-failable API before any event could reach them.

The line that actually decided it

The product requirement is that every user can subscribe to a match and start receiving notifications. Here is what that looked like in production on 31 August, after everything we had shipped to stabilise it:

{"msg":"match_subscribed","status":"partial_failure",
 "subscription_count":34,"subscription_failed":2,"duration_ms":9347}

A user waited 9.3 seconds and ended up with 2 of their 36 subscriptions missing. They would get corners and not goals, and nothing told them or us which two were gone.

No amount of rate-limit tuning fixes that, and this is the part worth generalising. While subscribing costs N external calls that can each fail independently, inside a user's request, partial failure remains available at all times. You can make it rarer. You cannot make it impossible. The only way to remove it is to stop the request depending on N external calls, which is what a single INSERT does.

Deferring the broker, rejecting Kafka

The broker is deferred rather than rejected, and the honest reason is that it solves a real problem which is not this one. Our in-process worker pool is a bounded channel with capacity equal to workers times ten. Anything queued in it is lost on crash or deploy. That is a genuine durability gap. At 1.2 events per second it is also not urgent, and adopting a broker without also owning the fan-out changes nothing about the outage we had just had.

If we do adopt one it should be SQS, which is already in the platform for another service, is managed, and costs no operational attention. It is the slowest of the three by latency, adding perhaps 30 to 100 milliseconds, and that is immaterial against a product budget measured in seconds.

Kafka is rejected. It earns its cost through replay, event sourcing and multiple independent consumers at high throughput, and we need none of those. At roughly one message per second, a cluster with partitions, consumer groups and a retention policy is a permanent operational liability bought for capabilities that are not in the requirements. RabbitMQ is honestly a better shape for this workload than Kafka is: per-message acknowledgement, a dead letter queue, and priority queues that would express "live matches first" natively instead of the sort we hand-wrote. It is still a broker to run, and SQS is already there.

What made it affordable

The reason this is a staged migration rather than a rewrite is that most of the hard parts existed already.

internal/apns is a complete direct APNs client: HTTP/2 to Apple, ES256 JWT authentication, topic headers. It has been in production for a while serving Live Activity, which is the highest-frequency push we send. internal/worker already builds the full FCM v1 message JSON, because SNS wants that payload passed through. And the three tables the fan-out query needs are the tables we already query.

The only genuinely missing component was an FCM sender: OAuth2 with a service account, then one HTTP request per token. That is a week of work at most, and it is work with a well-documented contract on the other side of it.

What we take on

Writing the negatives down mattered as much as the decision.

Token lifecycle becomes ours, and it is the main risk of the whole change. Today SNS holds the token in a platform endpoint, marks it disabled when Apple or Google reject it, and reports that in a feedback log we already consume. Afterwards we hold the token and have to interpret every rejection ourselves, deciding which ones mean the device is gone and which mean we are being throttled. Getting that wrong deletes real users' devices, and it is the same failure class as the outage: silent, and invisible on every dashboard.

Retries, backoff and partial-batch handling become ours too, all currently free from the SDK.

And two delivery paths coexist during the migration, so a bug can hide in the difference between them. That one is mitigated by shipping per notification kind and comparing the two, which turned out to be more subtle than it sounds.

What I took from it

The question "which broker" almost always arrives before the question "which of these jobs is actually the problem". Splitting the dependency into the roles it plays for you is cheap and it changed my answer completely.

A quota that cannot be raised is a property of the design, not of the operations. If the only lever is to sit further beneath it, then capacity scales with churn rather than with traffic, and every new device tightens it.

And when a requirement is stated as an absolute (every user can subscribe and start receiving), check whether the current design can satisfy it in principle rather than in the common case. Ours could not, and the 9.3-second partial failure had been sitting in the logs for a while looking like an operational blip rather than the requirement being violated.

---

Part of Deleting a Bottleneck, on the SNS Subscribe outage and the migration to direct FCM delivery.