Concurrency Is Not Throughput
The subscription backfill in our notifications service ran 50 goroutines against an AWS API with a quota of 100 calls per second. Next to the constant that set the worker count was a comment explaining that this left headroom under the quota.
It had been there for months. It read like something someone had thought about. It was wrong by more than an order of magnitude, and it is the reason push notifications stopped for 48 hours.
A count and a rate are different units
Concurrency is how many operations are in flight at once. Throughput is how many complete per unit of time. They are related, and the relationship is not "roughly the same number".
For a call that takes 30 milliseconds, one worker issuing them back to back does about 33 per second. Fifty workers doing that do about 1,650 per second. The quota was 100.
You can get this from Little's Law without deriving anything: throughput equals concurrency divided by latency. The moment latency gets faster, throughput goes up while the worker count stays exactly where it was. So a concurrency limit is not a rate limit, and it is not even a stable proxy for one. Ours drifted with the response time of the API it was calling.
What makes the mistake easy is that both numbers are small integers that sound like capacity. Fifty and one hundred look like they belong on the same axis. They do not, and no amount of staring at the constant reveals that, because the missing term is the latency, which is not in the file.
The limiter we already had could not have helped
The uncomfortable part is that this service already had a rate limiter, internal/ratelimit, and it was already wired into the subscription service. It just was not wired into the path generating 99% of the load.
Except it would not have mattered much if it had been, which took me longer to see.
AdaptiveRateLimiter.Wait sleeps a per-call delay and returns. There is no shared accounting: each caller waits its own delay, independently. So with a 50 millisecond delay and one caller you get 20 calls per second. With 50 callers you get 1,000 calls per second, because they all sleep concurrently.
A per-call sleep is a pacing mechanism for a serial caller. Under parallelism its throughput is concurrency / delay, which is the same formula that broke the worker pool. The limiter had the same bug as the code it was supposed to protect.
This is a general shape worth recognising. If your limiter's state lives in the calling goroutine, it cannot enforce a global anything. A ceiling has to be a thing all the callers contend for.
What a ceiling actually looks like
The fix was a token bucket, in the same package, with one instance shared across every worker. Tokens refill at a set rate, a caller takes one or waits, and the accounting is in the bucket rather than in the caller.
I wrote it rather than adding golang.org/x/time/rate, which is a defensible dependency and would have been the obvious choice on a normal day. This was not a normal day: production had been broken for two days and I did not want the fix carrying a go.mod change into a release nobody had time to review carefully. A token bucket is forty lines. git diff go.mod go.sum came back empty, which was the point.
Proving it took a moment's thought too. The natural test is to assert the limiter was called, which proves nothing about the rate. The criterion we wrote instead runs 50 goroutines all trying to acquire at once, for a measured window, and asserts the number that got through never exceeds the ceiling for that window. That is a timing lower bound rather than a mock assertion, which makes it slower and slightly less pleasant, and it is the only version that would have caught the original bug.
Recognising throttling is its own problem
A ceiling stops you exceeding the quota you think you have. It does nothing about the quota you actually have, which in our case AWS reduced to single digits partway through the incident.
For that you have to hear the rejection and slow down. Which means classifying errors, and the errors are messier than the documentation suggests. AWS returned all of these for the same condition:
Throttling: Rate exceeded, the documented oneThrottlingException, from a different code pathexceeded maximum number of attempts, 3, ... api error Throttling, where the SDK has already retried three times and wrapped the original
And these, which are not throttling and must not be treated as it:
Endpoint is disabled, meaning the device is goneTopic does not exist, meaning our own state is wrong
The classifier reuses aws-sdk-go-v2/aws/retry, already a direct dependency, so the typed check is real rather than string matching. Testing it needed a fake typed API error, and rather than pull in smithy-go for one test I declared a local type with an ErrorCode() string method, which is all the interface asks for.
Two rules came out of this that I would now apply anywhere:
Throttling is never a reason to delete a token. It says nothing about the device, only about your rate. We had a device-pruning path that could have been reached from a throttled response, which would have deleted real users' devices to punish our own pacing.
Throttling is never a reason to trip a circuit breaker either. A breaker exists to stop hammering a dependency that is broken. A throttle means the dependency is fine and you are too fast: the correct response is to slow down and keep going, not to open a circuit and stop. That one needed a change to circuitbreaker.Config.IsFailure, because our breaker had been counting throttles as failures and opening on them, which converted a pacing problem into an outage of the whole path.
One ceiling per process is not one ceiling
The last piece is the part I would have got wrong if the incident had not made it obvious.
The service runs more than one task. A token bucket in a process caps that process. Two processes with the same ceiling produce twice the rate, and the quota is per account.
So the ceiling moved into Redis: a token bucket keyed per operation, returning a wait hint so a caller that cannot proceed knows how long to sleep instead of spinning. Three details in that turned out to matter more than the bucket itself.
It fails open to the local ceiling. If Redis is unavailable, the limiter falls back to the in-process bucket rather than refusing to work. A rate limiter that becomes an outage when its store blinks is a worse problem than the one it solves, and the fallback state is reported on the log line so "we are running degraded" is visible rather than inferred.
Budget is charged per HTTP attempt, through SDK middleware, not per logical call. This is the one I like. If you charge a token when your code calls Subscribe, and the SDK internally retries three times, you have spent three units of quota and counted one. Hooking the charge into the middleware chain means the thing being counted is the thing the quota counts.
And the numbers in force are reported rather than assumed: the rate currently applied, the consumption against it, and whether the local fallback is active. During recovery, with AWS's protective limit in place, that log line was the only way to tell the difference between "we are pacing correctly" and "we are pacing correctly against the wrong number".
The budget that only counted wins
One more thing from the same file, because it is the same category of error.
Each backfill run had a cap on how many subscriptions it would create. It counted successes. When every call fails, nothing increments, so the cap is never reached and the run works through its entire candidate set: up to 500 topics times 1,000 devices of attempts, all failing, all feeding the retry spiral.
A budget that counts completions is a budget that disappears under failure. Count attempts. The resource you are protecting is spent whether or not the call worked.
What I took from it
Throughput is concurrency divided by latency, and only two of those three appear in your code. If a comment reasons about a rate from a worker count, the latency it assumed is doing the work, and it is not written down anywhere.
A limiter whose state is per-caller cannot enforce a global rate, no matter what it is named. Check where the accounting lives before trusting it.
Charge the limiter where the requests actually leave, which is usually lower than where you think. SDK-internal retries spend real quota.
And a throttle is a pacing signal, not a fault. Everything downstream of your error handling needs to know the difference: breakers, prune jobs, alarms, and the run's own success flag.
Part of Deleting a Bottleneck, on the SNS Subscribe outage and the migration to direct FCM delivery.