Idempotency at a million events a second

At 1.2 million events per second, a 0.01% duplicate rate is 120 extra charges every second. Retries are a normal part of distributed systems, so the ledger has to treat duplicates as routine, not exceptional.

The contract with senders

Every event carries an idempotencyKey. We promise that two events with the same key and customer, received within 35 days, count once. Senders can retry as often as they like.

Where the check happens

A naive design looks up every key in a database before writing. That works until about ten thousand events per second, then the lookups become the bottleneck.

Instead, each ingest node keeps a sharded, time bucketed set of recent keys in memory:

function seen(key: string, at: number) {
  const bucket = buckets.get(Math.floor(at / HOUR));
  if (bucket?.has(key)) return true;
  bucket?.add(key);
  return false;
}

Events are routed by customer, so the same customer always reaches the same shard. Hot buckets stay in memory; older buckets are compacted to a probabilistic filter backed by an exact check on disk.

Numbers from production

A duplicate that reaches the ledger costs a support ticket. A duplicate dropped at ingest costs a log line.

Late events

Events can arrive after the period closes, for example from a device that was offline. They are accepted, deduplicated against the same window and land as an adjustment line on the next invoice with a link to the original period. Customers see exactly why the number moved.

The result is boring in the best way: senders retry freely, invoices match usage to the cent, and the on call engineer sleeps.