Rate Limiting an AI Feature So One User Cannot Drain the Budget

Rate limiting an ordinary API is about protecting your servers. Rate limiting an AI feature is about protecting your bank account, and that difference changes almost every design decision.

A normal endpoint handling ten times its expected traffic costs you some CPU. An AI endpoint handling ten times its expected traffic costs you ten times the money, immediately, with no upper bound until somebody notices. The most expensive account I have investigated was not abuse — it was an integration retrying a failing call every thirty seconds for eleven days.

This is one of the six components of Template 04, and it goes in before launch alongside cost logging rather than after the first surprising invoice.

Why do AI endpoints need different limits?#

Three properties make them unlike the rest of your API.

Cost per request varies enormously. A short classification and a long-context generation might differ by a factor of a thousand in spend while both counting as one request. Any limit expressed in requests is therefore a poor proxy for the thing you actually care about.

The damage is financial and immediate. An overloaded database recovers when load drops. Money spent does not come back, and there is no natural backpressure — providers are happy to keep serving requests indefinitely.

Legitimate use is bursty. A user uploading forty documents at once is behaving normally, and a limit tuned to block that will annoy exactly the customers you most want. The limit has to distinguish a burst from a runaway, which a naive per-minute cap does not.

What should the limit actually count?#

Cost, with a request-rate limit sitting alongside it for a different purpose.

A monthly spend ceiling per user is the primary control, because spend is what you are protecting. A per-minute request limit is the secondary one, and its job is not budget — it is catching loops, because a runaway is defined by frequency rather than by total.

Running both means the two failure modes are covered separately. A heavy legitimate user hits the monthly ceiling and gets a clear message; a broken integration hits the per-minute limit within a minute and never reaches the monthly ceiling at all.

The distinction matters because the correct response differs completely. A user at their monthly ceiling should be offered an upgrade — they are demonstrating value, and the limit is a pricing conversation waiting to happen. A client making four hundred requests a minute should be cut off immediately and someone should be paged, because nothing legitimate looks like that.

Systems with a single limit inevitably tune it for one case and get the other wrong. Tuned for the loop, it is too tight and blocks real users; tuned for the heavy user, it is too loose and a runaway spends thousands before it trips. Two limits with two responses is barely more code and it stops the tuning from being a compromise.

Where should the check happen?#

In the adapter, before the provider call — which is one of the better reasons to have a single provider adapter in the first place.

Putting the check at each call site guarantees that the fifth feature, written three months later by someone in a hurry, will not have it. And it will be missing quietly, because nothing fails — the feature works perfectly right up until it produces an unbounded bill.

One place, applied to every call, with the task kind available so limits can differ by operation. Reranking might be unlimited because it is cheap and internal; generation is limited because it is neither.

// Inside the adapter, before any provider is touched.
async function complete(input: CompletionInput, ctx: CallContext) {
  const check = await limits.check(ctx.userId, ctx.orgId, input.task);

  if (check.state === 'blocked') {
    throw new RateLimited(check.reason, check.resetAt);
  }
  if (check.state === 'degraded') {
    // Still serve the user — just more cheaply.
    input = { ...input, maxTokens: 500, task: 'generate_cheap' };
  }

  return providerFor(input.task).complete(input);
}

What should happen when a limit is hit?#

Degrade, do not fail. This is the design decision that separates a limit users never notice from one that reads as broken software.

Shorten the context#

Retrieving five passages instead of twenty cuts cost substantially with a modest quality reduction. For a user who has exhausted their allowance, a slightly worse answer is vastly better than no answer.

Route to a cheaper model#

A smaller model at a fraction of the cost is usually adequate for straightforward questions. The user notices little, and the ones who do are generally the heavy users who understand why.

Queue instead of rejecting#

For non-interactive work — batch extraction, background indexing — exceeding a rate limit should slow the queue rather than fail the job. Nobody is waiting on a screen, so latency is nearly free.

Only then, a clear message#

When degradation is not possible, say what happened, what the limit was, and when it resets. "Something went wrong" for a rate limit is a support ticket; "you have used this month's allowance, which resets on the 1st" is not.

How do you implement it without a new service?#

For spend limits, a query against the usage table you already have. For rate limits, a counter — Postgres is fine at small scale, and Redis or a Durable Object once the check itself becomes hot.

The spend check is straightforward: sum this month's cost_cents for the user, compare against their plan ceiling. That is a single indexed query on the table cost logging already writes, and at reasonable volume it is fast enough to run inline.

Where it gets slow, cache the answer for a minute. A user cannot meaningfully exceed a monthly ceiling in sixty seconds, so a slightly stale figure is harmless — and it turns a per-request query into one query per user per minute.

The distributed counter problem#

Per-minute limits across multiple instances need shared state, because each instance counting locally means the effective limit multiplies by instance count. A single Redis counter or a Durable Object solves it; an in-memory counter per process does not, and it fails silently by being too permissive.

What limits should you actually set?#

Set them from your own data rather than from intuition, once you have a month of usage.

Look at the distribution of monthly spend per user. Set the ceiling somewhere above the ninety-fifth percentile — high enough that normal use never approaches it, low enough to bound the worst case. On most products this lands surprisingly high, because the distribution has a long tail and a fat head.

For per-minute rates, look at what a legitimate burst looks like. A user uploading a batch of documents might legitimately make thirty requests in a minute; a loop makes several hundred. There is usually a wide gap between the two, and the limit belongs in the middle of it.

The 15–20% figure is the one worth internalising. If a customer's AI usage consistently exceeds a fifth of what they pay, the pricing is wrong rather than the usage, and a limit is a temporary measure while that conversation happens.

Should free tiers be limited differently?#

Yes, and more tightly than feels comfortable, because free-tier AI spend is unbounded by definition.

A free tier with a generous AI allowance is a marketing budget with no cap, and it will be found. The useful framing is to decide what a free trial is worth as customer acquisition, then set the ceiling to that number and treat overspend as an over-budget campaign.

Degradation matters even more here. A trial user hitting a hard error learns the product is broken; a trial user hitting a clear "upgrade for more" learns the pricing. Same limit, entirely different outcome.

How do you communicate limits in the interface?#

Before they bind, not at the moment they do. A limit encountered without warning feels arbitrary; a limit you have been watching approach feels like a plan.

Show remaining allowance where it is relevant — near the input for usage-based products, in account settings otherwise. Warn at around 80%, in a way that is informative rather than alarming, and make the upgrade path obvious at that point rather than at the wall.

Avoid exposing raw token counts. They mean nothing to most users and they invite questions expensive to answer. Express allowance in units that map to what the customer did — answers, documents, searches — which also insulates you from provider pricing changes.

What should you monitor?#

Limit hits are a product signal, not just an operational one, and they are frequently the most informative thing in your logs.

Hit rate by tier#

A rising proportion of paid users hitting ceilings means either the limits are too low or the pricing is wrong. Either way it is a commercial signal arriving before the churn does.

Which limit fires#

Per-minute limits firing suggests loops or abuse. Monthly ceilings firing suggests pricing. They are different problems with different owners, and aggregating them hides which you have.

Users who hit repeatedly#

An account hitting a ceiling every month is either a pricing mismatch or an unusually valuable customer being throttled. Both warrant a human looking, and neither is visible in an aggregate count.

How do limits interact with the job queue?#

Background work needs different treatment from interactive requests, and conflating them produces the two worst outcomes: interactive users blocked by a batch job, or a batch job failing because it was treated like a page load.

Separate the budgets#

Interactive and background work should draw on distinct allowances. A user who queues a hundred documents for extraction should not find their chat feature unusable for the rest of the month as a result, and the reverse is equally unwelcome.

In practice this means the spend check takes the task kind into account, with separate ceilings per category. It costs one more column and it removes an entire class of confusing support conversation.

Throttle the queue, do not fail it#

When background work approaches a limit, slow the worker rather than rejecting jobs. A queue that drains over four hours instead of one is almost always acceptable; a queue that drops a third of its jobs is not, and the user has no way to tell which third.

Cap concurrency per organisation#

Without it, one customer submitting a large batch monopolises the workers and everyone else waits. A per-organisation concurrency cap — say four simultaneous jobs — keeps the queue fair without needing a priority system, which is considerably more machinery than most products need.

What happens when the provider rate limits you?#

The other direction, and it needs handling too. Providers impose their own limits, and hitting theirs looks different from hitting yours.

A provider rate limit is retryable and usually short-lived. The adapter should recognise it, respect any retry-after header, back off with jitter, and retry — all without the feature knowing anything happened. Users should never see a provider limit surfaced directly, because it is not something they did or can act on.

Where it matters is capacity planning. Provider limits firing regularly means you are near an account ceiling that needs raising, and that is a request with lead time. Log them distinctly from your own limits so the two are separable — they have entirely different owners and remedies.

There is also a queue interaction worth anticipating. If a batch of background jobs all start simultaneously and all hit the provider limit, naive retries produce a thundering herd that keeps the limit saturated. Jitter and a concurrency cap between them prevent it, which is the same pair of controls that solves fairness.

What about abuse specifically?#

Genuine abuse is rarer than accidental overuse, and the defences overlap enough that it is worth treating as a secondary concern rather than the primary design driver.

The controls that matter are the ones already described: per-user ceilings, per-minute rates, and a global backstop. Add authentication requirements on anything expensive — an unauthenticated AI endpoint is an invitation — and rate limit by IP at the edge for whatever remains public.

Where AI features are genuinely different is prompt-based abuse: using your feature as a free general-purpose model by embedding unrelated instructions. Constraining what the feature will answer, and refusing clearly outside its scope, handles most of this and is worth doing regardless because it also improves quality.

Design for the accidental runaway first. A loop in your own retry logic will find your limits before an attacker does.

How do you test limits?#

Deliberately, because these paths never occur during normal development and always occur in production.

Write tests that drive a user past each ceiling and assert the degraded behaviour rather than an exception. Test the boundary specifically — at 99% and at 101% of the limit — because off-by-one errors here either block legitimate users or let the limit be exceeded.

Test the distributed case if you run multiple instances. A counter that works on one process and fails on four is a common and expensive bug, and it only appears under the load that made you scale out in the first place.

And run a periodic check that limits are actually enforced on every AI endpoint. A new feature that bypasses the adapter is the realistic way this protection disappears — which is why the adapter boundary is worth enforcing with a lint rule.

One test worth writing that people usually skip: assert that a degraded response is still a valid response. It is easy to write a degradation path that shortens the context so aggressively the answer becomes useless, or that switches to a model whose output does not satisfy the schema the caller expects. A degradation that produces broken output is worse than a clean refusal, because the user acts on it.

Conclusion#

Limit on cost, not on request count, because request count is a poor proxy for the thing that hurts. Put the check in the adapter so no feature can be written without it. Run a monthly spend ceiling for legitimate heavy use and a per-minute rate limit for loops, since those are different failures.

When a limit binds, degrade — shorter context, cheaper model, slower queue — and only fail with a clear message when degradation is impossible. Warn at 80% in the interface, express allowance in units the customer understands, and set the numbers from your own distribution once you have a month of data.

That is roughly two days of work. It converts the most uncomfortable property of shipping an AI feature — a bill with no ceiling — into a bounded, predictable line item, and it means the answer to "what is the worst case?" is a number rather than a shrug.

If you are adding this to something already live, do it in the order the risk sits. The global daily backstop first, because it takes an hour and bounds everything else while you work. Then the per-minute rate limit, because loops are the most common cause of a shocking bill and they are the fastest to trigger. Then per-user monthly ceilings, which need a month of data to set sensibly. The interface work — the allowance indicator and the 80% warning — can follow once the numbers have settled, since it is the only part users see and the last part worth guessing at.