A failure mode rate limits can’t catch
Rate limits prevent the most obvious AI failure: a misbehaving client hammering the API. They are necessary. They are also insufficient for autonomous agents.
Consider an agent that’s well-behaved by rate-limit standards. It makes one call every two seconds. It never spikes. Looks healthy on every dashboard. But because the agent is in a degenerate planning loop, each call is consuming 30,000 input tokens (the entire context, replayed each step) and producing 4,000 output tokens. At one call every two seconds, that’s about $50/hour of cost on a frontier model. Over a weekend before anyone notices, that’s $2,400.
The rate limit didn’t fire because the call rate was fine. The cost was the problem. The cost was visible in the inference bill at the end of the month and nowhere else in time to do anything about it.
Budgets are the control that catches this. They are a different kind of control from rate limits and most teams confuse the two.
The clean separation
Two controls. Different purposes. Different parameters.
Rate limits are about system stability. They protect your infrastructure (and your providers’) from being overloaded. They are denominated in requests per second, requests per minute, concurrent connections. They fire on calls-per-time.
Budgets are about financial sustainability. They protect your business from spending more than you can recoup. They are denominated in dollars per day, dollars per task, dollars per tenant. They fire on cost-per-time.
A well-architected agent enforces both. Most agents enforce only the first.
The four budget surfaces that matter
Budgets are most useful at four scopes. Each catches a different class of failure.
1. Per-tenant daily budget
A hard ceiling on what one customer can cost per day. Set based on per-tenant unit economics — the daily cost above which this customer is unprofitable.
This is the headline control. When a tenant’s usage spikes 50x (because of a bug in their integration, or because their CSV ingestion went wild, or because their AI surface got featured in a TechCrunch article), the per-tenant budget catches it. The customer hits the ceiling, your agent throttles them, you call them in the next 30 minutes. The damage is bounded.
Without this, you find out about runaway customers when you see the bill.
2. Per-task budget
A hard ceiling on what a single agent task can cost. Set based on the expected cost distribution — typically 5-10x the median task cost.
This catches the degenerate loop pattern. An agent that should cost $0.40 per task starts costing $12 per task because it’s planning poorly. The per-task budget trips, the agent aborts, the alert fires. The task didn’t complete, which is the right answer — completion at 30x expected cost is itself a failure.
The interaction matters: per-task budgets should be implemented in the agent’s own logic, not just in the gateway. The agent should know its budget and respect it.
3. Per-agent daily budget
A hard ceiling on what one agent (across all tenants and tasks) can cost per day. Set based on the agent’s expected workload.
This catches systemic agent failures. A code update that introduces a 3x cost regression across all traffic shows up as a per-agent budget breach. The control kills the agent before the bill arrives.
4. Program-wide daily budget
A hard ceiling on the company’s total AI spend per day. Set generously, but real.
This is the catastrophic-failure backstop. If somehow every other control fails — multiple tenants hit at once, an agent goes haywire, a credential leaks — the program budget caps the damage. It’s the control that prevents an AI program from accidentally bankrupting a small company.
The implementation pattern
Budgets are enforced server-side, at the inference gateway, with the following architecture:
- A metering layer that tags every call with tenant ID, agent ID, task ID, and computes cost in real time.
- A real-time aggregator that maintains rolling windows by each scope.
- A policy engine that decides whether to allow, slow, or block based on the current state vs the configured budgets.
- A budget service that holds the configured budgets and serves them to the policy engine.
The gateway should be the single point of enforcement. If budgets are enforced in client code, they will be bypassed. If they’re enforced in product code, they will be inconsistent.
For teams without a centralized gateway, the minimum viable version is a small in-process middleware that wraps every model client call, consults a shared counter (Redis is fine), and refuses calls when the relevant budget is breached. Two hundred lines of code. Good enough for most mid-scale systems.
The graceful-failure behavior
What happens when a budget trips matters as much as the budget itself.
Per-tenant budget tripped → return a structured error indicating budget exhaustion. The product layer surfaces this to the customer with a clear message (“you’ve reached your daily limit, contact us to expand”). Do not silently degrade — the customer needs to know.
Per-task budget tripped → the agent aborts the task, logs the abort, returns a structured error to the caller. Do not retry. Do not “try a cheaper model.” Aborting is the safe failure.
Per-agent budget tripped → the agent disables itself, pages the on-call. New tasks are rejected. In-flight tasks complete if cheap, abort if expensive. Manual investigation before re-enable.
Program budget tripped → page everyone. Investigate immediately. Do not auto-restart. Whatever caused this is bigger than any single control catching it.
The pattern: smaller budgets fail safe within the system, larger budget breaches require humans.
The metrics that prove budgets are working
A working budget system produces specific metrics. Three to watch:
Budget utilization distribution. What percentage of tenants are using <10%, 10–50%, 50–90%, >90% of their daily budget? You want most in the middle. A heavy tail at the high end means your budgets are too tight; everyone at the low end means you set them too loose.
Budget-breach incidents. How many breaches per week, by scope. Trending up is interesting (something’s drifting). Trending to zero is interesting (your budgets may be too loose). Stable at a low rate is healthy.
Time from breach to operator action. How long after a per-tenant breach does someone reach out to the customer? Under an hour is good. Under a day is acceptable. Multi-day is broken — your alerting isn’t working.
How to set initial budgets
The first time you set budgets, the parameters are estimates. The process:
- Run for two weeks without budgets but with metering on. Collect the cost distribution.
- Compute baselines: per-tenant median and P95 cost-per-day, per-task median and P95 cost, per-agent daily cost.
- Set initial budgets at 3x P95 for soft warnings, 8x P95 for hard limits. These are starting points.
- Adjust over the next two months as you see real breach patterns.
The wrong way is to set the budgets based on what the product team hopes the cost will be. Real cost distributions are always more skewed than the team expects.
The conversation with finance
A side benefit of having budgets: the conversation with finance changes shape. Without budgets, finance asks “what’s our AI spend going to be” and the engineering answer is “we don’t know.” With budgets, the answer is “we cap at $X/day; here’s the current run rate against the cap.”
This is the difference between AI spend being an open-ended liability and a bounded operational cost. Finance teams handle bounded operational costs comfortably. They struggle with open-ended liabilities. Reshape the conversation by giving them the bounds.
What budgets don’t solve
Budgets are a financial control, not a quality control. An agent that’s cheap and bad still costs you in different ways. Budgets don’t replace eval suites, output review, or human-in-the-loop on high-impact actions.
Budgets also don’t replace good architecture. An agent that’s well-designed should rarely come close to its budgets. Budgets that fire constantly mean the agent is poorly designed and the budget is a band-aid.
The take
Rate limits and budgets are different controls and most teams have only the first. Add the second. Four scopes (per-tenant, per-task, per-agent, program-wide), enforced server-side, with clear graceful-failure behavior. The engineering cost is bounded. The protection is real. AI programs that scale safely have budgets that work. AI programs that don’t, eventually have a story about the month they spent more than they meant to.
Budget architecture is part of the production-readiness checklist for our Custom AI Builds. If your current AI spend feels open-ended, schedule a call.
