The bill that arrived on day 47
Every team that has shipped an LLM application has a story about the bill. Sometimes it’s the bill the model provider sent that was 10x the projection. Sometimes it’s the bill from a customer who used the product 100x as hard as the median user. Sometimes it’s the bill that was fine on average but had three days where a runaway loop quietly burned $8,000.
Cost metering is the infrastructure that makes those stories not happen. Most teams build it after they get the first bad bill. The teams that build it before are the ones that scale.
Why this is a different problem from cloud-cost FinOps
Cloud FinOps has tooling. AWS Cost Explorer, Datadog, Vantage, CloudHealth. None of them are built for token-based cost. They report cost-by-resource (instances, databases). They are blind to cost-by-decision (which agent, which tenant, which workflow step).
LLM cost is per-call. The cost of a call depends on input tokens, output tokens, the model selected, cache hits, and provider pricing tier. There are 20–50 dimensions per call that matter for attribution. None of the standard FinOps tools handle this natively, and the model providers’ billing dashboards expose only the aggregate.
The result: a team using $40k/month of inference often has no idea which 10% of customers drive 80% of that cost, or which one workflow is responsible for the spike, or whether their per-tenant unit economics are positive. They are running a business they cannot see.
The five things a cost meter needs to do
A cost meter for LLM applications has to do five things. If yours is missing any of them, you don’t have a meter — you have a bookkeeper.
1. Attribute every call
Every model call gets tagged with: tenant ID, user ID, workflow ID, agent ID, tool ID (if invoked from a tool), model, provider, input tokens, output tokens, cache hits, computed cost. Tagging happens at the wire boundary, not after the fact.
2. Aggregate in real time
The meter is not a batch job. Real-time aggregation by tenant, by workflow, by model, by hour. The dashboard is queryable for “how much has tenant X cost in the last 4 hours.” If you can only get that answer end-of-day, you can’t catch a runaway loop.
3. Enforce ceilings server-side
Every tenant has hard per-day and per-hour ceilings. When the ceiling is approached, the meter slows the tenant. When the ceiling is hit, the meter blocks. Server-side enforcement means a misconfigured client cannot escape the limit. Soft limits are not limits.
4. Emit cost alerts that humans actually read
The meter pages someone when:
- A tenant exceeds 5x their 28-day rolling baseline.
- An individual workflow step exceeds expected cost by 3x.
- The aggregate hourly burn rate would project to a daily cost above the budget envelope.
Alerts that page nobody are decoration.
5. Power the per-tenant unit-economics view
The CFO question is “are we making money on each customer.” The meter answers this by feeding the same attributed cost data into the finance dashboard that owns customer revenue. If your cost meter and your revenue system can’t be joined on tenant ID, you don’t know your unit economics.
What the meter looks like in code
The architecture we use is uncomplicated:
- A thin wrapper around every model provider client that emits a structured log per call.
- The log contains all the attribution fields, in JSON, to a streaming sink (we use a managed message bus).
- A downstream aggregator computes the rolling windows by tenant, workflow, model.
- A separate enforcement service consumes the aggregator’s outputs and applies the per-tenant rate limits at the wrapper layer.
- The aggregator feeds two places: an observability dashboard for engineering, and a finance feed for unit economics.
The whole stack is roughly 1,500 lines of code if you build it. There are also off-the-shelf options (LangSmith, Helicone, OpenMeter) that handle 60–80% of the job out of the box. Pick the option that matches your sophistication; the worst answer is no meter at all.
What it costs to not have one
Three failure modes we see repeatedly:
The runaway customer. A single customer’s usage spikes 50x due to a bug in their integration. The bill arrives at the end of the month. The customer disputes it. The internal post-mortem reveals you’d been losing money on this customer for weeks. With a meter and per-tenant ceilings, the spike would have been blocked at 5x and you’d have called the customer the same day.
The expensive workflow. One of your workflows uses 6x the tokens of similar workflows because of an inefficient prompt. You discover this in a quarterly review when the model bill is up 35%. With a meter, the per-workflow cost view would have caught it inside 48 hours.
The pricing-tier surprise. You discover that 8% of your usage is hitting the higher pricing tier because a router downgrade isn’t working. The aggregate cost is 18% higher than it should be. With a meter, you’d see the tier distribution and catch the routing bug immediately.
The cumulative cost of being blind to these is usually 10–25% of the model bill. For a $500k/year inference budget, that’s $50–125k of avoidable spend per year, in addition to the risk-of-incident exposure.
How to set ceilings without making the product unusable
The hardest part of cost metering is setting the per-tenant ceiling. Too tight and legitimate use breaks. Too loose and the ceiling is decoration. The pattern we use:
- Baseline per tenant. 28-day rolling P95 of their daily cost.
- Soft limit. 3x the baseline. When approached, the system warns the tenant (in product) and pages the account owner.
- Hard limit. 8x the baseline. When hit, the system blocks new calls for that tenant until the next billing window or until a human approves an override.
- Burst tolerance. Allowed to exceed the soft limit for up to 60 minutes if cost flattens back out. Prevents legitimate batch jobs from tripping the warning.
These numbers are starting points, not gospel. Tune to your application. The pattern matters more than the magic numbers.
The product-level decisions cost metering enables
Once you have a meter, you can answer questions you couldn’t before:
- Should we price this product per-seat or per-usage? (Whichever matches the cost distribution.)
- Which features are we losing money on? (The ones whose per-tenant cost exceeds the per-tenant revenue.)
- Which customers should we be working harder to keep? (Often not the ones procurement thinks.)
- What’s our gross margin trajectory? (For the first time, you actually know.)
These are questions the AI-product industry as a whole is figuring out in the open. Some companies are going to discover their unit economics work and some are going to discover they don’t. The meter is the difference between discovering it on your own terms and discovering it from your board.
The take
Cost metering is unglamorous infrastructure. It does not appear in product demos. It does not get included in proof-of-concept budgets. It also decides whether your AI application is a business or a science project. Build it before you need it. The teams that get this right have boring meetings about cost and exciting meetings about product. The teams that don’t have the inverse, and the exciting meetings are not the kind you want to have.
Cost metering is part of the production-readiness checklist in our Custom AI Builds, and frequently the first thing we install in Fractional CAIO engagements. Schedule a call if you want to know what your current meter is missing.
