What you will learn
- Estimate per-request cost from token counts and a pricing map
- Track cumulative spend across a session or a day
- Log interaction telemetry (tokens, latency, model version) without storing PII
- Set budget guardrails that cut off runaway spend
Why this matters
LLM cost is usage-based and unbounded by default — a loop bug or a viral spike can turn a $50/day app into a $5,000 surprise overnight. Per-request cost estimation plus a running budget is the control that prevents that. And cost-per-interaction telemetry is exactly what product and finance ask you for. This is the unglamorous discipline that keeps AI features profitable.
Estimate cost where the tokens are
Cost is a function of tokens and a per-model rate. Count tokens with tiktoken, multiply by the model's rate, and you have a per-request estimate before or right after the call:
import tiktoken
class TokenCostEstimator:
def __init__(self, model_name: str = "gpt-3.5-turbo"):
try:
self.enc = tiktoken.encoding_for_model(model_name)
except KeyError:
self.enc = tiktoken.get_encoding("cl100k_base")
def count_tokens(self, text: str) -> int:
return len(self.enc.encode(text))
def estimate_cost(self, tokens: int, cost_per_1k: float = 0.002) -> float:
return (tokens / 1000) * cost_per_1kTrack input and output separately when their rates differ.
Telemetry: log the interaction, not the content
For every call, emit a structured event: an interaction ID, session ID, model version, token count, and latency — but not the raw query/response (that's PII). This feeds cost dashboards and per-feature cost attribution.
Budget guardrails
Estimation is observability; a budget cap is control. Keep a running total per session (or per tenant, per day) and refuse or downgrade once it crosses a ceiling. This is the backstop that turns a runaway loop into a graceful "limit reached" instead of an invoice shock.
Aggregate and attribute
Roll per-request costs up by feature, tenant, or model. That's how you answer "which feature is eating the budget?" and where caching or a cheaper model pays off.
import uuid
class CostBudget:
"""Per-session running spend with a hard ceiling."""
def __init__(self, limit_usd: float):
self.limit, self.spent = limit_usd, 0.0
def charge(self, cost: float) -> bool:
if self.spent + cost > self.limit:
return False # refuse before overspending
self.spent += cost
return True
def tracked_call(query: str, est: TokenCostEstimator, budget: CostBudget, llm, logger, session_id: str):
in_tokens = est.count_tokens(query)
if not budget.charge(est.estimate_cost(in_tokens)):
return {"error": "budget limit reached"} # graceful cutoff
t0 = time.time()
answer = llm(query)
out_tokens = est.count_tokens(answer)
budget.charge(est.estimate_cost(out_tokens))
logger.info("ai_interaction", # telemetry, no raw content
interaction_id=str(uuid.uuid4()), session_id=session_id,
model_version="gpt-3.5-turbo", tokens=in_tokens + out_tokens,
latency_ms=round((time.time() - t0) * 1000, 2),
est_cost=est.estimate_cost(in_tokens + out_tokens))
return {"answer": answer}Try it yourself
- Implement
TokenCostEstimatorand estimate cost for 5 prompts - Track input vs output tokens separately with different rates
- Add
CostBudgetand confirm it refuses the call past the ceiling - Emit interaction telemetry and aggregate daily spend from the logs
- Attribute cost by "feature" tag and find the most expensive one
Common mistakes
- No budget cap — a loop or spike runs up an unbounded bill
- Logging raw query/response — that's PII; log tokens and IDs instead
- One rate for input and output — output is usually pricier; track separately
- Estimating but never aggregating — you need cost-per-feature to act
- Charging only after the call — check the budget before expensive generation
Checklist
- Estimated per-request cost from token counts
- Tracked input/output tokens with separate rates
- Enforced a running budget cap with graceful cutoff
- Emitted PII-free interaction telemetry
- Aggregated and attributed cost by feature/tenant
Quick quiz
Q: What's the difference between cost telemetry and a cost budget?
How to explain it in an interview
"LLM cost is usage-based and unbounded by default, so I both observe and control it. Observability is per-request estimation — count tokens with tiktoken, multiply by the model's rate, tracking input and output separately since output is pricier — plus interaction telemetry: an event with interaction ID, session, model version, tokens, and latency, but never the raw query or response, which is PII. Control is a running budget cap per session or tenant that refuses or downgrades once spend crosses a ceiling, checked before the expensive generation call. Then I aggregate cost by feature or tenant so I can answer which feature is eating the budget and where caching or a cheaper model pays off."