Budgets & Metering

Markdown Open in ChatGPT Open in Claude

Model spend in Sema is ambient, not threaded. You do not return a (result, usage) tuple from every function and thread it up the call stack; you wrap a scope in with meter as u: and every model call inside it accumulates into u. When you need a hard limit, with budget(...) is the same thing with a cap that raises rather than overspends. And when you issue many calls at once, the scheduler batches and distributes them without you wiring threads or queues.

Why usage is ambient

Conventional LLM code threads usage everywhere: each call returns its token counts, each wrapper adds them up, and the accounting logic tangles with the business logic — SymbolicAI’s tracker was ~470 lines for exactly this. Sema removes the plumbing. A model call returns the value only; the surrounding meter observes usage as a side effect of the effect-handler stack (effects are handlers, and metering is one of them). Meters nest, and each call attributes to all enclosing frames.

Metering: `with meter as u:`

with meter as u: accumulates every model call’s usage within the block into u:

with meter as u:
    answer = write_report(facts)          # returns the value only
log.info("run", tokens=u.total_tokens, cost=u.cost, calls=u.total_calls)

The meter exposes:

u.total_calls — number of model calls
u.prompt_tokens, u.completion_tokens, u.total_tokens
u.cost — priced from [pricing] per_token in sema.toml

The verified research-agent example reads exactly these fields at the end of a synthesis step, with no tuple threading anywhere in the call chain:

mut calls = 0
mut cost = 0.0
with meter as u:
    _synthesis = generate("Summarize renewable energy findings", 64)
    calls = u.total_calls
    cost = u.cost

Budgets: `with budget(...)` — a hard cap

A budget is a meter with a ceiling. A model call that would push spend past the cap raises BudgetExceeded rather than silently overspending:

with budget(calls=200, tokens=1_000_000) as b:
    research = deep_search(query)          # BudgetExceeded if it overspends

Because BudgetExceeded is an ordinary typed failure, you catch and degrade rather than crash:

expect summary = deep_search(query):
    return summary
except BudgetExceeded as e:
    return partial_summary_so_far()

b exposes the same fields as a meter (b.total_tokens, b.cost, b.total_calls), so you can report and cap in one scope. Budgets and meters nest freely; an inner budget bounds a sub-computation while an outer meter still sees the whole thing.

Automatic scheduling, batching, and distribution

The other half of budget-consciousness is efficiency: making model calls fast without the programmer wiring threads, queues, or load balancers. By default a model has one warm instance and requests run on it. When a program issues many requests at once — generate_batch(prompts), or the SDK chat_batch — the scheduler distributes them across the configured resources: the local instance plus any remote API endpoints.

Configuration is optional and lives in sema.toml; the default is a single local instance:

# sema.toml — all optional
[scheduler]
endpoints = "https://api.example/v1/chat/completions,https://b/v1/..."
max_batch = 16    # requests coalesced per flush (the batching-window knob)

Round-robin partition across resources is deterministic and order-recoverable — results merge back in submission order.
max_batch caps how many requests coalesce into one flush.
Remote resources are I/O-bound, so their shares are dispatched concurrently across OS threads (payloads are plain strings — safe to send); the local share runs on the main thread.
A failed or unreachable endpoint degrades to a labelled result rather than crashing the batch.

The scheduler is substrate: it is deliberately not exposed as threads, queues, or futures. An ordinary program calls generate_batch and gets automatic distribution; experts tune endpoints / max_batch or supply a custom resource. This pairs naturally with simulate and its budget tokens=…, time=… descriptor clause, which bounds a single generative site, whereas with budget(...) bounds a whole scope.

Failure modes

A budget scope overspends → BudgetExceeded at the call that crosses the cap; no silent overrun.
A remote endpoint is unreachable → a labelled degraded result in the batch, not a crashed batch.
Pricing not configured → cost fields report from the available data; configure [pricing] per_token in sema.toml for accurate u.cost.

How it’s checked

Model calls run through the effect-handler stack, so metering and budgeting are handlers — the same mechanism that powers record/replay and mocking.
Every model call is journaled with its usage, so sema run (and replay) reproduce spend exactly.