Monitors & Drift

Markdown Open in ChatGPT Open in Claude

A monitor watches the distribution of a function’s outputs over time and answers one question with statistical honesty: has this stream shifted from what it was calibrated on? When it has, the monitor’s on drifted: block runs. This is the construct that keeps calibrated guarantees — every ~= similarity branch and semantics() check — truthful in production, because a calibrated certificate is only honest while the deployment distribution matches calibration.

Monitors are Sema’s answer to silent model degradation: a model that starts drifting does not quietly return worse answers, it trips a monitor that the language sees.

Why drift detection is in the language

A calibrated semantics() guard gives you a statistical(α) guarantee — a bounded false-verdict rate under exchangeability, i.e. as long as production data looks like the calibration data. That assumption erodes over time: prompts shift, the model provider updates a checkpoint, the domain moves. Nothing in a conventional stack notices. In Sema the gradual guarantee lattice makes this explicit: a statistical(α) obligation requires an active monitor on its input stream, or it decays to best_effort at the type level. The monitor construct is what discharges that obligation.

Syntax

monitor summary_drift on summarize:
    capture  topics, sentiment, result.embedding      # channels
    baseline from assure                              # reference profile from verification runs
    test     conformal_martingale(alpha=0.01)
    on drifted:   degrade(summarize, to=models.writer_large); alert("summaries drifting")
    on undecided: log.debug("insufficient evidence")

A monitor is a declaration that attaches to a callable’s output stream (on summarize). It has four parts:

capture — the channels to track. Each must be Semantic, numeric, bool, or enum. Booleans and enums are monitored as categorical counts sketches; an Option[T] channel captures presence plus the inner value.
baseline — the reference profile. baseline from assure derives it from the verification harness sampling the generative component; a string path (baseline "calsets/robot-telemetry@v2") pins a versioned calibration set.
test — the streaming statistic, restricted to the streaming-statistic library so monitors stay O(1) per observation.
on drifted: / on undecided: — the reactions.

Capture channel resolution

Capture expressions resolve against the monitored callable’s signature scope: parameter names, result, and field paths under either. A bare field name abbreviates result.<field> when unambiguous, otherwise it is a compile error naming both candidates. From the verified robotics-cell corpus:

monitor recovery_plan_drift on propose_recovery:
    capture summary.embedding, safe_steps, requires_operator
    baseline from assure
    test conformal_martingale(alpha=0.01)
    on drifted: alert("recovery procedure drafts drifted")
    on undecided: log.debug("recovery monitor undecided")

Conformal test martingales: anytime-valid honesty

The heart of the construct is how it decides. A naïve approach — run a fixed-sample statistical test repeatedly on a stream — is statistically dishonest: repeated tests eventually false-alarm no matter how stable the stream. Sema uses an anytime-valid test instead: a conformal test martingale (a power martingale over randomized conformal p-values computed against the stream’s own history — no external calibration set required). It bounds the false-alarm probability at ≤ α over an unbounded horizon.

Each monitored output is scored into a conformal p-value.
The martingale accumulates evidence; when it crosses the Ville threshold 1/alpha (from test conformal_martingale(alpha=…)), on drifted: runs.
An unscoreable observation runs on undecided:.
A stable stream does not raise a false alarm — under the null, the p-values are uniform, so the martingale does not drift.

Verdicts are three-valued — {conforming, drifted, undecided} — for honesty: the monitor never pretends to a conclusion the evidence does not support. The drift verdict is journaled as monitor.drift.

Reacting to drift

The on drifted: block is ordinary handler code, plus two governed actions:

`degrade(site, to=model)` — swap a model at a `simulate` site

degrade is a typed runtime action, not an ad-hoc callback. It atomically and journal-visibly swaps the model binding used by the named simulate site — scoped to the enclosing container/process — until the site’s monitors report conforming after burn-in, or a human operator resets the binding (an audited action). The target must be a compatible-role pinned model, and the site’s policy envelope must admit model.load for it. degrade targets only model-backed sites.

Deterministic reactions are events

For anything that is not a model swap — a safe stop, a mode change, a shutdown — emit an event. The robotics-cell monitor does exactly this, turning a drift verdict into a deterministic hardware reaction:

monitor telemetry_fault_drift on detect_fault:
    capture frame.pose.embedding, frame.gripper_force_n, frame.vibration_rms, result
    baseline "calsets/robot-telemetry@v2"
    test conformal_martingale(alpha=0.005)
    on drifted:
        # degrade() only swaps models at simulate sites; deterministic reactions
        # to drift are event emissions. Before burn-in this stays an alarm.
        emit CellFaultDetected(order_id=order.id, observed=frame)
        alert("robot telemetry distribution drifted")
    on undecided:
        log.debug("telemetry fault monitor undecided")

Derived monitors for decision sites

Requiring a hand-written monitor for every calibrated ~= branch and semantics() guard would be a tax that pushes authors toward best_effort — exactly the silent degradation the lattice exists to prevent. So the compiler auto-derives an input monitor for any calibrated decision site not covered by an explicit declaration.

Derived monitors are shared by judge identity: all sites keyed on the same (judge hash, calibration set) pair feed one aggregated monitor, because the exchangeability assumption they guard is the same assumption. The monitor population therefore grows with distinct judge+calibration pairs, not with syntactic sites. Each derived monitor is the same O(1)-per-observation mergeable sketch as a declared one and is charged to the module’s sketch-memory budget; sema doctor reports the per-monitor memory/CPU footprint, so the cost of a calibrated site is visible, never ambient. An explicit monitor on the same stream overrides and absorbs the derived one.

`monitor` vs `event` vs `collector`

These three are complementary and easy to confuse:

Construct	Question it answers	May drive control flow?
`monitor`	“Has this stream’s distribution shifted?”	Via `on drifted:` reactions
`event`	“This typed thing happened — react.”	Yes, deliveries do work
`collector`	“Record this value for later.”	No — never

A monitor is not the event system — it computes anytime-valid statistics and yields three-valued verdicts. But a monitor may attach to an event stream (monitor X on <EventType>:) as a capture source, and its on drifted: may emit an event.

Failure modes

Reference profile too small → undecided verdicts, surfaced amber at compile time.
Embedding-model drift (monitor-on-the-monitor) → judge-identity pinning makes a judge change a build event, not silent decay.
A degrade target that is not a compatible pinned model, or a policy without model.load → rejected; the swap does not happen silently.
A calibrated site the compiler cannot cover with a derived monitor → decays to best_effort with a diagnostic naming the missing monitor.

How it’s checked

sema check <project> validates channel types, capture resolution, and monitor-or-decay coverage of calibrated sites.
sema assure <project> samples the generative component to build the baseline prior.
Drift verdicts are journaled (monitor.drift), so replay reproduces exactly when a stream drifted.