tokensitterA
Paired tasks
Run baseline and optimized versions on the same task and repository state. Keep model versions, tool access, acceptance criteria and stopping limits explicit. Randomize run order and use repeated trials where possible.
Features/Methodology
The evidence standard
Make the comparison.
Keep the context.
A smaller token count is an observation. A savings claim needs a baseline, a quality outcome, and a clear definition of cost.
Lab method documented · 27 Oct 2026
The current bench protocol
Paired tasks.
Observed checks.
The documented bench compares each arm with the baseline on the same task and repetition pair. Cost change is the relative change of the arm's total cost against the baseline's total cost. A seeded percentile bootstrap (2000 resamples) over the pairs gives the confidence interval; a row with fewer than 8 pairs is indicative only. Pass rates use Wilson intervals; differences use Newcombe intervals. API-error runs are excluded and listed. Other feature-specific studies use their own methods.
Scope. The current bench is a synthetic direct-API agent, not Claude Code, with relay logic imported. Routing uses a simulated clock (15 s per call) and the classifier's cost is omitted. Repeated tasks may be correlated: consider task-cluster uncertainty before generalizing. No result here establishes quality on real customer sessions.
Recommended experiment design
B
Cache-aware conditions
Declare cold-cache and warm-cache conditions separately. Record cache reads and writes; avoid silently giving one arm a warmed prefix. Keep provider outages and rate limits in the run record.
C
Feature isolation
First compare one feature at a time. Then test the full configuration. Use ablations to understand overlaps; never add individual percentage reductions as if they were independent.
Quality
Quality is an outcome,
not a reassuring adjective.
Use deterministic tests when available, plus blinded task-specific review for answers that cannot be judged mechanically. Define success and allowed degradation before seeing the savings.
Task pass rate
Report baseline and optimized passes, failures and sample sizes. “No statistically significant difference” does not establish equivalence.
Equivalence claims
Choose a non-inferiority margin and a suitable analysis in advance. Report uncertainty and whether the study can support the claim.
Rework
Include retries, fallback models, extra user turns and repairs in cost and latency. Do not drop failed runs to improve the savings percentage.
Known limits
Publish workload boundaries and failure examples. A coding benchmark is not evidence for all writing, reasoning or multilingual work.
Cost
Count every class
at its own rate.
Use a dated price snapshot, the model used for each call, and correctly separated token classes. Preserve currency and time window.
Cost = Σ(input × input rate + output × output rate
+ cache write × write rate + cache read × read rate)
+ applicable request, tool and provider charges
Net cash benefit = baseline cost − optimized cost − incremental service fees
+ cache write × write rate + cache read × read rate)
+ applicable request, tool and provider charges
Net cash benefit = baseline cost − optimized cost − incremental service fees
Only use disjoint usage fields: some provider totals already include cached tokens. Normalize to a documented schema before applying the formula.
Labels
Three meanings.
Three labels.
Savings can mean different things. A figure carries the label that says which one it is.
API cash saved
A reduction in billed API cost for equivalent completed work, under a defined counterfactual. Explain which leg is observed and which is estimated.
Plan usage value
Avoided subscription usage valued at API list prices. This may describe allowance efficiency; it is not a discount on the subscription bill.
Potential savings
A counterfactual for unoptimized traffic. Label as estimated, disclose assumptions, and keep separate from achieved historical results.
The record
Publish the record,
not only the headline.
Report paired per-task deltas, aggregate totals, medians, spread and uncertainty. State the bootstrap unit and preserve task-level clustering when replicates share a task. Do not conflate a task-average percentage with aggregate spend reduction.
Required evidence record
feature + configuration + code revision
run date + model version + provider + price snapshot
dataset + task count + repeats + run order
cache condition + stopping and retry policy
baseline and optimized usage by token class
costs + fees + duration + failures + quality outcomes
aggregation method + confidence interval + limitations
raw report location + reproduction instructions
feature + configuration + code revision
run date + model version + provider + price snapshot
dataset + task count + repeats + run order
cache condition + stopping and retry policy
baseline and optimized usage by token class
costs + fees + duration + failures + quality outcomes
aggregation method + confidence interval + limitations
raw report location + reproduction instructions
See the standard
applied to each feature.
Each feature page states its baseline, quality outcome and limits.