coder: 5.66× cost efficiency on fable 5 & SWE quality

1 · Cost efficiency: two independent levers

coder lowers billed cost through two separate mechanisms, and this page keeps their accounting separate: compression (measured) and saver tiers (a pricing fact). Each lever has its own formula, its own evidence, and its own row in the methodology table.

1.1 · Compression (measured)

The pipeline delivers the model a compact representation of the same session. Per request, the ledger records the uncompressed baseline and the actually-sent input: factor = baseline ÷ sent. Median 2.83× (p10–p90 2.26–3.08×) on the latest full ledger window (2026-07-23, n = 7,222 requests).

Quality evidence for this lever: the SWE / SWE Pro compression ON vs OFF A/B (§2) — identical tasks, official Docker grading, pipeline on versus off — is the quality evidence for this compression: resolution outcomes track the uncompressed arm within noise. Formula and sample window for the 2.83× itself are in the methodology table; this paragraph is about the compression lever only — saver has its own section below.

💰 #.#x = savings rate — the bottom-left status-bar indicator in this unedited screen recording is the live compression savings multiple: it starts at the 2.0× floor and climbs to 5.1× by run end (peak 6.1×) as cached/compressed context accumulates — 92.4% cached-input share across the run, per the session rollout ledger (12 API usage entries, 287,269 input tokens). The task is a real public SWE task on pallets/flask (CLI feature + tests, all green). Playback is sped up 8× (annotated top-right); content untouched.

1.1b · Effective context window (same measurement, second dividend)

Compression has a second effect beyond cost: the model's physical context window holds the compressed representation, so its effective capacity scales by the same measured factor. At the measured median of 2.83× (p10–p90 2.26–3.08×), a 200K-token physical window carries roughly 450K–620K tokens of session content (200K × 2.26–3.08). The practical consequence: long sessions rarely hit context-window pressure — less compaction, less dropped context, stable long-running work.

• Conversion line: effective capacity = physical window × measured compression factor. Same data source as §1.1; no separate measurement.
• Real example from the same ledger day: one 2026-07-23 session accumulated 58,815,232 tokens of would-have-sent input, delivered as 18,622,941 actually-sent tokens (3.16×) — a working day of context traffic carried at roughly a third of its raw size (session ledger rollout-2026-07-23T09-27-46-019f8fcd….jsonl, cumulative imgctxOriginalInputTokens vs sent).

1.2 · Saver tiers (pricing fact)

Independently of the measured compression lever above, coder offers two half-price delivery lanes. Formula: billed = 50% × standard lane price — a pricing-schedule fact that needs no test evidence. It enters the headline only as the official priced factor (50% of list ⇒ 2.00×) and is never blended into the measured compression figure itself:

• batch — asynchronous submit-and-retrieve for non-interactive workloads (bulk evaluation, background analysis, scheduled jobs): 50% of standard cost.
• flex — the same synchronous streaming interface, scheduled on flexible capacity, for work that doesn't need reserved-capacity delivery: 50% of standard cost.
• Same models, same quality, same output — the tiers change scheduling, not results.
• Headline accounting: headline factor = compression (measured median 2.83×) × saver (official pricing 2.00×) = 5.66×. The saver term is taken from the official price schedule (batch/flex at 50% of list ⇒ 2.00×), not from this-window traffic — the 2026-07-23 headline window itself carried no saver-lane traffic, which is exactly why the term is priced, not traffic-weighted.

Fleet-wide pure compression (conservative cross-check)

The numbers above come from one machine's ledgers. As a cross-check at fleet scale, we measured pure token compression only across 7 days of real gateway traffic:

2 · SWE / SWE Pro: compression ON vs OFF

The efficiency pipeline is only worth shipping if it preserves output quality. We ran the industry-standard SWE-bench suites as an internal A/B — identical tasks, identical models, official Docker grading harness, pipeline ON (compressed) versus OFF (direct) — both suites rerun on the same current pipeline build.

Protocol notes: fixed instance samples (Lite 10 / Pro 18 pairs); both arms of a pair run the same instance, same model, same batch. Verdicts come from the official Docker harnesses only (Lite: swebench run_evaluation on princeton-nlp/SWE-bench_Lite; Pro: SWE-bench_Pro-os, jefzda/sweap-images) — no LLM judging. “Agreement” = tasks where both arms reached the same resolved/unresolved verdict. Pipeline locked at origin/main 8a9175d for both reruns.

The single divergent Pro task, navidrome-0488, was settled with a dedicated N=5-per-arm equivalence rerun: ON 3/5 vs OFF 3/5 — the two arms are equivalent within noise (Fisher exact p = 1.0). All verdicts come from the official Docker harness (10/10 runs graded, rc = 0), and all 10 run windows show collapsed_turns = 0 with zero rewrites, i.e. the compression path left the transcripts untouched. Variance accounting: the 4 failed runs split ON 2 + OFF 2 and share the same model-self-error pattern — the uncompressed arm steps into the same holes. Verdict: the original divergence is agentic run-to-run variance, not compression damage (N=5 verdict report). Scope note: N = 5 per arm on one instance, same protocol and instance as §2, pipeline 8a9175d — an equivalence read at small N, not a universal claim.

• SWE-bench Lite (n=10 pairs, current-pipeline rerun): ON 10/10 vs OFF 10/10, agreement 10/10 — zero divergent instances; every pair resolved identically in both arms. Lite skews easy, so this suite measures parity at equal difficulty, not superiority.
• SWE-bench Pro (n=18 pairs, current-pipeline rerun): ON 15/18 vs OFF 16/18; verdicts agreed on 17 of 18 tasks. The single divergent task is settled above (navidrome-0488 N=5 equivalence rerun: ON 3/5 vs OFF 3/5, Fisher p = 1.0, agentic-variance verdict).
• Failure alignment (Pro): apart from the single divergent task, every Pro failure failed in both arms (ansible, one element-web instance).
• Context fidelity: 98/98 answerable recall/state probes correct in both arms, and 0/16 fabricated answers on deliberately unanswerable probes in both arms.

3 · Methodology: where every number comes from

Our rule for this page: a number appears only if it can be traced to a machine-written record or a reproducible script. Formulas and sample windows below.

Known limitations (stated, not hidden)

• The compression term of the headline (2.83×) is measured on one build machine's session ledgers — the saver term (2.00×) is a pricing fact, not a measurement; the fleet-wide cross-check reports pure compression at fleet scale.
• Benchmark sample sizes are small (10 + 18 pairs; the equivalence rerun is N=5 per arm on one instance). We report parity, agreement and within-noise equivalence, not superiority.
• The first-line pricing account blends one machine's one-day cached-input share (94.9%) with official list prices; cache-write premiums are excluded from the blend and stated as such. Effective-price figures are measured-account arithmetic, not a billing quote.
• Run-total dollar figures across arms mix in turn-count variance and are deliberately not used as compression evidence anywhere on this page.
• Context-fidelity matrix is one model, one seed, one day (98 answerable probes per arm).
• Currency conversions in ledgers use an estimated FX reference (frankfurter, 2026-06-30), not invoice FX.