Define an experiment in YAML + SQL. A rigorous engine computes the cumulative effect, its confidence interval and power over the experiment's lifetime — and refuses to call a winner until the estimate stabilizes.
Assignment, variants and comparisons in one file, referencing a reusable library of metrics. Everything correctness-critical — the cohort join, the window filter, per-unit dedup, alpha — is packaged, never hand-repeated. Like dbt, like detectkit.
Ported from a battle-tested legacy engine and improved deliberately — never a number changed silently. Each method is a plugin: relative & absolute effects, CI, p-value, MDE/power, multiple-testing correction.
The signature chart: cumulative effect + CI, one point per day since launch. The band
tightens as sample accrues. The daily view peeks — so abk validate measures
the real cumulative-peeking FPR, and the readout refuses a pre-horizon WIN / LOSE.
Every metric resolves to exactly one state — colored from the same five tokens across the chat readout, the HTML report and the cockpit. SRM is a hard gate: it is checked before any effect is trusted.
abk validate runs an A/A false-positive + power matrix that measures your
method's real α — including the honest cumulative-peeking FPR — not the
nominal 5% you assumed.
abk explore is a local, chart-first cockpit: turn method knobs — CUPED,
stratification, alpha — and watch the stabilization chart recompute live, with A/A
calibration always in view. The priority interface, built first.
Idempotent and re-runnable — an experiment is a finite, deterministic recomputation. Run it by hand or schedule it with Prefect.