Overview

ab-analysis-kit (import package abkit, CLI abk) is a Python library and command-line tool for A/B experiment analysis. It is dbt-like: your experiments and the reusable metrics they measure live as declarative YAML + SQL in a project directory, and you run them with one command. The statistical core is pure numpy (no pandas), and ClickHouse, PostgreSQL and MySQL are all first-class — only the connection and the SQL dialect of your metric queries differ between them.

abkit is the sibling of detectkit: the same design DNA (CLI-first, declarative config, self-contained reports, a chart-first tuning cockpit) with detectkit’s anomaly-detect stage replaced by a statistical compute stage, and the primary entity flipped from metric to experiment (architecture.md).

A directory becomes an abkit project when it contains an abkit_project.yml. Database connections live in profiles.yml, with secrets pulled from the environment via ${ENV} / env_var(...). The abk CLI exits non-zero on failure, so it drops straight into an orchestrator (cron, Prefect) as the unit of automation.

The primary entity is the experiment

Unlike a metrics tool, abkit’s primary entity is the experiment. One experiment YAML declares:

• its cohort — the assignment/exposure source (abkit reads it; it does not randomize for you),
• its variants and the expected traffic split,
• its window and cadence (a pinned start date, a horizon end date, and how often cumulative cutoffs are computed),
• and a list of comparisons — each pointing one reusable metric at one statistical method.

Metrics are a shared library: a metric is its own YAML + SQL file (a one-row-per-unit query), and many experiments can reuse the same metric (declarative-config §2).

One namespace, globally unique names. An experiment name and a metric name share a single namespace and are the database key — not the filename. An experiment cannot share a name with a metric.

A typical project looks like this:

my_project/
├── abkit_project.yml      # project config (statistical defaults, limits)
├── profiles.yml           # database connections (secrets via ${ENV})
├── experiments/           # experiment YAML — the PRIMARY entity
│   └── example_signup_test.yml
├── metrics/               # reusable metric YAML (one per metric)
│   ├── example_signup_cr.yml
│   └── example_arpu.yml
├── sql/                   # shared SQL (assignment sources, metric queries)
└── seed/                  # the synthetic example dataset

The pipeline: load → compute → readout

Every abk run executes three stages per experiment (architecture.md):

1. load — reads your assignment SQL and joins metric SQL against it live (the default — nothing is persisted; set assignment.cohort_copy.enabled: true to keep an incrementally-updated _ab_exposures copy instead), then runs each metric’s SQL against your source database over the cumulative window. Each metric query joins the cohort through the packaged assignment macro, so you never hand-roll the join: import it and select the exposed units, for example {% import 'abkit_assignment.jinja' as ab %} then join ab.exposed_units(...). One row per unit goes in; additive aggregates come out (declarative-config §4).

2. compute — runs the configured statistical method at each planned cutoff, gates for SRM (sample-ratio mismatch), and writes one row per (experiment, metric, variant-pair, method_config_id, end_ts) to _ab_results. This is idempotent: cutoffs already computed are skipped via an anti-join on end_ts, honoring the data_lag completeness watermark (cumulative-intervals §2).

3. readout — the WIN / LOSE / FLAT / INCONCLUSIVE decision. It is recomputed at render time from _ab_results (never persisted), applying the two-tier alpha correction and the SRM / horizon gates (data-contract-and-reporting §5). You see it via abk run --report, the abk explore cockpit, and any BI tool pointed at _ab_results.

The statistical methods are plugins — twelve are registered today (t-test, paired-t-test, z-test, cuped-t-test, paired-cuped-t-test, ratio-delta, plus six bootstrap variants). Nothing in the pipeline special-cases a method by name; a comparison just names the one it wants. See compute methods.

Reading the verdict

The readout turns a cumulative effect and its confidence interval into one of four calls, driven by the experiment’s main metric while guardrail metrics watch for regressions (data-contract-and-reporting §5):

Verdict	Meaning
WIN / LOSE	The CI excludes zero in one direction and is stable (excluding-then-recrossing zero is not a winner).
FLAT	The CI includes zero and the comparison is adequately powered — a real “no difference”. Requires a min_effect on the comparison to distinguish it from noise.
INCONCLUSIVE	Anything else — most often not yet enough data / power.

Two states can withhold a verdict entirely:

• SRM (sample-ratio mismatch) — a χ² gate (anytime-valid multinomial below a 1-day cadence). If the observed arm split does not match the expected split, randomization or the cohort query is broken; abkit sets srm_flag and blocks the decision. Check SRM before trusting any effect.
• insufficient data — the comparison has not accumulated enough to speak.

A crucial default: fixed-horizon CIs are only valid at the planned horizon. Reading the daily cumulative chart early and stopping (“peeking”) inflates the false-positive rate, so by default the readout withholds WIN/LOSE (and FLAT) before the horizon. Peeking-safe, always-valid intervals are one opt-in toggle away — sequential: {enabled: true} on a sequential-eligible method — after which pre-horizon WIN/LOSE become legitimate (statistics-changes §4). See sequential analysis.

The cockpit-first workflow

The recommended way to work is hands-on and iterative:

1. abk init my_project scaffolds a project with a runnable seed example, so abk run --select example_signup_test produces real results on a fresh machine.
2. abk run loads and computes; add --report to emit a self-contained HTML readout per experiment.
3. abk explore --select <experiment> serves the interactive cockpit on localhost. You turn the method’s knobs on your real persisted series and watch the cumulative effect and CI stabilization recompute live, with the A/A calibration status always visible. Only on an explicit Apply does it write the tuned config back into the experiment YAML (archiving the previous file under experiments/.history/). This is the primary interface — prefer it whenever you want to be in the loop. See explore.
4. abk validate --select <experiment> answers “is this method actually calibrated on my data?” by running placebo A/A splits (label permutation) on the experiment’s own cohort and reporting the empirical false-positive rate against α — both single-look and the honest cumulative-peeking rate — plus power and coverage (aa-false-positive-matrix §1). It is an audit, not a config lint. Its results light the explore cockpit’s calibration chip.
5. abk plan --select <experiment> sizes an experiment before it runs — read-only required-N / achievable-MDE / achieved-power at the effective alpha.
6. abk dashboard (alias abk ui) serves the project-level view once you have more than a couple of experiments: one row per experiment (verdict, effect, sparkline), with buttons that spawn the very commands above as subprocesses, plus an editor that creates, edits and deletes the experiment YAML in place — validated at both levels before it lands, archived under experiments/.history/, and round-tripping your comments and layout verbatim. It computes no statistic and takes no pipeline lock: every verdict on the page is the readout’s, and every job is a real abk subprocess doing the work in its own process. See dashboard.

The full command surface is init, init-claude, run, explore, dashboard (alias ui), validate, plan, unlock, clean, verify-incremental, and test-report (cli-and-dx §1). Two things that are easy to confuse: abk run --steps validate is the config lint (no database), while abk validate is the A/A false-positive matrix.

Where results land: the _ab_* tables

abkit creates its internal state tables automatically as needed (no manual migration; _ab_exposures only under assignment.cohort_copy.enabled — the default no-copy mode never creates it). They live in the profile’s internal_database / internal_schema, separate from the data_database your queries read. The one to know is _ab_results — the stable, documented BI contract: one row per cumulative cutoff carrying the effect, CI, p-value, per-arm stats, SRM flag, and horizon marker. Point Grafana / Lightdash / Metabase / Superset here; abkit owns the numbers (data-contract-and-reporting §2). The other tables (_ab_exposures, _ab_experiments, _ab_aa_runs, _ab_tasks, _ab_unit_state) are covered in visualizing results.

AI-assisted onboarding

Running abk init-claude installs assistant context into your project — a managed block in CLAUDE.md, reference rules under .claude/rules/ab-analysis-kit/, and task skills — so an AI assistant can help you author experiments and metrics, tune methods, and debug a verdict against your own project. Re-run it after upgrading abkit to refresh the context.

Where to go next

Get running

• Installation — install ab-analysis-kit (Python ≥ 3.10) and pick your database extras.
• Quickstart — abk init && abk run to a first result on the seed dataset.

Build your project

• Configuration — abkit_project.yml and statistical defaults.
• Databases — profiles.yml, connections, and the _ab_* internal tables across ClickHouse / PostgreSQL / MySQL.
• Experiments — variants, split, comparisons, cadence, and the horizon.
• Metrics — the reusable metric YAML, metric shapes, the one-row-per-unit SQL, and the assignment macro.
• Compute methods — choosing and tuning t/z/CUPED/ ratio-delta/bootstrap methods, their params, and method_config_id identity.

Run, read, and trust the numbers

• The explore cockpit — live tuning and Apply.
• The dashboard — the project-level cockpit (abk dashboard, alias abk ui): one row per experiment, buttons that spawn abk commands, and creating / editing / deleting the experiment YAML in place.
• Reading a readout — WIN/LOSE/FLAT/INCONCLUSIVE, SRM, and the horizon gate in depth.
• Sequential analysis — always-valid CIs and safe peeking.
• Validate (A/A) — proving a method is calibrated on your data.
• Plan — pre-launch sizing.
• Visualizing results — the _ab_results contract and BI recipes.

Reference

• CLI reference — the authoritative command surface: every abk command, its options, the two-level selector model, and exit behavior.
• Internal tables — the _ab_* schema in full, including the _ab_results BI contract, column by column.

See the examples for end-to-end walkthroughs.