Compute methods

Every comparison in an experiment binds one statistical method to a metric. The method is the compute stage of abkit’s load → compute → readout pipeline: it turns a metric’s per-unit values into an effect, a confidence interval, a p-value, and a reject flag, writes them to _ab_results, and hands them to the readout that renders the WIN / LOSE / FLAT / INCONCLUSIVE verdict.

Methods are plugins. You pick one by its registry name and pass its params; the pipeline, database layer, and CLI never special-case a method name. This page is the catalogue: the 12 registered methods, when to use each, their params, the identity rules that decide when a result series is orphaned, and the legacy branches that are deliberately quarantined.

Binding a method to a metric

A method lives inside a comparison, under the experiment YAML’s comparisons list:

comparisons:
  - metric: example_signup_cr
    is_main_metric: true
    method: {name: z-test, params: {test_type: relative, calculate_mde: true}}

  - metric: example_arpu
    method:
      name: cuped-t-test
      params: {test_type: relative, covariate_lookback: 14d}

method is a {name, params} object (abkit/config/method_config.py). Params are validated at config-validation time by instantiating the method — an unknown param, a bad value, or a quarantined combination fails when you run abk run or abk validate, not silently at compute time.

Match the method to the metric shape

A method declares which container family it consumes (input_kind), and it must match the metric’s type or the config layer rejects the experiment (declarative-config §8):

Metric type	input_kind	Method family
fraction	fraction	z-test (proportion / rate)
sample	sample	t-test, cuped-t-test, bootstrap, poisson-bootstrap, post-normed-bootstrap
ratio	ratio	ratio-delta

A mismatch (e.g. a z-test on a sample metric) is a config error with an explicit message. See metrics for how a metric’s type is set.

The 12 registered methods

Family	Registry name	Aliases	Params beyond test_type
parametric	t-test	ttest	calculate_mde, power
parametric	z-test	ztest	calculate_mde, power, interval
parametric	cuped-t-test	cuped-ttest	calculate_mde, power, covariate_lookback
parametric	ratio-delta	—	(none)
parametric, paired	paired-t-test	paired-ttest	(none)
parametric, paired	paired-cuped-t-test	—	covariate_lookback
bootstrap	bootstrap	bootstrap-test	n_samples, stratify, weight_method, stat, pvalue_kind, seed, max_block_bytes
bootstrap	poisson-bootstrap	—	n_samples, stratify, stat, pvalue_kind, seed, max_block_bytes
bootstrap	post-normed-bootstrap	—	as bootstrap (relative-only — see quarantine)
bootstrap, paired	paired-bootstrap	—	as bootstrap
bootstrap, paired	paired-poisson-bootstrap	—	as poisson-bootstrap
bootstrap, paired	paired-post-normed-bootstrap	—	as bootstrap (absolute-only — see quarantine)

Names are case-insensitive and normalise _ to - (Z_Test → z-test).

Choosing a method

Quick decision:

• Binary outcome per unit (conversion, click-through) → z-test — normal approximation of a proportion.
• Continuous mean per unit (revenue, duration, counts) → t-test — the general default.
• Continuous and you have a pre-experiment covariate → cuped-t-test — variance reduction gives a tighter CI and more power at the same N.
• Metric is a ratio of two sums (CTR = Σclicks / Σimpressions) with per-unit correlation → ratio-delta — delta-method variance.
• Heavy tails, a non-mean statistic, or you distrust the normal approximation → bootstrap.
• Unsure → t-test.

Bootstrap methods are slower and more memory-hungry than the closed-form parametric methods. Prefer a parametric method unless robustness genuinely requires resampling.

Parametric methods

Closed-form estimators (normal / t approximation), golden-tested at relative 1e-9 against the captured legacy baseline (statistics-baseline §3).

Param	Applies to	Default	Meaning
test_type	all	relative	relative = lift over control; absolute = raw difference
calculate_mde	t-test, z-test, cuped-t-test	false	also solve the per-arm MDE at power
power	t-test, z-test, cuped-t-test	0.8	target power for the MDE solve (must be in (0, 1))
covariate_lookback	cuped-t-test, paired-cuped-t-test	—	pre-period covariate window, e.g. 14d — identity-bearing
interval	z-test	pooled	confidence-interval construction: pooled (legacy) or score — identity-bearing
interval	t-test, cuped-t-test, ratio-delta, the paired variants	delta	how the relative interval is built: delta (legacy) or fieller — identity-bearing

ratio-delta and the paired variants take test_type and interval.

interval: score — a proportion interval that is valid away from zero

By default the z-test’s interval is effect ± z·SE, with the SE computed under the null (the pooled proportion). That is the same number the p-value uses, so “the CI excludes zero” and “p < α” always agree — but the interval is only a valid confidence set at zero. Under strong arm imbalance the pooled SE can be the smaller one: at a 900/100 holdout it is 24% too small, which inflates the real error rate by 2.7× at alpha: 0.05 and 30× at 1e-4. Pooling is not the conservative choice it is often assumed to be, and the damage grows as the multiple-testing correction shrinks your alpha.

interval: score replaces the interval — and only the interval — with the inversion of the test abkit already runs (Miettinen–Nurminen):

method: {name: z-test, params: {test_type: relative, interval: score}}

• your p-values do not change — the statistic at the null is the same pooled z, so interval: score re-reads the same evidence with a better ruler. (One exception, and it is a fix: a metric with no conversions in either arm has no pooled statistic at all, and now reports p = 1 where the legacy path reported nothing.)
• the interval is valid at every effect size, asymmetric about the point estimate (report it as [low, high], never as ±), and confined to a possible range — a lift can no longer read below −100%;
• the relative interval is the same construction on the ratio scale, so the lift interval, the difference interval and the p-value are one decision;
• empty cells stop being a hole: with test_type: absolute, a metric with no conversions in either arm at an early cutoff now reports p = 1 and a real interval instead of a blank row. Under relative (the default) that row stays blank either way — a lift over a zero baseline is undefined whatever the interval method.

Two things to know before turning it on. It is identity-bearing, so switching starts a new result series (see identity below) — plan it like any method change. And it cannot be combined with sequential: {enabled: true}: the always-valid transform needs a symmetric interval, and config validation refuses the pair rather than quietly widening the wrong shape.

On a relative metric with few conversions the row carries a weakly identified warning: the precision of a lift is governed by conversions, not traffic (z·√(1/x₁ + 1/x₂)), so ten times the users at a tenth of the rate buys nothing. The interval is still reported — the warning tells you not to read it as a measurement.

interval: fieller — a lift interval that agrees with its own p-value

The mean-based methods report a relative lift as θ̂ ± z·SE, with the SE computed at the observed lift. It is a valid interval, but it is a different test from the p-value beside it: “the lift is 0” and “the difference is 0” are one hypothesis, and a Wald interval on the ratio can contain 0 while the difference test rejects. Two things follow, and the second is the practical one:

• your report can say “significant” and show a lift interval covering zero;
• the two-sided coverage is right while the tails are not. Measured: at a control-mean CV of 5% the interval misses low 1.7% of the time and high 3.3% — against 2.5% bought on each side. Every abkit verdict (WIN, LOSE) is a one-sided claim, so the error rate you actually run at is up to 1.6× the one you configured. It does not depend on the true effect, which is why an A/A run cannot see it.

interval: fieller inverts the test at every candidate lift instead of only at the observed one:

method: {name: t-test, params: {test_type: relative, interval: fieller}}

• the lift interval and the p-value become one decision — under fieller the relative p-value is the absolute comparison’s, so “the interval excludes zero” and “p < α” cannot disagree;
• both one-sided error rates land on 2.5%, at every control-mean CV;
• the reported lift itself does not change — only the bounds and the p-value;
• the interval is asymmetric about the estimate: report it as [low, high], never as ±.

The honest cost: when your control mean is not clearly different from zero, no bounded lift interval exists at that confidence level (this is a theorem, not an implementation limit — a procedure with guaranteed coverage must sometimes decline). abkit then reports the effect and the p-value with empty bounds and a warning naming the reason, where the legacy branch would have printed a finite interval it could not stand behind. In that state the readout treats the row as a gap, so a comparison can be significant on the absolute scale and still not be called a WIN. In practice this never fires on a well-powered metric (0% of draws at a control-mean CV of 10%; 8.5% at 30%).

Like interval: score, it is identity-bearing (switching starts a new result series), it cannot be combined with sequential: {enabled: true}, and it is only meaningful for test_type: relative — writing it beside absolute is a config error rather than a silent no-op, because it would fork your series for nothing.

CUPED — variance reduction with no extra SQL

cuped-t-test reduces variance by regressing out a pre-experiment covariate. With covariate_lookback set, abkit re-renders the same metric query over the pre-period window (with the exposure filter dropped) and uses the pre-period value as the covariate — you write no additional SQL (declarative-config §3; statistics-changes §5).

The lookback is a fixed whole-day window: it must be a whole number of days and at least 1d (14d, 28d); a sub-day or fractional-day value is a config error, and a lookback under 7d warns. Because the covariate is the pre-period render, a different lookback is a different covariate — and therefore a different result series (see identity below).

Bootstrap methods

Resampling estimators with a percentile CI and a sign-based p-value, reproduced verbatim from the legacy engine (statistics-baseline §4). Results are reproducible, not random — see seed below.

Param	Default	Identity?	Meaning
test_type	relative	yes	relative | absolute
n_samples	1000	yes	number of resamples (≥ 1)
stratify	false	yes	resample within strata (needs a strata column on the metric)
weight_method	min	yes	how per-stratum weights pool (min / mean); not on Poisson
stat	mean	yes	statistic resampled: mean or median built in (Poisson is mean-only)
pvalue_kind	sign	yes	sign = legacy 2·min(P(boot>0), P(boot<0)) (default); plugin = (#extreme+1)/(n+1) smoothing (statistics-changes §2)
seed	—	no	RNG seed — identity-excluded
max_block_bytes	—	no	memory cap for the resample matrix; never changes results

poisson-bootstrap drops weight_method — it reweights with independent Poisson(1) draws instead of index resampling, and post-stratification uses a fixed 1/count unit scale rather than pooled per-stratum weights — and is mean-only; it is cheaper than bootstrap at very large N. post-normed-bootstrap divides out a covariate ratio for variance reduction (the bootstrap analogue of CUPED) and requires a covariate on the metric.

Bootstrap results are deterministic

seed is identity-excluded: two runs at the same config produce byte-identical bootstrap rows regardless of the seed value, because abkit derives a deterministic per-row seed from the row’s identity (experiment, metric, arms, cutoff, n_samples). You get reproducibility for free; changing seed never starts a new series and never re-shuffles a published one.

Paired variants are notebook-only

The five paired methods (paired-t-test, paired-cuped-t-test, paired-bootstrap, paired-poisson-bootstrap, and paired-post-normed-bootstrap) align two arms by unit for same-unit before/after or crossover designs. The v1 declarative pipeline serves independent-arm experiments only — configuring a paired method in an experiment YAML is a validation error that points you at the notebook API. Use a plain (unpaired) variant unless the design is genuinely paired, and reach for the paired methods through abkit.stats directly in a notebook.

test_type: relative vs absolute

test_type controls the estimand for every method. relative (the default) reports lift over control; absolute reports the raw difference. The persisted effect — and any min_effect FLAT-verdict threshold you set on the comparison — is in these units, so choose deliberately: a min_effect: 0.02 means “2 % lift” under relative but “0.02 raw units” under absolute.

method_config_id — identity and orphaning

Each result series in _ab_results is keyed by a method_config_id: a SHA-256 of the method name plus its non-default identity params, with ALGORITHM_VERSION appended only when greater than 1 (declarative-config §7):

method_config_id = sha256( name + json_dumps_sorted(non-default identity params) + [ALGORITHM_VERSION] )

• Editing any non-default identity param orphans the prior series. Changing test_type, calculate_mde, power, interval, n_samples, stratify, weight_method, stat, pvalue_kind, or covariate_lookback away from its default produces a new method_config_id; the old cumulative rows stay stranded. After retuning a method, recompute the experiment (abk run) and prune orphans with abk clean (see the CLI guide).
• seed and max_block_bytes are identity-excluded — they never change the series identity (seed because results are deterministic, max_block_bytes because it only bounds memory).
• alpha is not identity, either. It is the post-correction, experiment-level significance level: two-tier Bonferroni splits main vs secondary metrics to different alphas, while the read-time schemes (Benjamini–Hochberg, Holm) leave the raw alpha on the row and decide across the family at read time. Changing alpha re-decides reject without orphaning the series.

Sequential (peeking-safe) eligibility

The opt-in always-valid confidence sequence — enabled per experiment with sequential: {enabled: true} — makes early reads peeking-safe (statistics-changes §4; see experiments). It applies only to parametric methods, which have the symmetric normal CI the transform inverts. Bootstrap methods are not sequential-eligible: their percentile CIs are asymmetric, so enabling sequential leaves them fixed-horizon and the readout still withholds WIN / LOSE / FLAT before the horizon. If you need valid early stopping, choose a parametric method.

The same requirement rules out the two inverted-test intervals — z-test with interval: score and the mean methods with interval: fieller — because both are asymmetric. Either combination is a config error naming both settings, not a silent downgrade: the transform recovers the standard error by inverting the CI width, which for an asymmetric interval would produce a confident-looking number that is not a standard error.

Quarantined branches

Some legacy method/param combinations are known to be numerically broken. abkit raises QuarantinedMethodError rather than returning a wrong number — it never silently substitutes a different estimator (statistics-changes §3):

• poisson-post-normed-bootstrap — the whole method is quarantined. The legacy class did no post-normalisation (it was a verbatim copy of poisson-bootstrap), so the name refuses to resolve. Use poisson-bootstrap for identical behaviour, or ratio-delta for a principled ratio estimator.
• post-normed-bootstrap with test_type: absolute — the legacy absolute branch is an unusual estimand; only relative is reproduced. Use relative, or ratio-delta.
• paired-post-normed-bootstrap with test_type: relative — the legacy relative branch divides by a near-zero denominator (the ratio explodes); only absolute is reproduced. Use absolute, or ratio-delta for a ratio metric.

Each error message names the fix. This is a facet of a hard project rule: a statistical number never changes silently — any deviation from the baseline is an owned ALGORITHM_VERSION bump plus an A/A re-validation.

After choosing a method, validate it

Picking a method is a hypothesis; confirm it on your cohort:

• Run abk validate — the A/A false-positive matrix. It uses placebo label-permutation splits of your own data to confirm the method’s false-positive rate is ≈ α and that it has power on this cohort. It is not a linter; it exercises the actual estimator.
• Use abk explore to turn test_type, n_samples, alpha, and the method choice live against your real series before committing the config.