Validating with A/A (abk validate)

abk validate answers one question about your experiment analysis: is this method actually calibrated on your data, or does it lie about its α?

It draws placebo A/A splits on the experiment’s own cohort — where by construction there is no true effect — reruns the candidate method(s) over the real cadence grid, and reports the empirical false-positive rate (FPR), power, achieved MDE, and CI coverage. A well-calibrated method gives FPR ≈ α; an inflated one is the single most important thing to catch before you (or a stakeholder) trust a WIN or LOSE verdict.

abk validate is a statistical audit, not a config check. It runs out-of-band from abk run — its own lock, never in the pipeline hot path — and writes only the _ab_aa_runs audit table, never _ab_results or _ab_exposures. The placebo shuffle is in-memory only; it never touches your real assignment.

`abk validate` is NOT a linter

This is the most common confusion, so state it up front. There are two different things called “validate”:

abk run --steps validate — the config lint: schema / SQL / cross-reference checks, no database, no statistics. (--steps defaults to validate,plan,load,state,compute; naming validate alone runs only the lint.)
abk validate — the A/A statistical matrix described on this page: placebo splits, empirical FPR, power, coverage. It reads persisted data and writes _ab_aa_runs.

They share no code and serve opposite purposes. If you want to know “does my YAML parse and do my metric references resolve?”, that is abk run --steps validate. If you want to know “does my z-test actually hold its 5% error rate on this clustered metric?”, that is abk validate.

Prerequisite: a queryable assignment source (and, in copy mode, a prior run)

In the default (no-copy) mode, abk validate resolves the cohort itself: it renders and validates your live assignment source — the same build_cohort_backend factory abk run uses — and permutes labels in memory. No prior abk run is required for the cohort; your assignment SQL and metric fact tables just need data to read.

With assignment.cohort_copy.enabled: true, validate instead reads the last persisted _ab_exposures copy, so at least one prior copy-mode run must have populated it:

abk run --select checkout_flow_v3
abk validate --select checkout_flow_v3

Either way, an empty or not-yet-launched cohort fails loudly with a reason — validate never resamples invented data.

How the placebo split works

The mechanism (aa-false-positive-matrix §1) is a standard permutation-A/A:

Source = the experiment’s own pooled cohort, rendered over the real cadence grid — the same grid, cadence, cohort, and metric SQL the pipeline uses. There is no separate historical window and no exposure-free loader.
Pool the per-variant unit arrays and permute the unit→arm labels. Permuting labels destroys any true treatment effect, so the split is an exact null by construction, while still exercising your real metric SQL and cadence.
A/A (false-positive): repeat N times. Since 0.2.0 the default N is tied to each cell’s effective alpha — max(2000, ⌈200/α⌉), so ≈4000 at the 5% main tier and ≈40000 at a 0.5% secondary tier — keeping the FPR estimate’s relative precision roughly constant across tiers (a flat 2000 starves tight secondary alphas). -n/--iterations remains a hard override for every cell; an auto-N above 100 000 logs a warning but is never capped. Each split runs the candidate method(s) and records whether it falsely rejects H₀ at the effective α. The FPR is the share of placebo runs that flagged significance. The significance rule is the readout’s own CI-excludes-zero test — not the raw reject flag — so z-test and bootstrap edge cases match what the readout would actually say.
Power: with --inject-effect, a known synthetic effect is injected into one placebo arm; the rejection rate becomes power, plus achieved MDE at the target power and CI coverage.
Persist one _ab_aa_runs row per cell, then emit the recommendation.

Seeds are derived from ("aa", experiment, metric, method_config_id, iteration) with no wall-clock input (aa-false-positive-matrix §1), so the FPR numbers are a deterministic, reproducible invariant — the same command on the same data returns the same matrix.

Single-look vs cumulative-peeking FPR

This is the headline of the matrix, because the abkit product is the daily cumulative chart, and a chart invites peeking. The matrix reports two FPRs side by side (aa-false-positive-matrix §3):

single-look FPR — significance at the horizon cutoff only. This is the honest fixed-horizon number.
peeking FPR — the share of placebos whose CI excludes zero at any look across the grid (pre-horizon refusal off, horizon included, so peeking is always ≥ single-look). This models the analyst who eyeballs the daily chart and stops the first time the CI clears zero. It is the optional-stopping hazard, not the official readout verdict.

The readout’s own rule (CI-excludes-zero and stabilized) is the defense against exactly this trap, and validate deliberately does not measure the defended rule — it measures the trap, so you can see how bad naive peeking would be. A high peeking FPR on a low single-look FPR means “turn on sequential” (see below), not “the method is broken”.

Validate also reports effect exaggeration at stop (the winner’s curse): conditional on stopping early, the estimated effect is biased away from zero. FPR alone hides this; it is a first-class column beside FPR.

The headline reads, e.g., “nominal α 5%, real peeking FPR 12.7%”. Very dense grids may be subsampled (cap ~100 points, denser early); the matrix states the (kept, total) count when it did.

Reading the matrix — the three classic failures

A worked matrix over the synthetic fixture (α = 5%, budget = α × 1.5 = 7.5%, 14-day grid; aa-false-positive-matrix §8):

metric (kind)	method	single-look FPR	peeking FPR	power @ δ=15%	coverage	verdict
`arpu` (sample)	`t-test`	5.3%	8.6%	96%	95%	well-calibrated
`conversion` (fraction, `nobs`>1)	`z-test`	42.4%	43.5%	95%	55%	FPR inflated, do not use
`ctr` (ratio)	`ratio-delta`	4.8%	12.7%	100%	95%	calibrated single-look; peeking breaks budget

Well-calibrated — a t-test on a per-unit continuous metric: single-look FPR sits on α, coverage at the nominal 95%. This is the reference row.
FPR inflated — a z-test on a clustered proportion (per-unit nobs > 1, correlated within-unit trials): it pools trials as independent Bernoulli draws, underestimates variance, and gets worse as days accumulate. Coverage collapses to 55%. Reach for a delta-method ratio (ratio-delta) or re-express the metric per-unit.
Peeking breaks a calibrated method — ratio-delta is correctly specified (single-look 4.8%), but 14 correlated looks push the optional-stopping FPR to 12.7%. FPR alone would green-light it; the peeking column exposes the trap. Enabling sequential brings it back toward α.

The --report matrix renders budget-band colors (green in-band, red out), an explicit Recommended row with a one-line rationale, a plain-language per-method verdict, and the peeking headline.

The sequential column

If you set sequential: {enabled: true} on a sequential-eligible experiment (see the sequential guide), abk validate adds the always-valid twin beside the fixed peeking column (aa-false-positive-matrix §8.1). The always-valid confidence sequence widens the CI so it is peeking-valid at every look, pulling the optional-stopping hazard back toward ≈ α:

metric (kind)	method	peeking FPR (fixed)	peeking (always-valid)
`ctr` (ratio)	`ratio-delta`	12.7%	≈5%

The fixed column diagnoses the trap; the always-valid column is the defense, at the cost of a wider interval. This is how the fixed-horizon default stays honest without being changed — and the composed family sweep below carries the same twin.

The composed family sweep

An experiment runs a family of metrics under one shared assignment, corrected by two-tier Bonferroni (compute-time) then a read-time family rule (Benjamini-Hochberg, or Holm since 0.8.0). Per-cell FPR is necessary but not sufficient, so abk validate can also sweep the empirical family-wise error rate and false-discovery rate over one shared union-cohort placebo assignment per iteration, under the same composed rule the readout applies (aa-false-positive-matrix §8.1).

Behavior change in 0.2.0: the family sweep is now opt-in via --family-sweep. It used to run automatically on every multi-metric invocation, silently roughly doubling the run’s cost; a bare multi-metric abk validate now prints a one-release notice pointing at the flag instead.

On the complete null the two rates coincide (every rejection is false) and sit at the composed rule’s nominal rate — ≈ α per tier, so ≈ 2α whole-family under the default two-tier Bonferroni (which protects the main and secondary tiers each at α, by design). Under a read-time scheme the anchor is the members’ own level (≈ α), since BH controls the complete-null family error there and Holm controls the FWER at α by construction. The budget is anchored to that nominal rate, so “over budget” means the methods are miscalibrated (clustering, variance underestimation), not that the correction is loose. It surfaces as one sentinel _ab_aa_runs row and a composed-family band above the report matrix.

With sequential: {enabled: true} on a sequential-eligible family, the sweep mirrors the per-cell recovery at the family level: alongside the single-look fwer/fdr it also reports the peeking pair (fwer_peeking/fdr_peeking — the fixed CI peeked across every look, inflated) and the always-valid pair (fwer_sequential/fdr_sequential — the same looks under the confidence sequence, back toward ≈ the composed nominal). It composes over the same sequential-eligible member set (an all-bootstrap family shows only the single-look column) and lives additively in the sentinel row’s details.family — no schema change, no number change. Only alpha_spending/group-sequential remains a future item.

Command and flags

abk validate --select <exp> [--method <m>]... [--metric <m>] [--iterations N] \
             [--family-sweep] [--inject-effect <rel>] [--scoring fpr|power|mde] \
             [--report] [--notify] [--force] [--profile <name>]

Flag	Meaning
`--select`, `-s`	Experiment selector — name, path glob, `tag:<tag>`, or `*` (repeatable; default all).
`--method`, `-m`	Extra registered method(s) to score beyond the declared comparison — this is the method-grid axis (repeatable).
`--metric`	Validate only this metric (default: every declared comparison).
`--iterations`, `-n`	Placebo A/A splits per cell (default: auto, `max(2000, ⌈200/α⌉)` at each cell’s effective alpha — ≈4000 at 5%, ≈40000 at 0.5%). An explicit N overrides every cell.
`--family-sweep`	Also run the composed multi-metric FWER/FDR sweep — roughly doubles the cost. Opt-in since 0.2.0 (it previously always ran when `--metric` was omitted).
`--inject-effect`	Inject this relative effect (e.g. `0.05`) to measure power / achieved MDE / coverage.
`--scoring`	`fpr` (default) / `power` / `mde` — the objective for the Recommended row only. All columns are always computed regardless.
`--report`	Emit a self-contained HTML matrix report (best-effort). Bare flag → `reports/<exp>__validate.html`; pass a directory or an `.html` path to override.
`--notify`	Send a `calibration_red` notice to the `profiles.yml` channels when a cell’s measured FPR exceeds its budget (best-effort). See notification channels.
`--force`	Take over a held validate lock (use with care).
`--profile`	Profile name (default: `profiles.yml` `default_profile`).

The --method axis lets you compare candidates without editing the experiment. To ask “my declared z-test is inflated on conversion — would ratio-delta or t-test do better?”, score them side by side:

abk validate --select checkout_flow_v3 --metric conversion \
             --method ratio-delta --method t-test

To measure power (not just FPR), inject a known effect:

abk validate --select checkout_flow_v3 --inject-effect 0.05 --scoring power

Notes on behavior (aa-false-positive-matrix; validate.py):

Own out-of-band lock — (experiment, "pipeline", "validate"), distinct from the run lock; clear a stranded one with abk unlock. Because it writes only _ab_aa_runs, it need not serialize behind nightly abk run.
Exits non-zero on any experiment-level (harness) failure, so it is safe to wire into CI. A single unscoreable cell is recorded as a status='failed' row rather than aborting the run. --report and --notify are the best-effort exceptions — a bake or a delivery failure yellow-skips and never fails the validation itself.
--notify repeats itself only when the red cells change. A nightly validation that keeps finding the same inflated cell announces it once; a second cell going red is a new message. Set notify.cooldown_seconds on the experiment to be reminded about an unchanged one.
--scoring sets only which row is marked Recommended (persisted as the mode column); the FPR / peeking / power / coverage columns always compute so the explore calibration chip can light regardless.

The `aa_fpr_budget` override

aa_fpr_budget colors the FPR cells: green in-band, red out. It resolves in this order (aa-false-positive-matrix §“Budget bands”):

the metric YAML aa_fpr_budget: field, then
the project statistics.aa_fpr_budget, then
the built-in default (flag if FPR > α × 1.5).

Set it on a metric to tighten or loosen the band for that metric. The value is a fraction in (0, 1]:

name: conversion
# ... sql, type ...
aa_fpr_budget: 0.06   # this metric is judged against a 6% budget, not the default

Or set a project-wide floor in abkit_project.yml:

statistics:
  aa_fpr_budget: 0.07

The `_ab_aa_runs` audit table

Validate persists one row per cell (run_id = "{run_stamp}:{cell_hash}"; no ReplacingMergeTree collapse). The table is informational and is never pruned by abk clean (aa-false-positive-matrix §7). Columns include:

experiment, run_id, metric, method_name, method_params, method_config_id, mode (the --scoring objective), iterations, alpha (the effective post-correction per-comparison alpha), injected_effect, fpr (single-look), peeking_fpr, power, achieved_mde, coverage, effect_exaggeration, verdict, details (JSON: peeking curve, (kept,total) note, rationale, warnings), status (success | failed), error_message, and created_at.

The persisted alpha is the same effective per-comparison alpha the readout, chip, and Apply seam use — main and secondary metrics land at different alphas under two-tier Bonferroni. That is what lets the explore chip look a cell up by (metric, method_config_id, effective α).

The calibration chip and Auto mode in explore

The real-α signal must not live only in a separate command, so abk validate feeds the explore cockpit:

A persistent calibration chip shows calibrated / FPR = X.X% vs nominal α for the current knob combination (red when out of budget), read from the _ab_aa_runs rows validate wrote.
Auto mode runs validate server-side (reduced N) and greens the live chip in place — no explore restart needed.
Apply is gated when the active method_params have never passed validate — on an empty _ab_aa_runs, every Apply takes the confirm_uncalibrated path. Running abk validate (or Auto mode) is what flips the chip to calibrated, so you can’t ship a mis-calibrated method without seeing the cost.

Gotchas

Editing method_params orphans the calibration. The chip keys off method_config_id; retuning a method starts a new series and the old _ab_aa_runs rows no longer match. Re-run abk validate on the new params.
Peeking FPR is the hazard, not the verdict. A high peeking FPR on a low single-look FPR is a signal to enable sequential — it is not evidence the method is broken.
Bootstrap A/A is the expensive corner. Validate multiplies per-interval compute by N × the method grid. Closed-form methods run in microseconds per split; prefer them for the FPR sweep and lower --iterations for a quick look, raise it for a tight estimate.