Changelog

All notable changes to ab-analysis-kit will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

Once implementation begins, CHANGELOG.md is authoritative for behavior changes — in particular every statistical deviation from the captured legacy baseline is recorded here alongside an ALGORITHM_VERSION bump and a statistics-changes.md entry (never a silent number change).

[0.8.0] - 2026-08-05

Fixed

• _ab_experiments.contrasts is now actually written, and an existing install can migrate to it (M13 STAT-6, found by the milestone exit gate). STAT-1b added the column to the table model and emitted it from catalog_record, but _EXPERIMENT_FIELDS — the catalog writer’s field whitelist — never learned it, so the value was dropped in silence and BI could not tell a vs_control family from an incomplete run, which is the one thing the column exists for. The column was also declared NOT NULL with no default, a shape ensure_columns refuses: the first 0.8.0 run of any already installed project would have failed with a “drop and recreate” error for a column whose own comment promised it needed none. It now carries DEFAULT 'all_pairs' — additively migratable, and factually right for every pre-0.8.0 row, since the knob did not exist before it — sized, because MySQL maps an unsized String to TEXT and rejects a literal DEFAULT on it (error 1101), which would have replaced a broken migration with a broken CREATE TABLE on that backend alone. upsert_experiment additionally refuses a record with fields it would drop, the check save_results has always made in the other direction.

Added

• STAT-6 — the M13 exit gate (tests/e2e/test_m13_exit_gate.py), plus the batch A/A revalidation the change-control process owes every deviation (docs/research/2026-08-m13-revalidation/). The milestone’s №1 assertion is asserted against the released code rather than against itself: two surfaces captured from a v0.7.0 checkout — the scaffolded project as shipped, and a three-arm five-comparison experiment that reaches every default M13 touched — reproduce here row for row (discrete columns exactly, continuous at rel-1e-9, JSON payload columns parsed first), and the ONLY delta across all 89 persisted rows is the one added catalog column above. The gate also pins that opting into an interval param forks method_config_id while correction: holm re-decides the series in place, that narrowing the contrast set loosens the level without forking anything, and that no method bumped its ALGORITHM_VERSION. New goldens for the new numbers (tests/golden/test_golden_m13_new_numbers.py) anchor each estimator on an independent reference — brentq on the constrained likelihood for the score interval, numpy.roots for Fieller, the step-down definition for Holm — beside a literal pinned at rel-1e-9.

  The revalidation measures every knob pair on the SAME placebo draws, which is what makes its headline readable: interval: score agrees with pooled to the last float on the FPR, the sign split and the power — Z(0) is the pooled z by construction, so they reject the same placebos — and differs only in relative-scale coverage, 93.4% against 90.6% (nominal 95%). interval: delta, by contrast, matches Fieller on both two-sided columns while 64% of its false positives fall below zero at a 5% control-mean CV (76% at 10%): the blindness STAT-2 shipped to fix, reproduced end to end.

• STAT-4 — interval: fieller: the relative lift interval becomes the inversion of the test it already runs (M13). Opt-in and no default moves — interval is a new identity-flagged param on the five mean-based closed-form methods (t-test, cuped-t-test, paired-t-test, paired-cuped-t-test, ratio-delta) whose default delta is the legacy branch byte-for-byte, and no ALGORITHM_VERSION was bumped. z-test is deliberately not among them: STAT-3’s ratio-scale score interval is the exact analogue for proportions. Full deviation record: statistics-changes.md §4.5.

  The defect it closes, and why nothing had caught it. The relative branch reports θ̂ ± z·SE with the variance taken at the observed lift — a Wald interval, and therefore a different test from the p-value printed beside it: “the lift is 0” and “the difference is 0” are one hypothesis, and a report could carry “significant” next to a lift interval covering zero. Worse, its two-sided coverage is nominal while its tails are not: measured over 200k draws, the interval misses low 1.68% of the time and high 3.27% at a control-mean CV of 5% (0.83%/3.93% at 10%), against 2.5% bought on each side. Every abkit verdict — WIN, LOSE — is a one-sided claim, so the directional error rate is up to 1.6× the configured one. The imbalance depends on the denominator’s noise and not on the true effect, which is why an A/A run measures the live experiment’s error faithfully and still reports “calibrated”: both estimators’ A/A false-positive RATES agree to the third decimal (0.0498 vs 0.0499). The column that tells them apart is STAT-2’s fpr_negative_share — 0.66 for delta against the derivation’s predicted 0.659, 0.50 for Fieller.

  What ships. {θ : (a − θ·b)² ≤ z²(V_a − 2θV_ab + θ²V_b)} over exactly the five moments the delta path already had, so CUPED — whose numerator is adjusted and whose denominator is not — is covered by the same code rather than a special case. Under fieller the relative p-value IS the absolute comparison’s, bit-for-bit, which is what makes “the interval excludes zero” and “p < α” one event by construction. The reported lift does not change. Both one-sided error rates land on 2.5% at every control-mean CV up to 10%.

  The honest cost, stated rather than hidden. When the control mean is not clearly distinguishable from zero (z²·Var(m̂₁)/m̂₁² ≥ 1), no bounded lift interval exists at that level — a theorem (Gleser–Hwang 1987), not an implementation limit: a procedure with guaranteed coverage must sometimes decline, and delta’s always-finite interval is why its guaranteed coverage is zero. abkit then reports the effect and the p-value with empty bounds and a warning naming the reason, and the readout treats such a row as a gap — so a comparison can reject on the absolute scale and still not be called a WIN. It fires on 0% of draws at a control-mean CV of 10% and 8.5% at 30%. An empty confidence set (a non-PSD moment triple, i.e. the mixed-ddof anomaly the delta branch reports as a negative variance) gets its own sentence: five causes of missing bounds, five messages.

  Consequences. The interval is asymmetric, so it inherits every STAT-3a refusal with no new surface code — sequential: {enabled: true} beside it is a level-2 config error naming both knobs (also at the explore knob and its Apply seam), abk validate scores the fixed columns and omits the always-valid one, explore’s α tier answers with a gap, and every surface renders [low, high] rather than ±. Writing interval: fieller beside test_type: absolute is a construction error, not a silent no-op: it would compute nothing and still fork method_config_id. And abk plan’s Wald sizing turns out to be closer under Fieller than under the shipped default — get_ttest_mde’s relative branch is the null-variance rule, which is Fieller’s own rejection boundary — so its caveat now claims a difference in half-widths rather than in decision rules.

• STAT-3 — interval: score: the z-test’s confidence interval becomes the inversion of the test it already runs (M13, Miettinen–Nurminen in its Farrington–Manning form). Opt-in, and no p-value moves — interval is a new identity-flagged param on z-test whose default pooled is the legacy branch byte-for-byte, and no ALGORITHM_VERSION was bumped. Opting in changes method_config_id, so it starts a new results series for the operator who asks for it and for nobody else. Full deviation record: statistics-changes.md §4.4.

  The defect it closes. The z-test computes its p-value from the pooled (null) variance — and built its interval from the same frozen number. That kept “the CI excludes zero” and “p < α” in exact agreement, which the readout relies on, at the price of an interval that is a valid confidence set at zero only. An SE mis-scaled by r inflates the achieved error rate by (1/r)·exp(z²(1−r²)/2): at a 900/100 holdout the pooled SE is 24% too small — pooling is not the conservative choice it is widely believed to be — which is 2.7× at α = 0.05 and 30× at α = 1e-4. The damage grows as the multiple-testing correction shrinks α, so the two defects compound.

  What ships. One statistic used three ways: the p-value is 2(1 − Φ(|Z(0)|)) (identically the pooled z, hence unchanged), the absolute interval is {δ : Z(δ)² ≤ z²}, and the relative interval is the same construction on the ratio scale, whose statistic at ratio 1 is again that same Z. Coherence is therefore preserved by construction, on both scales at once, while the interval becomes valid everywhere, asymmetric in the direction the sampling distribution actually is, and confined to [−1, 1]. Boundary tables stop being special cases: with test_type: absolute, x₁ = x₂ = 0 — the first cutoff of any sparse metric — now reports p = 1 beside the Wilson zero bound ±z²/(n+z²), where the pooled path returned a NaN p and NaN bounds, a row no reader can act on. That is the one table where a p-value moves (from absent to 1), and the one place this release’s “no p-value moves” needs its exception read aloud; a lift over a zero baseline stays undefined under relative whatever the interval method. Such a row is also newly informative, so under a read-time correction (BH / Holm) it joins the family and tightens the threshold its siblings are judged against — conservative, but a cross-metric consequence of a per-metric knob.

  Consequences, each recorded rather than assumed. The relative interval is the ratio-scale score construction, not the difference interval divided by p̂₁ (which drops the denominator’s sampling error and can return a lift below −100%); it carries a warning — never a suppression — when the lift is not pinned to better than ±50%, stated in conversions, because the width law z·√(1/x₁ + 1/x₂) reads counts and not exposed units. abk plan prints a caveat: it sizes on the normal power formula while the analysis inverts the score statistic, and the two half-widths differ by C·z²/n_arm (measured: C = 4.01 at a 5% baseline, 0.0595 at 30% — 0.15% at 10k units per arm). In abk explore, the ±CI chip renders [low, high] for such an interval instead of a half-width, and a cutoff that cannot be reconstructed exactly at a dragged alpha becomes a reported gap rather than an “approx” point — Tier α re-derives a symmetric normal CI, which for this interval approximates nothing. (Ordinary rows are unaffected: Tier E is exact for them and is tried first.)

• STAT-3a — the asymmetric_ci guard: SE-by-CI-inversion refuses instead of mis-recovering (M13, decision D17). No number moves — no shipped method builds an asymmetric interval — true when it shipped, and superseded above by STAT-3, which adds the first configuration that does. Every refusal below was unreachable at the time and the suite was byte-identical. It is the prerequisite STAT-3 (Miettinen–Nurminen) and STAT-4 (Fieller) cannot ship without.

  The defect it closes. The always-valid transform never receives a standard error: sequential.se_from_ci_length infers SE = ci_length / 2z, which is the SE only for an effect ± z·SE interval. Handed a score/Fieller-type interval it returns the mean half-width over z — a finite number that is not the SE — and sequentialize then centres a symmetric confidence sequence on it. No NaN, no exception, silently wrong, and worst exactly where the new intervals are worth having (the recovery degrades as asymmetry grows).

  What ships. BaseMethod.asymmetric_ci (default False — today’s truth for all twelve methods, pinned by a roster gate) and sequential.require_symmetric_ci, which every inversion entry calls. The method argument is required and keyword-only on se_from_ci_length, se_from_ci_length_array and to_always_valid: a caller may not invert a CI without saying whose it is. The flag is resolved per bound instance, not per class, because STAT-3 ships its interval as an identity-flagged param on z-test — a class-level flag would answer for the default params and miss the configuration it exists to catch; passing a class raises TypeError rather than being answered.

  An asymmetric method is not blocked — it declares supports_sequential: false and its series simply stays fixed, which is what every eligibility gate already tests. The refusal fires only for a method claiming both.

  A twelfth entry point, which the design’s count of eleven had missed: tuning/recompute._alpha_inverted_bounds — explore’s Tier α — open-codes the same premise (se = (right − left) / 2z) instead of calling the helper, and re-derives a symmetric normal CI at the new α from persisted numbers. It takes the same refusal; which UX an asymmetric method eventually gets there stays a named sub-task of STAT-3/STAT-4. An AST gate now fails on any new open-coded inversion and on any guarded call that omits its method.

• STAT-1 — correction: holm, and the FWER claim stated precisely (M13, decisions D7/D9). Two things ship, and only one is code.

  The claim. Under the default two-tier Bonferroni, abkit tests each arm pair’s main comparison at α/P and that pair’s k secondaries at α/(P·k). Precisely: the main tier spends a full α per main comparison — with the usual single main metric that is “the probability of shipping on a spurious main-metric win is ≤ α”, true and unchanged — the secondary tier independently spends a second full α, and the experiment-wide bound is therefore (M+1)·α for M main comparisons, i.e. 2α at the default. It is flat in the arm count and in the number of secondary metrics, and linear only in the number of independent ship decisions the experiment declares; a guardrail_correction: none guardrail (STAT-1c) adds its own raw-α test on top. Those are exactly the levels of a valid serial-gatekeeping procedure whose gate abkit does not enforce, deliberately: enforcing it would suppress a secondary metric exactly when it is most diagnostic (“the main metric is flat but retention dropped”). So the defect was in the claim, not the arithmetic — no number moved, no ALGORITHM_VERSION was bumped (statistics-changes §4.3).

  The code. correction: holm is a fourth scheme (project or experiment, opt-in, default unchanged): the step-down rule, FWER ≤ α under arbitrary dependence, uniformly more powerful than dividing α by the family size. It is read-time, beside Benjamini-Hochberg, over one cutoff’s informative rows — because no fixed per-comparison level can reproduce a step procedure (α=0.05, m=2, p₂=0.03: Holm rejects H₂ when p₁=0.001 and refuses when p₁=0.9). BH and Holm now share one body in composed_significance and differ only in the adjuster. Note that Holm is not uniformly more powerful than the two-tier scheme, whose main tier sits at α/P: that looseness is what the 2α above pays for.

  Fork B is now ratified and visible. A verdict and the interval stored beside it MAY disagree under a read-time scheme — abkit has been in this position under BH since M3 without documenting it. The divergence is one-directional (a family rule is never looser than the member’s own raw alpha), so the observable case is an interval excluding zero under a verdict that declines to call it; readout.evaluate() now attaches an explicit caveat to exactly that pair and sets PairVerdict.family_divergence. The HTML report and abk dashboard render the caveat; every notification channel renders its own sentence off the flag — a message shows an interval beside a verdict with no report to click through to, and the M12 rule is that a notification cannot disagree with the report about the same experiment. abk explore renders neither, by design: it never calls evaluate and shows uncorrected per-comparison inference. The readout’s rationale also stops saying “CI excludes zero” when the family rule is what decided — that sentence named a per-comparison fact as the reason for a family-level decision.

  Three read-time-only behaviour changes the adversarial review forced, each conservative and each applying to benjamini_hochberg as well. (1) FLAT is withheld when the pair’s own interval excludes zero: under a family rule “no cutoff was significant” no longer implies “the interval covers zero”, so FLAT — an affirmative claim of no meaningful effect — would have been called against the pair’s own interval. (2) FLAT’s power story is disclosed as optimistic: the MDE is solved at the row’s raw alpha while the family threshold is α/m at worst. (3) A guardrail_correction: none guardrail leaves the read-time family too — D8’s declaration has two halves, and at read time the family IS the divisor, so honouring only “raw alpha” made D8 a silent no-op under every read-time scheme. A family whose rows carry mixed alphas (reachable because alpha sits outside method_config_id) is now warned about, since the rule then controls the error rate at the loosest of them.

  _ab_results.reject keeps its name and meaning and is redocumented as the pre-family, per-comparison flag it has always been (it is a published BI contract); the composed decision is not persisted at all, because under a read-time scheme it exists only at read time and a stored copy would go stale the moment a metric was added. abk plan now says in its header that a read-time scheme sizes at the raw alpha. A scheme is classified compute-time or read-time in ONE place (stats.correction.READ_TIME_CORRECTIONS), with a roster test asserting the union equals the config literal — the != "benjamini_hochberg" tests that used to be spread across the readout and the A/A runner would each have silently degraded a new scheme to the per-row CI rule.

• STAT-1b — contrasts: an experiment declares the family it claims (M13, decision D15). contrasts: vs_control on an experiment says the family is the g−1 many-to-one contrasts against the control arm (the first declared variant) instead of all C(g,2) variant pairs. The Bonferroni divisor falls accordingly, which multiplies every tier’s level by g/2: ≈ +10 points of power at four arms, +6 at three — more than Holm gives, for a config field rather than new math. Default all_pairs is the pre-0.8.0 behaviour, so a project that changes nothing reproduces 0.7.0 (m13 D1).

  The knob is one declaration with two halves, and shipping either alone is a defect. Under vs_control the treatment-vs-treatment pairs are also not computed — no _ab_results rows, no verdicts, no read-time BH family members. Loosening the divisor while still writing those rows would hand levels bought for g−1 contrasts to a family of C(g,2), i.e. a false FWER claim in the dangerous direction; narrowing the enumeration alone would leave the experiment needlessly conservative. Both halves read the SAME source: ExperimentConfig.contrast_pairs(), now the only place in abkit/ allowed to enumerate arm pairs (AST-gated). Before this WP four modules each carried their own combinations(variants, 2) — the analyze stage that writes the rows and the three read filters (report, dashboard, notifications) — and notify/dispatch.py had predicted in a comment that “a fourth copy should force the extraction”.

  There is deliberately no project-level default: the family a surface reads must never depend on whether that surface resolved one, and it is a statement about an experiment’s design rather than a project-wide policy.

  Rows written for a pair the narrowed family no longer claims are ignored loudly by every read surface — the path a renamed arm already took. The remedy named in that warning is now abk run --full-refresh (which deletes the window before rewriting the declared pairs); abk clean never removed them — it prunes series by method_config_id. The CLI lines that printed a literal C(g,2) divisor (abk run, abk validate, abk plan’s header) now name the family they counted, and the explore client mirrors it — a page dividing by C(g,2) while the server it queries divides by g−1 would disagree with its own rows.

  Four things the adversarial review added, each a hole the knob opened. (1) The planner’s anti-join is now complete at (cutoff × declared pair): widening the family back — or adding an arm — used to leave every historical look touched but incomplete, so the newly declared contrasts existed only from the flip onward while their siblings kept an alpha bought for the narrower family, which is the anti-conservative direction. (2) readout.evaluate() filters undeclared pairs itself, so a direct caller cannot build a read-time BH family the experiment does not claim. (3) The explore cockpit — the fourth reader of persisted rows, and the one without a filter — no longer recomputes series for pairs that are not on the page. (4) A project-level statistics: {contrasts: …} is a loud error rather than a silent no-op, because every neighbour in that block does have a project default. _ab_experiments gained a contrasts column (additive) so a BI join cannot re-derive a divisor no row carries, and the HTML report names the family beside the arm list — otherwise a 4-arm vs_control report shows four arms, three pair blocks and α = 0.05/3 with nothing reconciling the three.

  Opt-in, so no default moves and no ALGORITHM_VERSION is bumped (D1/D4). Since alpha is deliberately outside method_config_id, flipping the knob writes new-alpha rows into an existing series (abk run --full-refresh re-homogenises it) and _ab_aa_runs rows keyed on the old effective alpha read alpha_mismatch until abk validate re-runs them.

• STAT-2 — the A/A matrix records WHICH SIDE each false positive fell on (M13). abk validate now reports fpr_negative_share beside the FPR: the share of single-look false positives whose CI sat below zero. A correct interval is sign-symmetric under the null, so it is 0.5 up to Monte-Carlo noise.

  It exists because the FPR count structurally cannot do this job. Several relative-effect formulas share an identical rejection set at the null, so their measured FPRs agree to the last false positive while their false positives lean opposite ways — the sign is the only column that separates them, and the lean it detects grows as α shrinks, i.e. it is worst in exactly the corrected tier. The number rides in the persisted details JSON (no schema change) and a verdict line names a lean when it clears two gates: at least 100 false positives, and a departure from 0.5 of at least 3 standard errors (sqrt(0.25/hits)) — a test rather than a fixed percentage, which would fire constantly on small cells and never on large ones. Silent otherwise, because a noisy claim about the estimator trains the operator to ignore it.

  Measurement only: no statistical number moves, and the scalar and vectorized engines agree exactly (the share is a ratio of two block-invariant mask counts, so it joins the parity gate’s exact class, not its continuous one).

• STAT-1c — guardrail_correction: guardrails can stop being corrected like growth metrics (M13, decision D8). A guardrail exists to catch harm, so correcting its alpha makes the engine less able to do the one job the metric was declared for — the error points the dangerous way. The new project- (and experiment-) level knob takes inherit (the default, the pre-0.8.0 scheme: a guardrail shares the secondary Bonferroni budget) or none (the guardrail is tested at the raw experiment alpha).

  none is two changes, not one: the guardrail is re-routed and it stops counting towards the secondary divisor, which loosens alpha for the screening metrics that remain in the tier. One main + one screening + one guardrail over two arms moves from main 0.05 / secondary 0.025 to main 0.05 / secondary 0.05 / guardrail 0.05.

  The flip is inert unless correction: bonferroni (every other scheme already hands out the raw alpha at compute time), and it is opt-in because it moves persisted numbers. Two consequences worth stating: alpha is deliberately outside method_config_id, so flipping the knob writes new-alpha rows into an existing series (abk run --full-refresh makes a series homogeneous again); and the A/A calibration chip keys on the effective alpha, so existing _ab_aa_runs rows for guardrail metrics read alpha_mismatch until re-run. Documented in declarative-config.md §6.1.

  No default moved: a project that changes nothing reproduces 0.7.0 exactly, and no ALGORITHM_VERSION was bumped (M13 D4 — under opt-in, an operator who changes a level orphans only their own series, at the moment they ask for it).

Changed

• sequential.enabled together with an asymmetric interval is now a level-2 config error naming both knobs, instead of an AsymmetricCIError raised mid-run once the cohort had already been loaded. The same sentence refuses the combination at the abk explore knob and at its Apply seam, decided off the experiment’s own sequential.enabled rather than off the baked rows — a toggle flipped but not yet re-run would otherwise let the cockpit write the very pair abk run refuses. The STAT-3a guard remains the backstop under all three.
• abk validate DEGRADES on an asymmetric interval instead of failing. STAT-3a shipped a failing cell, which was right while no method could declare the flag and wrong the moment one could: the τ² anchor runs unconditionally at the top of both scoring engines, so the refusal failed every cell — leaving the A/A matrix unable to measure the estimator the change-control process invokes it to certify, and explore’s calibration chip permanently uncalibrated with no command able to clear it. Such a cell now scores its fixed columns and simply has no always-valid column (exactly what a bootstrap method gets), with a note naming that reason rather than “τ² could not be anchored”. The always-valid mode can be built on a score interval — by substituting the sequence’s critical value inside the root-find — but to_always_valid cannot express it: it widens a finished interval, recovering an SE the method does not have. That is a named future extension, not a limitation of the score interval.
• abk plan now binds each comparison’s method to resolve its interval shape, so a comparison whose method params are invalid is refused by name rather than silently sized against defaults it never had.

[0.7.0] - 2026-08-03

Added

• NTF-6 — verdict_change fires, and the whole feature has an exit gate. The kind was declared in NTF-1 and emitted by nothing for four work packages, so on: [verdict_change] matched nothing and was silent for a reason no operator could guess. It is now the narrow view of a readout it always read like: a verdict whose WORD differs from the one last announced — not the first message about a comparison (news, but nothing flipped) and not one re-sent because its SRM gate moved while the word stayed put.

  • tests/e2e/test_notify_pipeline.py runs the whole feature through the CLI: three experiments (healthy / broken sample split / failing pipeline) in one abk run --notify, with the four claims the milestone is worth nothing without — a scheduled run is quiet across separate invocations, the urgent signals reach an on-call channel while routine ones do not, a channel that raises cannot change an exit code, and every one of the six declared kinds is observed on a channel scoped to it alone.
  • docs/guides/notification-channels.md rewritten around what the commands now do rather than what was planned.
  • Zero statistical numbers moved across the whole of M12 (no ALGORITHM_VERSION bump; every number in a message is copied off a PairVerdict or a CellResult).

• NTF-5 — the two signals about the machinery: calibration_red and stale. Both are conditions abkit already detected and only ever printed; neither adds a detector. Both RECUR — the condition is still true on the next run — so they are deduped on WHAT is wrong rather than on the message.

  • abk validate --notify (new flag) sends calibration_red naming every cell whose measured A/A false-positive rate exceeded its budget: the matrix’s “do not use” verdict, delivered to the people not watching the terminal. Best-effort on --report’s terms — a delivery failure never turns a successful validation into a failed one.
  • abk run --notify now also sends stale when a metric’s computed series was more than three cadence steps behind the looks already due when the run planned it.
  • The stale message is retrospective, and says so. The run that detects a backlog is the run that computes the missing looks, so what is behind is the SCHEDULE — a run that never fired, was locked out, or failed — not the warehouse. Its notice word changed from NTF-2’s placeholder “Data is stale” to “Schedule fell behind” for that reason.
  • Dedup is by signature, not by sentence: which metrics are behind, which cells are red — never the lag or the FPR, both of which drift on every run and would re-announce forever. A widened condition (a second metric behind, a second cell red) always sends; a condition that CLEARS is recorded as cleared, so the same one recurring next month is news again rather than a dedup hit against a months-old row.
  • notify.cooldown_seconds (new experiment-level field) re-announces an unchanged recurring condition after that many seconds. It applies to these two kinds only: D2 is unchanged, and a verdict flip never waits for a timer. NTF-3 deliberately withheld this field until a signal consulted it.
  • verdict_change remains the one declared kind nothing emits — NTF-6 either wires it or drops it from the vocabulary.

• NTF-4 — four more channels: discord, teams, googlechat, ntfy (nine in total). Thin adapters in abkit’s own idiom: each takes its platform’s wire format (and its limits) as fact, and its content from the same build_context() every existing channel uses — so a notice renders as a notice on all nine without any of them knowing what a signal kind is. None of the donor’s alerting semantics (severity, anomaly/recovery/no-data) came along; abkit still has none.

  • discord — one embed per readout. Its colour is a decimal int (the one channel that will not take #RRGGBB), and mentions ride in the top-level message content with allowed_mentions, because a mention inside an embed renders as text and pings nobody. They are stripped from the embed body so a handle does not appear twice.
  • teams — an Adaptive Card over a Power Automate “Workflows” webhook, not the retired O365 connector. Two Microsoft consequences: the message posts as the flow’s identity (no per-message bot name or avatar), and the status colour is a named card token, so this is the only channel where the brand hex is mapped rather than passed through.
  • googlechat — Cards v2, where \n is not a line break: text is HTML-escaped and then newlines become <br>, or the message arrives as one run-on paragraph.
  • ntfy — JSON publish to a topic, with the status cue carried by the tag emoji and the priority. The priority override applies only to the urgent kinds (LOSE, a failed SRM gate, errors): a WIN is good news, and the channel must not be configurable into buzzing a phone over one. The body is capped on a byte budget, since ntfy’s limit is bytes and a multibyte body would pass a character count and be rejected by the server.
  • abk test-report now exercises all nine types end to end — a type its smoke test cannot construct and send through is a type that does not really ship.
  • Zero statistical numbers moved (no ALGORITHM_VERSION bump).

• NTF-3 — --notify stops repeating itself. Until now every completed run re-announced the same verdict, which makes the flag unusable on the schedule it was built for. The new _ab_notify_states table records what each comparison last ANNOUNCED, and only a change is sent.

  • The rule (maintainer-signed as D2 before implementation): a change always sends, an unchanged value never re-sends. cooldown_seconds is deliberately not consulted for verdict dedup — a timer that could swallow a WIN→LOSE flip would be a silence, not a crash, and silence is the failure mode a notification system cannot afford. is_in_cooldown ships as the primitive a future recurring signal will need.
  • “Unchanged” means the verdict AND its SRM gate. A pair sits at INCONCLUSIVE for days before its horizon, so a newly broken sample-ratio gate keeps the verdict word identical — deduping on the word alone would have swallowed the SRM alarm NTF-2 just built, on exactly the experiments most likely to need it.
  • A message nobody received is not recorded. If every channel was down or filtered out, no state is written and the next run tries again: an announcement that reached no one must not become history, because nothing re-derives what was never sent.
  • The dedup key is the FULL comparison identity — experiment, metric, arm pair and method_config_id — so re-tuning a comparison starts a fresh announcement history instead of inheriting the previous method’s.
  • abk clean --orphaned-experiments purges the new table with the rest, which resets the dedup. That is not tidiness: an experiment name deleted and later reused would otherwise inherit the old one’s history and have its first real verdict deduped away.
  • Zero statistical numbers moved (no ALGORITHM_VERSION bump).

• NTF-2 — the urgent half: a failed run and a broken split now reach the on-call channel. --notify shipped able to say only one thing (“here is a verdict”), which is the half an operator is least likely to be woken for.

  • A pipeline error is a signal with no readout behind it. A failed experiment now sends a notice carrying the reason. Every statistical field stays empty on purpose and every renderer omits its statistics block — a crashed run rendered as “Effect: N/A · Flat” would be a claim about the experiment, when the truth is that abkit never got to look at it. This path is deliberately not gated on persisted rows the way the readout path is: the absence of a result is exactly what it reports. A notified failure still exits non-zero.
  • SRM is the same message, re-classified — never a second one. When the sample-ratio gate fails, the readout abkit already built answers to both the readout and the srm kind, so on: [srm, error] is a genuine on-call channel (broken splits and failures, no routine readouts) while a channel accepting both kinds still receives exactly one message. Nothing is re-evaluated to produce it.
  • ReadoutData learned kind + notice rather than growing a second payload type, so all five channels support notices through the transport they already have; BaseChannel.send_notice is the seam a channel overrides when its rich rendering assumes a verdict (only email’s HTML card does). It refuses a verdict payload loudly rather than rendering one blank.
  • No sixth brand colour was invented. The five tokens are verdict tokens and a notice is not a verdict, so all notice kinds reuse the --srm token — the one that already means “no trustworthy result here” — and the word plus emoji carry the distinction.
  • Zero statistical numbers moved (no ALGORITHM_VERSION bump).

• NTF-1 — abk run --notify: the readout finally leaves the terminal. abkit/notify/ has shipped five channels since 0.1.0, and the only thing that ever called them was abk test-report, which sends a synthetic readout. A real verdict has never been deliverable. The new abkit/notify/dispatch.py is the seam: after each completed experiment, abk run --notify reads the rows the run just persisted, evaluates them through readout.evaluate() — the same function abk run --report and the dashboard call, so a message cannot disagree with either — and sends one payload per verdict through every configured channel.

  • Opt-in twice over, and the two opt-ins mean different things. The flag is the operator saying send; the new experiment-level notify: block (channels / mentions / on) says to whom, and about what. With the flag and no block, every configured channel receives every kind — an operator who wired up notification_channels: never has to touch experiment YAML to hear from them.
  • Two on: filters that INTERSECT. A channel’s on: (new on notification_channels) narrows what that channel accepts; an experiment’s notify.on narrows what it sends. Neither can re-open what the other closed. Only the readout kind fires in this WP; all six kinds are accepted from the start, so a filter written today does not silently widen when the remaining signals land.
  • Fail-soft is the contract. A channel that raises, one that cannot be constructed (a rotated secret), one that returns False, or a warehouse read that fails — each is one yellow line and the run’s exit code is untouched. The --report precedent, applied to the same loop.
  • Nothing is sent about an experiment nobody computed. No results table, no rows of its own, or only rows for arm pairs the config no longer declares ⇒ silence: evaluate() over zero rows answers INCONCLUSIVE, which is a verdict about data, and delivering it would report a finding where there is not even an observation.
  • Zero statistical numbers moved (no ALGORITHM_VERSION bump) — every number in a message is copied off a PairVerdict.

Fixed

• A finished experiment no longer reports a growing backlog forever. The abk run backlog warning measured the computed series against the WATERMARK (wall-clock − data_lag), which keeps advancing after an experiment’s cutoffs stop at its horizon: a fully computed, past-horizon series reported a backlog of “now − horizon” that grew by a day every day, loudest when there was nothing left to do. It is now measured against the last cutoff that was actually DUE (last_due_cutoff). Found by NTF-5, which would otherwise have routed that warning as a notification to every experiment a team has ever finished. Cosmetic for the warning; load-bearing for the signal.
• A notification_channels: entry can now declare abkit’s own keys without breaking the channel. The block is extra="allow" and every sibling key was forwarded verbatim to the channel constructor, so the new on: routing key would have reached SlackChannel(on=[...]) and failed as “Invalid parameters” — including in abk test-report, which never asked about routing. ChannelFactory.create_from_config now strips the routing keys it owns, and a test asserts every field declared on the config model is classified as either the type discriminator or routing.

[0.6.4] - 2026-08-02

Added

• PERF-1 — the M9 additive read path is no longer silent, and the flip question is settled with numbers. compute.incremental_reads shipped in 0.4.0 behind the note “default false until verify-incremental bakes”, which was true then and was never revisited; meanwhile nothing in abk run or --cost-report ever mentioned the fast path existed. Worse, the scaffold declared state_additive: true on example_arpu while leaving the read flag off, so a default project paid the STATE write and never took the read — the one configuration strictly worse than either endpoint.
  • abk run now warns when a metric is day-additive (so the state step is materializing its moments every run), compute.incremental_reads is written nowhere, and the series has reached six looks. The warning names both ways out. It also discloses the two mirror silences: the flag on with nothing declaring state_additive (it is doing nothing), and how many looks an enabled read fell back to recompute for — the reader’s own warnings name the reason but are deduped per (metric, reason) and so could never report extent.
  • An absent key and an explicit false are now distinguished (via pydantic’s model_fields_set; the field is still a plain bool and every reader is untouched). They resolve to identical behaviour, but only the first is undecided — without the distinction the warning could never terminate, and a nag that cannot be answered is just a different silence.
  • abk run --cost-report prints the counterfactual. Under compute: it adds of which day-additive: — the same measured query deltas attributed to the eligible comparisons only — then what the other read path would do with it. The slice is part of the compute total, never a sibling; adding them double-counts.
  • abk init now scaffolds compute.incremental_reads: true, with a comment saying why and when to set it back. Its seed dataset never backfills, and M9 picked example_arpu as the additive demo deliberately — a scaffold that pays the write and demonstrates the read beats one that demonstrates neither.
  • The library default stays false, now for the real reason. The flag is not a correctness gate: abk verify-incremental reconciles both paths at rel-1e-9 and any state gap falls back to recompute unaided. It guards exactly one thing — an event backfilled later than data_lag freezes in already-materialized day state — which is a property of the operator’s ingestion SLA, not of abkit. The stale field text now says that.
  • The §4.1 criteria were executed and the evidence published in cumulative-intervals.md §4.2: three consecutive clean verify-incremental runs over the scaffold (zero divergences, zero unverified), and fact-row scans of 2.5× / 3.5× / 5.0× / 8.5× / 11.0× at 2 / 4 / 7 / 14 / 19 looks (total fact rows scanned; the COMPUTE stages behind them reproduce N·L(L+1)/2 recompute vs zero incremental at daily cadence, exactly). The threshold is stated in looks, not days — §4.1 said days because it assumed a daily grid, and an hourly cadence re-reads the window 24× a day.
  • Test impact, deliberately pinned both ways. The scaffold flip changed which path the e2e gates exercise by default: the milestone’s №1 assertion (incremental vs recompute persist the same numbers) silently became a comparison of the incremental path against itself, because its “flag off” leg simply appended nothing. Both legs now set the flag explicitly through one helper that asserts the edit landed, and each leg asserts — from --cost-report’s output — that it really took the path its name claims.
  • Zero statistical numbers moved (no ALGORITHM_VERSION bump).
• UI-1 — the abk dashboard cockpit edits experiment YAML. M11 shipped Show YAML as a read of the file on disk and recorded CRUD as “phase 2”; this is that phase. The row’s button is now Edit YAML, opening the raw text in a textarea with Save, Revert and Delete…, and the header gains New experiment and Reload configs. The write goes through the new abkit/tuning/config_files.py in the house order — validate → archive → write: both validation levels run (ExperimentConfig, then validate_experiment_level2 — the §8 matrix abk run --steps validate runs, reference integrity and the SQL render smoke included), the previous file is archived byte-verbatim under <experiments>/.history/<name>/, and the overwrite is atomic. Routes: POST /api/experiment/{save,create,delete}, POST /api/reload, GET /api/experiments.
  • The text round-trips verbatim — comments and layout survive. That is the one thing abk explore’s Apply cannot promise: Apply merges a structured edit and RE-EMITS the parsed document, so comments die and the archive is the recovery. An editor that silently reformatted the file it had just shown you would be worse than none, so the two seams stay different shapes and share only their archive/atomic-write primitives (both land in the same .history tree).
  • A save is refused, not merged, when something else wrote first. The read route hands out a digest of the file; the editor echoes it back and a mismatch refuses the write. Two writers make that ordinary rather than exotic: a second browser tab, and an abk explore Apply — which this dashboard can itself spawn, on the very experiment being edited. A save or delete is also refused while the cockpit has a running job on that experiment: a live abk run has already read the config it is executing, and a live cockpit will overwrite whatever is saved here on Apply.
  • Level 2 is overridable; level 1 never is. A config that pydantic rejects cannot be served as a row, so it is refused outright. A config that fails the §8 matrix is a statement about the whole project — a metric that does not exist yet, an SQL file still being written — and an editor that refuses until the project is coherent is unusable in exactly the situation it is opened for. So the refusal names the findings, offers Save anyway (force: true), and the saved file carries them back as SAVED WITH AN ERROR — abk run will refuse this: ….
  • A file too large to display cannot be saved. The read truncates at 512 kB and now says so with digest: null, editable: false — a digest over a prefix would let a save write that prefix back and silently drop the tail.
  • Deleting removes the YAML only. The experiment’s rows in _ab_results / _ab_unit_state stay until abk clean --orphaned-experiments prunes them, and the reply says so rather than stranding a series quietly; the archive is a -deleted.yml tombstone, so the delete is reversible by hand. A rename (the YAML’s name: changed) is allowed, keeps the file’s path, and warns that the persisted history does not follow.
  • Configs are no longer read once at boot. Every mutation re-resolves the cockpit’s own --select/--exclude from disk, re-bakes the page and returns the refreshed list, so a created row appears and a deleted one leaves without a restart; POST /api/reload (the Reload configs button) is the manual form, which also picks up an edit made by an editor, a git pull or an explore Apply. A reload that fails — a broken sibling YAML, a name collision — keeps the previous selection and rides back as a warning: the write has already landed, and turning it into a 500 would report a successful save as a failure.
• UI-2 — abk ui is an alias for abk dashboard. The donor’s project-level cockpit is dtk ui (its per-metric sibling dtk tune is our abk explore); abkit renamed both and the rename was never arbitrated. dashboard/explore say which surface you want where ui does not, so the canonical name stays and the alias is registered as the same callback object — the two names cannot drift in options or help.

Changed

• The dashboard’s launcher invariant is restated: “computes no statistic and takes no pipeline lock”. M11 wrote it as “computes a statistic, turns a knob, writes a config or takes the pipeline lock”, which folded in a clause no gate ever checked — the AST scan and the route spy have always been about acquire_lock/release_lock. A config write is the operator’s own declaration, not a result: no number on the page comes from it, and it cannot block a pipeline. The gates now also run over the editor routes (no lock, and readout.evaluate is never called), and the token gate’s hand-maintained POST route list is AST-checked against what _route_post dispatches on — it was GET-only, which left the routes that mutate covered by a list nothing verified.

[0.6.3] - 2026-08-02

Fixed

• A renamed paths.experiments now reaches selection — it used to reach nothing. project.paths.experiments has always been a config field, but discovery.select_experiments hard-coded "experiments", and all ten of its callers (run, plan, validate, explore, clean, unlock, dashboard, verify-incremental, test-report) took that default — so a project whose experiments live anywhere else answered “Nothing selected.” to every command, while abk dashboard’s own derivation honored the setting and the two disagreed about which files exist. The directory is now resolved inside select_experiments from the project config, so every caller gets it without a signature change (passing it explicitly still overrides). Deliberately tolerant: discovery runs before the CLI’s config load, so an unreadable abkit_project.yml falls back to the default rather than turning selection into a parse error — the caller that needs a validated project reports that failure itself.

[0.6.2] - 2026-08-01

Added

• PLAN-2 — abk plan --from-history <N d>: a baseline for an experiment that has never run. Sizing needs per-unit moments, and they came only from a previous abk run of the same experiment or a hand-typed --baseline — so the pre-launch case, the one planning is actually for, answered SKIPPED: no baseline. The flag reads each sizable metric over the N whole days before the experiment’s start and derives its moments through the pipeline’s own build_container → SufficientStats path (a hand-rolled np.std would disagree with the persisted std_1’s legacy mixed ddof, making two baseline sources quietly incomparable). No _ab_results number moves and abk plan stays read-only — no lock, no writes.
  • The read is POPULATION-wide, and the plan line says so. The design contract assumed the CUPED pre-period render was “cohort-free by construction” and could simply be reused. It is not: the packaged macro always INNER JOINs the cohort and ab_apply_exposure_filter=False drops exactly one predicate — correct for CUPED (a covariate is per enrolled unit) and useless before launch, where that join returns zero rows. So the render is a new capability: ab_apply_cohort_join (default true) swaps the cohort for a one-row synthetic relation, which keeps the _abk_exposures alias — and therefore ab.variant_col(), ab.stratum_col(), the loader’s macro-usage lint and an unchanged user metric SQL — working, while GROUP BY variant still groups by a relation’s column instead of a bare constant (PostgreSQL rejects the latter). Every default render is byte-identical to before.
  • Precedence is explicit --baseline > --from-history > persisted rows, and whichever wins names itself on the line (baseline mean=… (history 14d @ …)). A planning number whose origin is invisible is the thing to avoid.
  • Two disclosures ride the line: n counts everyone the metric SQL yields, not the units this experiment will enroll; and when assignment.added_filters narrows the real cohort, a population read cannot apply it, so the variance is indicative rather than this experiment’s own.
  • A render failure skips that comparison with the reason on its SKIPPED line instead of failing the command; the interval must be a positive whole number of days (the grain covariate_lookback uses and the pre-period window is aligned to).

Fixed

• abk plan could crash on a large standardized effect (numpy ≥ 2). statsmodels falls back to fsolve when brentq cannot bracket a solve and returns a shape-(1,) ndarray; _as_scalar was introduced for the MDE path, but get_ttest_sample_size, get_fraction_sample_size and get_ttest_power kept a raw float(...). numpy deprecated that in 1.25 and newer numpy raises outright, so an ordinary --baseline arpu:mean=999,std=1,n=42 failed the whole experiment’s plan with “only 0-dimensional arrays can be converted to Python scalars” — while an older-numpy machine only warned. All three now extract through _as_scalar. Value-preserving: the same number the working path already returned, golden tests unmoved, no ALGORITHM_VERSION bump.

[0.6.1] - 2026-08-01

Changed

• PLAN-1 — abk plan sizes CUPED on the covariate correlation it already has. A cuped-t-test comparison was sized on the raw persisted variance and flagged “ρ not persisted”; M9 WP1 made that false in 0.4.0, which persists corr_coef_1/2 on every _ab_results row, while stats/power.py has shipped cuped_adjusted_std + get_cuped_ttest_{sample_size,mde,power} since M1. The planner now routes those three solves through the same deflation validate/scoring.py uses, so required-N for a CUPED comparison drops to (1 − ρ²)× its previous value — a planner-output change, deliberate and documented (no _ab_results number moves, no ALGORITHM_VERSION bump; abk plan remains read-only, takes no lock and writes nothing).
  • The ASN and the runtime consume the same deflated variance. A deflated required-N beside an ASN simulated on the raw variance is one line contradicting itself — an expected stopping size sitting above a requirement computed for a lower-variance estimator, with nothing saying why.
  • Every line names the variance it was sized on, because a required-N whose basis is invisible is the number an operator plans a launch around. Three cases keep the raw bound, each with its own note: no ρ on the row (written before 0.4.0 — still a conservative upper bound), ρ = 0 (a measurement: this covariate reduces nothing, not a missing value), and a ρ that leaves no usable residual variance.
  • A near-perfect correlation is refused, not obeyed. 1 − ρ² < 1e-12 means the deflation factor is decided by rounding noise in a Float64 ρ rather than by the data — and it is the shape a leaking covariate takes. The project abk init scaffolds is exactly that case (its synthetic pre-period value is collinear with the metric, so ρ rounds to 1); an ungated deflation printed “required 10/arm” for a series whose raw requirement is five orders of magnitude larger. A naive |ρ| ≥ 1 gate does not catch it — the persisted value is a hair below 1.
  • A computable-but-implausible reduction is disclosed, not hidden. Above the degeneracy floor a ρ can still imply a >100× drop in required-N (1 − ρ² < 0.01); that is the measurement, so it is used — and the line says “check that the covariate is not derived from the metric”, because this is the number someone sizes a launch on.
  • --baseline <metric>:mean=..,std=..,n=..,corr=0.6 supplies ρ for an experiment that has never run. It is validated at the door (|ρ| < 1 and a usable residual variance) and refused on a comparison whose method applies no covariate — deflating a plain t-test would promise a variance reduction the analysis will never perform.

[0.6.0] - 2026-08-01

M11 — abk dashboard, the project-level cockpit (the implementation record is docs/specs/m11-implementation-plan.md). One row per experiment over the whole selection: headline verdict, effect + CI, p/α, a sparkline of the cumulative series, and buttons that spawn real abk subprocesses (Run — whole experiment or one metric — Unlock, Clean, Explore, Open report) and stream their logs. The server is a launcher — it computes no statistic, writes no config and never takes the pipeline lock; every verdict on the page is readout.evaluate()’s, the same one abk run --report bakes. No statistical numbers changed anywhere in this release: no ALGORITHM_VERSION bump, no statistics-changes.md entry, the golden tests untouched at rel-1e-9.

Added

• M11 DASH-7 — the dashboard exit gate: a real session, end to end. tests/e2e/test_dashboard_session.py drives the server abk dashboard builds over live HTTP — token-gated on every route, a boot payload asserted metadata-only field by field, three distinct row states in one list (computed / warehouse-raises / never-computed) proving row isolation, the on-demand report route, a real abk subprocess spawned through /api/run and polled to a terminal status on absolute offsets, the one-at-a-time refusal against a long-lived occupant, /api/job/<id>/stop, and a whole-session spy proving no pipeline lock is ever taken. No statistical numbers changed.

  The gate found two real defects in already-merged code, both fixed here:

  • A never-computed experiment claimed a verdict. An experiment with no rows of its own — in a project where others had run — reached evaluate() over zero rows and rendered INCONCLUSIVE. That is a verdict about data: on a row nobody computed it says “we looked and could not tell” where “nothing has run yet — press Run” is the truth, and it made the dashboard’s third row state unreachable per experiment. It now reads the same “no data” state a never-run project does; the lock probe still runs, since a crashed first run leaves a lock on an experiment with no rows at all.
  • The “abkit is not installed” warning was suppressed by stale packaging metadata. A checkout whose install was removed keeps its dist-info, so the probe reported “installed” while every spawned job died with ModuleNotFoundError — exactly the case the warning exists for. It now asks the child’s own question through both import mechanisms (a sys.path search with the CWD dropped, then the sys.meta_path finders that answer for a strict editable install) and ignores metadata entirely.

  Also hardened: the payload point caps were only ever tested under a monkeypatched small cap; a 50 000-look series now exercises the shipped constants, so the per-row reply stays bounded however long an experiment peeks.

• M11 DASH-6 — abk dashboard: the project-level cockpit is a command. The DASH-1..5 server, rows and client get their entry point: abk dashboard [--select <sel>]... [--exclude <sel>]... [--window 24h|7d|30d|90d|all] [--no-open] [--profile NAME]. No statistical numbers changed (no ALGORITHM_VERSION bump; this WP adds a launcher, a CI gate and documentation — the compute path is untouched).

  • The whole selection, not one experiment. --select/--exclude resolve exactly as they do on abk run (all experiments when omitted); the one-match restriction is abk explore’s alone. --window sets the INITIAL sparkline window and is validated at startup, before a port is bound, where the operator typed it — the page switches it afterwards without a restart, and every verdict is always the full cumulative series’ either way.
  • Three deliberate divergences from abk explore’s shell, each following from the dashboard being a launcher rather than a reader: a project that has never run serves (explore no-ops — here that is the first case, every row reads “no data — press Run”, and nothing creates internal schema); there is no startup orphan scan (one list_method_config_ids per row would put N warehouse queries in front of a metadata-only boot); and an empty selection warns and exits 0 without serving, the abk run/abk validate idiom.
  • One startup warning, so N jobs cannot fail identically. Every button spawns python -c with '' and the project root dropped from sys.path (deliberately — a click.py in the operator’s project must not shadow the real one), so an abkit that was never installed makes every job die with the same ModuleNotFoundError inside its own log drawer. abk dashboard now says so once, before the page opens, and only when BOTH probes agree — installed-distribution metadata alone false-alarms on a PYTHONPATH install, and a sys.path probe alone false-alarms on a strict editable install. It warns, never refuses: the read-only rows work either way.
  • The Open button’s two server fields are wired here, as DASH-5 left them: the project’s MetricConfigs become the on-demand report’s metric descriptions, and the raw manager (the same one InternalTablesManager wraps, used under the server’s db_lock) lets the no-copy default snapshot the live cohort for the SRM chip’s observed counts. Without them the page still renders — with no descriptions and a silent “0 / 0” beside a green chip, which reads as a broken cohort.
  • dashboard.js is now asserted in the wheel. The CI packaging-DoD job’s bundle namelist is a hardcoded tuple (a source-tree glob would pass on a wheel that shipped none of them); abkit/tuning/assets/dashboard.js joins report.js/explore.js there and in the release-readiness e2e’s self-contained (zero external host) check. The marker-grep, hex-containment and freshness gates needed no edit — verified by re-running all three locally against the new file rather than assumed.
  • The pending-bundle degradation is gone. render_dashboard_html used to fall back to a “run cd web && npm run build” note, because between DASH-3 and DASH-5 the bundle could not exist. Now that it is committed AND wheel-asserted — the exact condition explore.js’s undegrading read was always keyed to — both go through one reader that RAISES: a missing bundle is a packaging bug, and a page telling a pip install user to run a build step blames them for something they cannot fix.
  • Documented in docs/guides/dashboard.md (what it is, the row anatomy, the buttons, the job discipline, and the URL-is-a- credential warning) + the CLI reference, with the packaged init-claude operator rules and the docs site nav in the same PR.

• M11 DASH-5 — dashboard.js: the dashboard’s client bundle (the third committed renderer). web/src/dashboard/dashboard.ts renders the baked boot payload as one row per experiment — name, tags, verdict, effect/CI, p/α, elapsed, a canvas sparkline and the Open / Explore / Run buttons — and fills the statistics lazily. Authored fresh against the donor’s patterns (it ships no TypeScript for its cockpit) over abkit’s own primitives: the sparkline goes through the same shared/chart.ts scale/line code the report and explore charts use, and every color resolves through the one brand-token layer, so no new hex enters the CI containment scan.

  • The boot render touches the network zero times. The metadata-only list paints first with every verdict cell pending; a fixed-concurrency-3 worker pool (the donor’s Vn=3, JS-only — there is no server pool and no cache) then pulls names off one shared queue through GET /api/stats/<experiment>. A rejected fetch or a row.error field paints THAT row’s error cell and the pool keeps going, so one unreadable experiment costs one row. A window change or Refresh supersedes the fill in flight by epoch and aborts it, so a reply that lost the race can never paint over a newer one.
  • The §4 peeking-honesty markers are per row now, on the verdict chip and a one-line note under it (not hidden in the expand): abk-srm-fail for a failed gate, abk-insufficient when the headline look was demoted, and abk-prehorizon when a fixed-CI verdict is withheld before the horizon — a verdict decided early under an always-valid sequence is deliberately NOT marked. verdict: null reads as “no data — press Run” or as an error depending on error, which is the only thing that tells the two apart.
  • _ab_results.insufficient_data reaches the row as a new insufficient field on the DASH-2 shape: the chip needed the state and the alternative was matching English prose in the rationale. It is the HEADLINE look’s own persisted cell, read through the readout’s own _flag (the report’s looser bool(value) would disagree on a "0" string cell), so no display window can move it and the chip cannot contradict the rationale beside it.
  • GET /experiment/<name> (new route) renders the report the Open button opens — the same build_report_payload → render_report_html pair abk run --report writes, on demand for one experiment, never read off disk (a reports/ file exists only if someone passed --report, and would be as old as that run). A cohort source that cannot be read costs the SRM chip’s observed counts and names itself in the page’s warnings rather than denying the page; the DB half runs under db_lock, the bake outside it. Both HTML routes now send Cache-Control: no-store for the reason the JSON ones already did: a report reopened after a Run must not be the pre-run render.
  • The expanded row carries the readout’s own rationale, caveats, warnings and per-arm-pair verdicts, one Run per configured comparison (a secondary metric never appears in verdicts and still needs its own recompute — that is what DASH-4a’s --metric is for), Unlock, a confirmed Clean (it spawns the --execute form), and the read-only YAML. A repaint refreshes only the readout half: rebuilding the whole panel would collapse an open YAML pane, dismiss a confirm box mid-read, and drop an in-flight source reply into a detached node.
  • The job chip reads the server’s own pipeline_active flag (the kind ∈ PIPELINE_KINDS ∧ running rule is never re-derived in JS) and names running jobs from their own status field; a click opens a log drawer that polls /api/job/<id>?offset= on the server’s absolute offsets, so a job chattier than the 5000-line buffer keeps streaming and discarded output is disclosed instead of leaving a hole. A job that finishes re-reads its OWN row — never the list.

• M11 DASH-4 — the dashboard’s job routes: every button is a real abk subprocess. POST /api/run (optionally one metric, DASH-4a), /api/unlock, /api/clean, /api/explore and /api/job/<id>/stop, plus GET /api/experiment-source/<name> for the read-only “open in your editor” affordance. The dashboard stays a launcher: no route computes a statistic, reads a knob, writes a config or takes the pipeline lock — the spawned child does all of that in its own process, exactly as if the command had been typed (pinned by a spy over every job route, not just the module-level AST gate). Run/unlock/clean are one at a time (the atomic spawn_pipeline gate → the donor’s 400 a pipeline job is already running); explore is not gated but deduped per experiment, so a second click reopens the live cockpit’s tab.

  • --select is the experiment’s YAML path, and the route proves it resolves before spawning — a name alone is not safe to spawn. select_configs resolves a bare name by trying experiments/<name>.yml before searching the name: fields, so a file named after another experiment shadows it and the cockpit would have run, unlocked, cleaned or explored something other than the row that was clicked, with nothing anywhere saying so. A path resolves to exactly one file (and satisfies abk explore’s “exactly ONE” by construction), with *, ? and [ in a file name escaped rather than abandoned — experiments/star*.yml left raw would also match a sibling. Every job route then re-resolves that selector through select_experiments, the child’s own resolver, and answers 400 unless it lands on exactly the clicked experiment: the served selection is read once at boot, and abk run/unlock/clean meet an unmatched selector with a warning, “Nothing selected.” and exit 0 — so a renamed or deleted YAML would otherwise show a green, successful Run in the drawer that computed nothing.
  • The spawned command runs this interpreter’s abkit with the project directory NOT importable. A bare abk off PATH would be a different abkit than the one serving the page in an unactivated venv — but pinning sys.executable -m abkit.cli.main has its own defect: -m puts the child’s CWD on sys.path[0], and a job spawns in the operator’s project root. A file there named after anything abkit imports (click.py, yaml.py, statistics.py, …) then breaks every button with a traceback nobody can connect to the click, and an abkit/ directory there runs a different abkit than the one just pinned — neither of which happens when you type abk in a terminal. The child is therefore started through a bootstrap that drops the CWD from sys.path before the first import and names itself abk in usage errors. Every spawned job consequently needs an installed abkit (pip install -e . or a wheel); a bare source checkout fails each job with a ModuleNotFoundError in its own output. Every flag the argv builders pass is test-checked against the click command’s own declared options, so a renamed option fails a test rather than surfacing as an exit-2 job in the drawer. /api/clean spawns the --execute form (a dry run would be a button that does nothing) and never the prompting --orphaned-experiments; the job label is derived from the argv that actually ran, so the drawer shows --execute instead of hiding it.
  • The explore URL scrape requires the scheme. abk explore prints Explore: <experiment name> before the server prints Explore: <url>, so a literal port of the donor’s "Tuner:" in line predicate matched the header and handed the client an experiment name as a URL. One function is both the wait predicate and the extractor.
  • POST /api/explore is a long request by design — it holds the response until the cockpit prints its URL (90 s cap), so a client needs a long fetch timeout there and nowhere else. A repeat click on a starting cockpit waits on the same job but only briefly (10 s), because every waiter holds a request thread; when no URL arrives the 400 distinguishes “exited without serving” from “our deadline lapsed” from “another tab’s cockpit is still starting”, and only a job that request spawned is stopped.
  • An unknown body field is a 400, not a silent drop. A client posting {"full_refresh": true} to /api/run would otherwise get a plain run and no hint that the flag went nowhere. Same discipline as ?window= on the GET side: a blank metric is refused. A null value is the one exemption — it asks for nothing, so a helper that always emits {select, metric} and spells “the whole experiment” as metric: null works on the routes that take no metric too.
  • Status codes extend DASH-3’s map positionally: a bad body is a 400 (an unserved experiment, a selector that no longer resolves, an undeclared metric, malformed JSON — including a body nested deeply enough to raise RecursionError, which is a RuntimeError and would otherwise read as a server defect — and a bodyless POST), a bad path stays a 404 (unknown job id, unknown experiment source), a registry teardown racing a route is a 503 (None means “busy”, which a teardown is not), a job that is no longer running is a 400 while an unknown id is a 404 (the donor conflates both), and a spawn that cannot start at all is a 500 naming the project root it tried to run in.

• abkit.tuning.jobs: spawn_deduped() + url_for(), and children no longer inherit the cockpit’s stdin. The per-experiment dedup is now atomic under the same gate the pipeline gate uses — running_job_for() followed by spawn() is check-then-act across request threads, and a double-clicked Explore button would start two sessions on one experiment, each writing the whole YAML from its own startup snapshot on Apply. stdin=DEVNULL because a child that ever prompts (only abk clean --orphaned-experiments today, which no route spawns) would silently consume the terminal the dashboard was launched from; on a closed stdin it aborts loudly instead, with the reason in its own output.

• M11 DASH-4a — abk run --metric <m>: recompute ONE metric of an experiment. run was the last per-comparison command without the metric axis (explore, validate, plan and verify-incremental all had it), and the dashboard is a launcher — it can only offer what the CLI can do, so the per-metric Run button DASH-4/DASH-5 wire needed this first. It filters comparisons by metric name — the grain abk validate --metric already uses: inside one experiment that is exactly one comparison (a metric binds at most once per experiment), and across a broader --select every experiment declaring that metric. No statistical number moves (M7–M12 posture; no ALGORITHM_VERSION bump).

  • The alphas do not move — the milestone’s #1 assertion for this WP. effective_alphas() derives the two-tier Bonferroni scheme from the config’s comparison list, never from what a given run happens to compute, so the filter is alpha-invariant by construction. Pinned: a filtered run’s rows are byte-identical (alpha included) to the same rows written by an unfiltered run, on a fixture where a leak would halve the divisor.
  • --full-refresh --metric <m> is the real “reprocess this metric” after a SQL or method-param edit: delete_results is already metric-keyed, so the filtered loop deletes and rebuilds only that series and leaves every other series’ results untouched (not merely equal: never re-written).
  • Day state is the one thing a narrowed run may still touch elsewhere. A stale-but-contiguous _ab_unit_state day is invisible to the M9 reader’s gap check, which only detects ABSENCE — so a scoped --full-refresh truncates the withheld metrics’ day state (from the first day the window touches through the end of that series, the same tail semantics an unfiltered refresh has) rather than leaving it: not re-rendered (the operator scoped that cost away), contiguity preserved as a shorter prefix, reads past it falling back to full-window recompute, and the next run that includes the metric re-deriving those days from the current facts. Leaving it alone was a reviewed defect — with compute.incremental_reads on, healing a fact backfill for one metric only made a later routine run sum the stale day and silently persist an undercount (reproduced at 3334.5 vs 3434.5 before the fix; regression-pinned end to end).
  • The run says which of those outcomes applies, per experiment — rebuilt (copy-mode --resync-cohort), truncated (a refresh window), untouched, or “the state step is not selected, so day state is not touched at all” — and names the experiments when one selection mixes them. Both day-state effects live inside the STATE stage, so both follow --steps.
  • The cohort load and the SRM gate stay experiment-level (the gate must still block), and so does --resync-cohort in copy mode: it rebuilds the whole cohort and therefore re-materializes day state for every eligible metric even under --metric, because narrowing that would leave the other metrics’ day state derived from the copy the resync just declared poisoned — and stale-in-place state is exactly what the M9 gap check cannot detect. Only compute narrows, and the run prints a line saying so. In the direct (no-copy) default the flag is a no-op and day state narrows with everything else, so that line is not printed — the disclosure is mode-aware, not unconditional.
  • Selector semantics follow the repo idiom: an experiment in the selection that does not declare the metric is skipped with a printed line; a value matching no comparison in any selected experiment is a loud error naming what is declared, never a silent exit-0. --steps validate with --metric is rejected (the config lint is project-wide by construction).

• M11 DASH-3 — the dashboard’s localhost server (abkit/tuning/dashboard_server.py). The transport that joins DASH-1’s job registry to DASH-2’s row shaper: a stdlib ThreadingHTTPServer on 127.0.0.1 with a one-shot token, serving the metadata-only boot page (GET /) plus one lazily-fetched row per experiment (GET /api/stats/<experiment>, ?window= bounding the sparkline) and the job-poll routes (GET /api/jobs, GET /api/job/<id>?offset=). No command reaches it yet — abk dashboard lands in DASH-6, and the client bundle in DASH-5, so until then the page renders an honest “the dashboard client bundle is not installed” note — and no statistical number moves: every verdict is DASH-2’s, i.e. readout.evaluate()’s.

  • The dashboard is a launcher, never a worker. No route acquires the pipeline lock, runs a pipeline step or computes a statistic; the only mutation any button can cause is a real abk subprocess (DASH-4). The one InternalTablesManager is serialized by a db_lock, because a DB-API connection is not thread-safe.
  • The token gates every request, GET included — unlike abk explore’s server, which serves its page to an unauthenticated GET and gates only the POSTs. Here GET / enumerates every experiment in the project and GET /api/stats/… reads the warehouse. Authorization runs before routing, so a 403 is not an oracle for which paths exist, and the token is compared as bytes (compare_digest refuses non-ASCII str) so ?token=α is answered rather than dropped.
  • The server never shuts itself down. abk explore’s Apply is terminal and stops its server; the dashboard has no terminal action and serves until Ctrl-C, at which point it terminates every job it spawned — a source-level test gate keeps the pattern from being copy-pasted back in.
  • A row that fails to build is still a 200 carrying error (DASH-2’s isolation contract), an unknown experiment a 404, and an unknown window preset a 400. A blank ?window=/?offset= is a 400 too, rather than silently reading as “absent”, and every JSON reply carries Cache-Control: no-store — these are GETs at URLs that repeat between polls, and a heuristically cached row would show a verdict from before the run the operator just launched.

• M11 DASH-2 — the dashboard’s row shaper (abkit/tuning/overview.py). One row per experiment (the maintainer’s 2026-07-27 grain decision) off the persisted _ab_results: latest verdict, effect/CI/p-value/alpha, the SRM gate, and a sparkline bucketed to at most 160 points, plus the metadata-only boot list GET / will bake. No route or command reaches it yet — it is exported from abkit.tuning and consumed only by its own tests until the dashboard server lands (DASH-3) — and no statistical number moves: every verdict is abkit.pipeline.readout.evaluate()’s, the same function abk run --report calls, and a test now pins the row’s headline cells against build_report_payload’s so the two surfaces cannot drift. Worth knowing about the shape it settled on:

  • The window preset (24h/7d/30d/90d/all) is the sparkline’s x-range and nothing else. _ab_results rows are cumulative looks from a pinned start, so filtering them by end_ts before the readout does not produce a shorter experiment — it produces a truncated stabilization history. Measured: it turned a 14-look daily WIN into INCONCLUSIVE at 24h (i.e. every daily experiment), and inverted a 6h-cadence INCONCLUSIVE into a WIN. Verdict cells are always the full series’.
  • Rows for an arm pair the config no longer declares are dropped before the readout (the report already filters this way). They do not reach evaluate’s series lookup, but they do join the read-time Benjamini-Hochberg family and tighten every threshold — nine renamed-away pairs were enough to turn the report’s WIN into the dashboard’s INCONCLUSIVE on identical rows.
  • The SRM flag is read window-independently (through the report’s own srm_summary over all persisted rows), so a preset can never silence a red assignment gate; the sparkline is filtered to the comparison’s current method_config_id, so an orphaned series left by an edited identity param cannot interleave a second generation’s points into one curve; the 20 000-look cap is display-only, like the window.
  • A verdict the readout qualified never renders as an unqualified one: under guardrail_policy: warn a WIN is kept with a mandatory loud caveat, so the row carries rationale, caveats and guardrail_regressed — each listed pair carries its own two, and the row-level flag is ORed across all of them, because a regression on the second arm must not leave a green flag on the row that lists it. The readout’s own warnings (renamed arms, orphaned series — the two states abk clean exists for) ride along in warnings, so a renamed arm no longer looks exactly like a never-run experiment.
  • The row’s per-pair list is named verdicts, matching ExperimentReadout.verdicts; comparisons stays the boot entry’s CONFIGURED list. The dashboard client merges the two payloads by experiment name, and one key holding two incompatible shapes is a trap.
  • Both surfaces carry the experiment’s timezone: every instant on them is naive UTC, and since 0.5.0 “the calendar day a look covers” is a timezone-sensitive contract.
  • project is a required argument, not an optional one: without it the readout falls back to stored-alpha CI significance and mis-scores a project-level benjamini_hochberg — loudly in its own warnings, but a glanceable row is the wrong place to discover that.
  • locked probes the run lock only (the out-of-band validate claim does not block abk run), and the probe is isolated in both directions: it runs in a finally so a failed read cannot report “unlocked” for the degraded row an operator is most likely to press Run on, and it swallows its own failure so an unreadable _ab_tasks cannot blank a verdict and an SRM chip that _ab_results could answer perfectly well.
  • An unknown window preset raises UnknownWindowPreset (a ValueError subclass), so a route can answer 400 for it and 500 for anything else — every other failure is swallowed into row["error"].
  • One bad experiment degrades to a full-shape row carrying an error string (with the fields filled before the failure kept) instead of sinking the list. A secondary or guardrail metric never appears in a row’s per-comparison verdict list — evaluate() only crosses main comparisons with treatment arms — but it does appear in the boot entry’s configured-comparison list, so the per-metric Run button DASH-4a/DASH-5 add still has something to bind to.

• M11 DASH-1 — the dashboard’s subprocess registry (abkit/tuning/jobs.py). Groundwork for abk dashboard (M11, 0.6.0): an in-memory registry that spawns the real abk CLI as a subprocess and pumps its merged output into a pollable line buffer. No route or command reaches it yet — it is exported from abkit.tuning and consumed only by its own tests until the dashboard server lands (DASH-3/DASH-4), and no statistical number moves (it reads no results and computes nothing). Deliberate deviations from the donor port it is based on: the job-kind vocabulary is abkit’s (run/unlock/clean/explore, with explore outside the one-at-a-time gate but deduped per experiment) and both spawn entry points validate against it rather than accepting any string; the dedup key is a purpose-built Job.experiment field; wait_for_line counts absolute line indices, so a job chattier than the 5000-line buffer cap before it prints the awaited line is still matched instead of failing at its timeout; a spawn racing shutdown() now refuses (JobManagerClosed) and kills and reaps the child it just created, instead of leaving a subprocess the teardown’s registry snapshot will never touch; the job status vocabulary is honest on both termination paths (a clean exit reports done even if a Stop was requested — “stopped” was otherwise reported for runs that had already succeeded — and a job the teardown kills reports stopped rather than being indistinguishable from a crash); and a poll reply carries dropped/truncated, so a log view can say that earlier output was discarded instead of inferring it from a hole in the offsets.

[0.5.0] - 2026-07-26

Changed

• BREAKING — M10 WP1: an experiment’s window is a pair of timestamps, and the config keys say so. start_date → start_ts, end_date → horizon_ts, and both now accept a bare date or a full timestamp (2024-07-01 14:30:00). There are no deprecated aliases: a config carrying an old key fails validation with a message naming the new one. Two things change with the rename, both deliberate:

  • A bare date is local midnight of THAT day, for both edges, so horizon_ts is the EXCLUSIVE right edge and the config value equals grid.horizon_ts exactly — one vocabulary, no +1 day translation anywhere. Port end_date: 2024-07-14 as horizon_ts: 2024-07-15; the rename error spells this out. An unchanged window keeps every persisted _ab_results number identical (see below).
  • A sub-day start is no longer silently floored to midnight. The old planner ran every edge through datetime.combine(..., time.min), which accepts a datetime and drops its time — so the “grid steps are sub-day but its anchors are not” contradiction was a silent truncation, not a limitation. Rejected loudly rather than coerced: an unquoted start_ts: 20240101 (pydantic would read it as a UNIX timestamp — 1970-08-23) and a UTC offset (the experiment’s timezone is the only source of truth).

• BREAKING — _ab_experiments window columns renamed and widened: start_date/end_date Date → start_ts/horizon_ts DateTime64(3), holding the resolved window in naive UTC — the same frame as _ab_results.start_ts, so a BI join lines up instead of differing by the timezone offset. A new interval_anchor String column records the knob below. ensure_tables() is create-if-not-exists-only and ensure_columns() is ADD-only, so a type change is not auto-migrated — see the combined recreate step below, which covers this table and _ab_results together.

• BREAKING — M10 WP3: _ab_results.start_date/end_date are removed. The window is a pair of instants (start_ts/end_ts) and nothing read the two derived Date columns — not the pipeline, the readout, explore, the HTML report, or the shipped Grafana/SQL BI examples. They were also degenerate at sub-day cadence — first-class since M2, and shipped in the same M2 bullet as these columns — where every look on one day collapses onto one end_date. What a BI query does instead: group and order by end_ts, which is exact at every cadence. If you genuinely want the calendar day a look covers, derive it — and keep both corrections, because dropping either moves the day: end_ts is the EXCLUSIVE edge (a daily cutoff carries the next day’s midnight) and it is stored in UTC (not the experiment’s timezone):

  -- ClickHouse
  toDate(end_ts - toIntervalMicrosecond(1), '<experiment timezone>')

  docs/reference/internal-tables.md carries the PostgreSQL and MySQL forms and worked examples; tests/pipeline/test_pipeline.py::TestTimezoneDates executes the recipe against a real Moscow run and pins that it reproduces the dropped columns to the day. No ALGORITHM_VERSION bump and no statistics-changes.md entry — schema-only, zero numeric change.

• Upgrading: recreate both internal tables, once. This release is where both breaking schema changes of the 0.2.0→0.12.0 polish track land deliberately, so there is one recreate step rather than a note per milestone. ensure_tables() only ever creates what is missing and ensure_columns() only ever ADDs, so neither change is auto-migrated. Before the first abk run on 0.5.0:

  DROP TABLE <internal_schema>._ab_experiments;   -- renamed + retyped (WP1)
  ALTER TABLE <internal_schema>._ab_results DROP COLUMN start_date;
  ALTER TABLE <internal_schema>._ab_results DROP COLUMN end_date;

  (Or drop _ab_results too and re-run — it rebuilds from the warehouse, at the cost of recomputing the series.) Do not skip the _ab_results step on ClickHouse. PostgreSQL and MySQL declare those columns DATE NOT NULL, so an insert that omits them errors and you cannot miss it (MySQL under any strict sql_mode — the default since 5.7; abkit does not set it); ClickHouse instead fills an omitted column with its type default, so a stale table keeps accepting writes and silently stamps 1970-01-01 into both columns — a dashboard still grouping by end_date would collapse every look onto one day rather than error. _ab_experiments is informational (the pipeline never reads it back for a decision); nothing is lost but the catalog row, which the next run rewrites.

• Materialized day state re-keys once. _ab_unit_state’s series identity folds in the window fields by name, so this rename orphans every existing series; the next abk run re-materializes it and abk clean sweeps the stale ids. No statistical number moves — abk verify-incremental stays green across the re-materialization.

• M10 WP4: abk explore answers knob turns while a Reload or Auto-validate is running. One coarse lock used to serialize all four POST endpoints, so an alpha slider drag queued behind a warehouse /reload or a several-hundred-split Auto /validate — the cheapest possible request waiting on the most expensive one. The lock is now scoped to what actually needs it (heavy_lock: /reload, /validate, /apply — own DB managers, the out-of-band _ab_tasks lock, the YAML archive/rewrite seam), and /recompute runs concurrently against a session whose Tier-S cache carries its own fine-grained lock. Three consequences worth knowing:

  • the accepted trade is that two recomputes racing may both run — wasted CPU and, for bootstrap knobs, one resample block of memory each; never a wrong number for the inputs each used (every input is immutable or read under the cache lock). One caveat, reproduced by the exit-gate review: a /reload installing cutoffs under a running pass makes one reply mix two renders of the same series, all points still labelled exact — the reply is not yet the unit of consistency, and making it one is a named follow-up;
  • staleness is re-checked after the compute, not just before it. A request can now be superseded while it computes; it replies 409 {stale: true} rather than overwriting the fresher answer already in the rail. The two-tab request_id machinery is otherwise untouched;
  • superseded work is now cancelled rather than queued. The old lock did double duty — it also dropped every queued request a newer knob turn had outranked, so a slider drag cost one compute. Without that, a 6-turn drag ran six full computes and the answer the user waits for took 3.40 s instead of 0.80 s, at 8.7× the CPU. The engine therefore polls the same staleness predicate between points and abandons a superseded recompute within one point (1.04 s / 1.14 CPU-s on the same drag) — the queue is gone, the cancellation is not. Polling alone cannot help a series with a single cutoff (a young experiment, a weekly cadence) and does not bound how many resample blocks exist at once, so a small admission semaphore (2 slots) fronts the compute and drops a request superseded while it waits before it computes anything.

• M10 WP5: dragging the alpha slider over a bootstrap series stops re-drawing the replicates. A Tier-S bootstrap point is a resample (n_samples draws — effectively the whole cost) followed by a percentile CI and a p-value at one alpha (microseconds). Only the second half depends on alpha, so abk explore now memoizes the first: the draw happens once per (metric, arm pair, cutoff, cache generation, method, resolved params) and every later alpha reuses it. Measured over 4 000 units × 10 000 replicates × 4 cutoffs, a six-turn drag went 6.01 s → 1.01 s — the first answer is unchanged, each turn after it 1.00 s → 0.002 s. Bounded by a value-counted budget (≈16 MB of replicates) with oldest-first eviction, and dropped for a cutoff whenever /reload re-renders it. The budget charges each slot a fixed overhead beyond its replicates (a value-only cap bounds the payload, not the number of slots), and when a knob state’s series does not fit — where an oldest-first policy degrades to no reuse at all — the reply says so instead of silently losing the speedup. The numbers do not move: the memoized outcome enters the same finalize step it would have, which is pinned per method class (from_samples == _resample + _finalize, bit for bit) and per engine path (five alphas against the same engine with the memo disabled). No ALGORITHM_VERSION bump and no statistics-changes.md entry.

Added

• M10 WP1 — interval_anchor: where the cutoff lattice sits. cadence says how far apart the cutoffs are; the new knob says where they land. Cutoffs are anchor + k·cadence, kept strictly after start_ts, with three forms: midnight (the default when the key is absent — local midnight of the day the window opens, i.e. whole calendar days), start (count from the start instant: a 14:00 start ⇒ 14:00 cutoffs), or an explicit timestamp to align to an external cycle. The anchor may precede start_ts — the forward snap is what makes that well-defined, and the first window is then legitimately partial (config-lint says so as a note, never an error). The abk init scaffold writes the default out explicitly with the alternatives in a comment. Day-or-coarser steps hold the anchor’s local wall-clock time across DST; sub-day steps stay absolute-duration.
• ExperimentConfig.grid() — the one factory composing window + cadence + anchor, mirroring m8’s build_cohort_backend contract. Nothing under abkit/ may call generate_grid directly; an AST gate (tests/core/test_grid_factory_is_the_only_entry.py) enforces it, because the alternative had already happened: the new knob reached none of the eight hand-copied call sites.
• Plugin API (M10 WP5): supports_resample_memo + the _resample/ _finalize split. A method that can separate its costly alpha-free work from its cheap alpha-dependent finish may declare the capability and implement _resample() -> ResampleOutcome; the base bootstrap class then composes from_samples for it and the explore engine memoizes the first half. Opt-in exactly like supports_vectorized (M7): the False default leaves a method fully functional through the ordinary compare_pair path — nothing special-cases a method name. The whole bootstrap family (6 classes) declares it; a roster gate keeps declaration and implementation in step.

Fixed

• Three M9 surfaces read the raw config field where they meant “the local day the window opens”, and would have broken the moment a start carried a time: IncrementalBackend compared a date against it (a TypeError on every cutoff under compute.incremental_reads) and passed it as a day key into _ab_unit_state; the STATE stage seeded its day loop from it, carrying a datetime into a Date column and into comparisons against get_last_state_day(). All three now ask the grid. The STATE stage additionally clamps the opening day’s render window to grid.start_ts, so a sub-day start cannot sum pre-experiment facts into day state.
• The CUPED pre-period stays whole-day under a sub-day start ([midnight(D − lookback), midnight(D))), instead of gaining a partial trailing day; statistics-changes.md §5 defines the lookback in whole days. Byte-identical at a midnight start, i.e. for every pre-m10 config.
• tz_midnight_utc now rejects a datetime instead of silently dropping its time-of-day — the mechanism behind the truncations above.
• Warning capture is thread-scoped (M10 WP4). warnings.catch_warnings saves and restores process-global state, so overlapping scopes on two threads interleave: a guard raised by one gets recorded against the other, an “ignore” filter set by one silences the other, and — the worst shape — exits in the wrong order leave a finished thread’s recorder installed, after which every warning in the process disappears silently. Unserializing /recompute made all three reachable in abk explore (a /recompute concurrent with Auto mode’s A/A scoring, which suppresses that very category), and abk run already fanned experiments out over a thread pool, where a guard could be persisted against the wrong experiment’s rows. abkit’s three warning scopes now route through abkit/utils/warn_scope.py: one process-global recorder installed by the outermost scope, per-thread frames, and one "always" filter per nest per category — no per-call writes to the global filter list. Warning routing only — no number moves.
• A breaking-release schema refusal reached the operator as a stack trace, not as an error line. ensure_tables() ran outside abk run’s failure handler, so the ValueError that names the drop-and-recreate remedy for a pre-0.5.0 _ab_experiments escaped uncaught: Click printed a traceback and the command’s own error line never appeared. On the one release where that message is the upgrade instruction, it was the message being buried. abk unlock had the same hole (abk validate and abk clean already echoed it).
• horizon_seconds() is the true elapsed window length, measured between the two resolved instants, where it used to be (end − start).days + 1 whole days. It now agrees with its own grid, which it used to contradict: pre-0.5.0 an October-to-November New York experiment reported horizon_days 12.04 beside a horizon_seconds of exactly 12 days. The change is exactly the UTC-offset difference between the window’s local edges — so ±1h across an ordinary DST transition, ±30 min in Australia/Lord_Howe, ±2h in Antarctica/Troll, ±24h across a date-line jump, and it also fires where there is no DST at all (a permanent zone shift, e.g. Moscow’s 2014 +4→+3). A window whose offset does not change is unaffected. Two consumers: the readout’s pre-horizon rationale line, and config-lint’s cadence gate — where a sub-day cadence sitting between the old and new lengths can flip accept↔reject, so a config that parsed on 0.4.0 may now be rejected (or vice versa) if a transition falls inside its window. No persisted column derives from it.

No ALGORITHM_VERSION bump and no statistics-changes.md entry anywhere in M10: WP1/WP3 are config/planner/schema changes, WP4 is server concurrency and warning routing, and WP5 is a structural refactor plus a cache. The numeric gate for the window rename is that an unchanged window produces unchanged numbers, and it is executable at two levels: every e2e byte-stability and cross-mode parity gate in the suite still passes, with the only edits to those tests being the renamed config keys and the ported horizon values, and tests/e2e/test_sub_day_anchors_and_explore.py compares grids, cutoffs and every derived number across 11 window shapes against a golden captured from the pre-M10 code itself — byte-identical but for the horizon_seconds() change above, which is pinned in both directions. WP4 touches no statistical code path at all; WP5 moves bootstrap code inside abkit.stats and is pinned by the untouched golden suite plus per-class bit-exactness of the split.

[0.4.0] - 2026-07-25

Added

• M9 WP6 — the milestone exit gate. The central promise is now executable over the project abk init ships: abk run twice with compute.incremental_reads: true plans nothing the second time and rewrites not one byte; a --full-refresh through the same path reproduces every number exactly; abk verify-incremental is green over the whole series with zero unverified cutoffs (a green report that verified nothing is the failure it exists to prevent) while the non-additive scaffolded metric is skipped with a reason; day state materializes for the declared-additive metric only, one row per (unit, day); cuped-t-test serves Tier E on every knob but covariate_lookback; and flipping the flag off reproduces every persisted _ab_results number — discrete columns exactly, continuous ones at the rel-1e-9 tolerance (partial-day sums associate differently than one full-window scan, so byte equality is the wrong assertion; the JSON payload columns are parsed before comparing because a CUPED θ legitimately differs in its last ULP). The red path is proven too: a late event backfilled into an already-materialized day (the documented WP4 limitation) makes abk verify-incremental exit non-zero with DIVERGED, and abk run --full-refresh --from/--to restores agreement. Against a real ClickHouse the gate additionally migrates an existing pre-M9 install (_ab_results created without the four covariate-moment columns) in place through ensure_columns’ real ALTER TABLE … ADD COLUMN and reconciles the additive path against real SQL. Reconciliation coverage now spans all three metric kinds, daily and sub-day cadence, and multi-arm experiments.

Fixed

• The state-series identity collapsed data, not just formatting (M9 WP6 round-1 review). The metric-SQL and assignment-SQL identity hashes normalized with a quoting-unaware " ".join(sql.split()), so WHERE campaign = 'Summer Sale' and … 'Summer Sale' — which select different rows — hashed identically: editing the literal did not orphan the materialized series, and the incremental read then summed two eras of one metric while recompute re-rendered the window under the new SQL. normalize_sql_for_identity now collapses whitespace everywhere except inside quoted spans (string literals and quoted identifiers), which travel byte for byte; comments are scanned as spans so an apostrophe in -- don't sum cannot open a phantom literal. Reformatting still never orphans a series. (Unreleased-only: state rows written by earlier builds of this branch simply orphan and re-materialize; abk clean sweeps them.)
• PostgreSQL installs silently skipped the additive schema migration when the schema name was not lower-case. table_exists/list_columns compare information_schema strings against the configured spelling, but schema and table names are interpolated into DDL unquoted, so PostgreSQL stores abkitinternal for internal_schema: AbkitInternal. Every lookup missed, ensure_columns never ran, and the run failed later on a column mismatch. Catalog lookups now go through a _catalog_name hook — identity by default, lower-cased on PostgreSQL; MySQL keeps the configured case deliberately (it does not fold, and on Linux database names are case-sensitive).
• A mid-run variant flip is no longer resolved silently. When the tail render’s arm disagrees with the run’s cohort snapshot for the same unit, the live arm still wins, but the incremental backend now warns once per (metric, kind) with the count and a sample unit and points at abk verify-incremental — the disagreement means the two read paths sampled the cohort at different instants, so their numbers may legitimately differ.

Added

• M9 WP5 — abk verify-incremental, abk run --cost-report, and the state GC. The reconciliation gate that makes turning compute.incremental_reads on a data-driven decision: for every already-computed cutoff of every state-eligible comparison, abk verify-incremental loads the data through BOTH backends and diffs the results field by field at rel-1e-9 (--rel-tol to tighten), reporting per-cutoff and whole-series pass/fail and exiting non-zero on any divergence. Whole-series by design — a drift that only accumulates over many days cannot hide behind a green latest cutoff. Read-only and lock-free (it persists nothing, so it never races a run), and deliberately never part of abk run. A fallback is not a pass: a cutoff where the incremental read fell back to recompute is reported as unverified, since both sides then ran the same code and agreeing proves nothing. Both this command and the driver build the reader through one build_incremental_backend factory, so the command certifies exactly the backend the pipeline runs. abk run --cost-report prints per-stage warehouse cost — wall-time, queries, rows returned, and rows scanned where the backend reports them (ClickHouse does; PostgreSQL/MySQL print n/a rather than passing rows-returned off as a scan count). The flag is --cost-report, never --profile, which keeps its one meaning on every command. abk clean gains a state sweep: _ab_unit_state series that no live (experiment, metric) pair claims — a removed comparison, a renamed metric, a deleted experiment, or a comparison that stopped being state-eligible — are reported and (with --execute) dropped. It is deliberately not narrowed by --select, since state rows are keyed by (source_table, column_set_id), not by experiment. The milestone’s executable perf gate ships with it: with N units over D days the recompute path scans N·D(D+1)/2 fact rows across the series while the incremental path scans N·D and its COMPUTE stage touches the fact table not at all — asserted with exact arithmetic, not wall-clock. The concrete criteria for flipping the default are recorded in cumulative-intervals.md §4.1. No ALGORITHM_VERSION bump; no statistical number changed.
• M9 WP4 — IncrementalBackend: the opt-in additive read path. With compute.incremental_reads: true (project-level; experiments override via their own incremental_reads; default false until the WP5 verify-incremental gate bakes), closed-form non-stratified comparisons load each cutoff from _ab_unit_state — one additive SUM(...) GROUP BY unit_id over the closed-day moments (cached per day within a run) plus, for sub-day cutoffs, a fact scan of at most the current-day tail (cumulative-intervals §6.4) — instead of re-scanning the full cumulative window. The reshaped per-unit totals feed the SAME MetricLoadResult → build_container → SufficientStats path, so no new numerical code computes the arm statistic; the CUPED pre-period covariate keeps its one cached recompute-side load. Safety posture (m9 §0.2): any state gap (series absent, trailing, or truncated) falls back to full recompute for that cutoff with a per-(metric, reason) warning — never a silent undercount; a non-finite current-day tail falls back too; bootstrap/stratified/explicit-covariate comparisons always stay on recompute (the same eligibility predicate the STATE writer uses); --full-refresh/--resync-cohort without the state step disable incremental reads for that run (re-planned results must not read day state the refresh made stale). Arm split: tail units carry the tail render’s own arm; state-only units join this run’s cohort map — the free LOAD snapshot in direct mode, the persisted _ab_exposures in copy mode — and a state unit missing from it (enrolled between LOAD and the STATE render) triggers ONE quiet re-read of the live assignment source, so a static cohort never pays a second validation query. The flag changes HOW a number is computed, never the number: flag-off behavior is untouched, and flag-on agrees with recompute at rel-1e-9 (float summation order differs between the two read paths by design — the M7 lesson; never byte-compared). Documented limitation (the m8 copy-mode precedent): an event backfilled into an already-materialized day LATER than data_lag is frozen in day state — data_lag is the declared ingestion SLA; abk run --full-refresh --from/--to re-materializes and recomputes, and WP5’s verify-incremental is the drift detector. No ALGORITHM_VERSION bump; adversarially reviewed (2 rounds, 5+3 lenses with per-finding skeptic verification).
• M9 WP3 — the STATE stage: per-(unit, day) moment materialization. A new state pipeline step (between load and compute; the abk run --steps default is now validate,plan,load,state,compute) renders every STATE-eligible metric over each not-yet-materialized closed local day and replaces the per-unit additive moments into _ab_unit_state via the long-tested replace-not-sum primitive — the write-only half of cumulative-intervals.md §4’s v1 strategy (the WP4 IncrementalBackend reader flips the read path in a later WP; nothing reads the rows yet). Eligible: closed-form (unseeded) comparisons over non-stratified sample/fraction/ratio metrics with no explicit columns.covariate role (a snapshot covariate is not additive across day renders — such metrics stay on full recompute), whose SQL does not reference ab_cov_*, and that declares state_additive: true — the author’s promise that every role column is a plain sum()/count() over the window, so per-day partials add up to the window total. The declaration is required because additivity cannot be read off SQL: a staging CTE, an outer re-aggregation, a UNION branch or an identity sum() over a renamed per-unit max() all look additive textually. abkit still refuses projections that visibly contradict the promise (a bare max(), a constant, DISTINCT, a window function, multi-branch SQL) — a veto-only filter that can take eligibility away but never grant it — and abk verify-incremental is the empirical oracle. The hazard is not hypothetical: the scaffolded example_signup_cr projects max(signed_up) and 1 AS visits, and with the old always-on eligibility its per-day rows summed to eleven trials where the window has one. Bootstrap-only metrics never pay the write. The per-day render goes through the SAME M8 build_cohort_backend factory as every other cohort reader (never a hand-rolled _ab_exposures join — both cohort modes are parity-tested). The state series identity is source_table = "{experiment}/{metric}" + column_set_id = hash(column roles + whitespace-normalized SQL body + the cohort-shaping config: assignment SQL, added_filters, unit_key, variants, timezone, start_date — plus end_date only when the assignment SQL references ab_end_*, so a routine experiment extension never orphans an end-invariant series): editing any of them orphans the stale series (swept on the next run), mirroring how method_config_id orphans results; reformatting alone never does. The series is strictly contiguous — every day <= get_last_state_day() is materialized — and every failure path preserves that by TRUNCATING the tail: --full-refresh --from/--to deletes from the first day its window touches before re-rendering through the end of the series (a crash mid-refresh leaves a self-healing prefix, never silently stale days), so a backfill can’t leave stale state; in copy mode day-close is clamped to the copy’s coverage and --resync-cohort rebuilds day state together with the copy. Non-finite moments (NULL warehouse values) truncate the series from the failing day with a loud warning — earlier days are retained, the retry costs one render per run, and reads past the last valid day stay on full-window recompute; never a silent undercount. No statistical numbers changed: the stage only writes _ab_unit_state; _ab_results math is untouched (no ALGORITHM_VERSION bump).

Changed

• M9 WP2 — CUPED is Tier E in abk explore. The three recompute gates in tuning/recompute.py that demoted the covariate family are relaxed: cuped-t-test now reconstructs each arm’s full covariate SufficientStats from the M9 WP1 persisted moments and reruns compare_pair exactly over the whole grid — for every knob (test_type, calculate_mde, power, alpha, correction) except covariate_lookback itself, which correctly stays Tier R (a different lookback is a new pre-period render; the reconstruction is refused whenever the live lookback differs from the one the row was computed with). Pre-migration rows (NULL covariate-moment columns) and degenerate covariates (NULL corr_coef) gracefully keep the old Tier S / α-inversion / baseline fallbacks — never an error. The golden round-trip gate pins the reconstruction against a from-scratch pipeline run — incl. θ — at rel-1e-9 (round-off-exact, not bit-identical: the documented Tier-E tolerance). No ALGORITHM_VERSION bump: the numbers are computed by the same from_suffstats math from losslessly-persisted moments; only where the cockpit computes them changed. Riding along (the adversarial-review round-1 fixes): the CUPED achieved-power chip now reads the control-arm correlation off the reconstructed result first (cache-free, agreeing with the exact point beside it) and falls back to the session cache for pre-migration rows; the knob surface exposes cache.covariate_moment_rows and the explore client no longer demands a warehouse reload when switching back to the configured CUPED method whose rows reconstruct (rebuilt explore.js).

Added

• M9 WP1 — persisted CUPED covariate moments + the schema-migration primitive. _ab_results gains four Nullable(Float64) columns — cov_std_1/2, corr_coef_1/2 — populated by cuped-t-test only (NULL for every other method and for pre-migration rows). Together with the existing cov_value_1/2 they complete each arm’s covariate sufficient statistics (cov_m2 = cov_std²·n, cross_c = corr_coef·√(m2·cov_m2)), the prerequisite for CUPED Tier-E reconstruction in abk explore (M9 WP2). A degenerate covariate (zero pooled variance) persists corr_coef as NULL via the existing NaN→NULL cleaning — never an error.
• ensure_columns() — the project’s first post-release schema-migration primitive. ensure_tables() now additively syncs every existing _ab_* table to the current model: it diffs the live columns (system.columns on ClickHouse, information_schema.columns on PostgreSQL/MySQL) against the declared schema and emits ALTER TABLE … ADD COLUMN for anything missing — additive-only (never drops/renames/retypes), idempotent, safe on every CLI invocation. MySQL has no ADD COLUMN IF NOT EXISTS, so its path pre-checks via the diff and swallows the duplicate-column race (errno 1060). New columns must be nullable or carry a default; the primitive refuses otherwise, loudly. Upgrading an installed project is therefore automatic: the next abk run migrates _ab_results in place, old rows read the new columns as NULL.

No ALGORITHM_VERSION bump: this is a schema/plumbing change, not a statistics change — nothing here deviates from the captured statistical baseline that statistics-changes.md’s change-control governs, and no persisted statistical number moves (the schema-not-statistics framing itself lands in statistics-changes.md at the M9 exit-gate docs sync, per the plan’s WP6). The new moments are pinned against independent np.std/np.corrcoef computations at the golden rel-1e-9 tolerance (tests/golden/test_golden_parametric.py).

[0.3.0] - 2026-07-21

M8 — assignments: no-copy default + incremental copy (the implementation record is docs/specs/m8-implementation-plan.md). No ALGORITHM_VERSION bump — zero statistical numbers changed anywhere in the milestone: this is a data-provenance/performance release (where cohort reads come from, never the math over them); the cross-mode parity gates pin _ab_results/_ab_aa_runs/the baked explore payload identical across modes.

Documentation

• M8 WP7 — the three-way docs sync (docs only, no behavior change). All three single-source bodies (docs/, .claude/rules/, the packaged init-claude assets) now describe the M8 as-built cohort semantics — a code-grounded audit found 75 stale/missing spots across 36 files: docs/reference/internal-tables.md marks _ab_exposures optional, copy-mode only and documents the append-only incremental write pattern (watermark resume, grid-anchored closed-interval batches) in place of the old delete+reinsert description; docs/guides/experiments.md gains the “Persisting the cohort: assignment.cohort_copy” section with the prominent KNOWN-LIMITATION callout (late-backfilled rows are silently missed by the watermark — stay on the no-copy default or recover with abk run --resync-cohort); the plan guide/reference carry the no-copy cost caveat (arrival-rate derivation re-executes the assignment SQL at invocation time); declarative-config.md §4/§5/§8 document ab_cohort_source as the one mode switch + the copy-mode {{ ab_added_filters }} lint; the abk init scaffold comments describe the live-join default and ship a commented-out cohort_copy: example; and the same stale “persisted once per run” claims are fixed where they lived in code — module docstrings and abk plan --help text (cli/main.py, cli/commands/plan.py, database/tables.py, compute/recompute_backend.py, planning/). Status lines across README/CLAUDE.md/rules flipped to “0.2.0 published on PyPI”; the m8 plan became the implementation record (done table, per-WP as-built notes, exit-gate log).

Changed

• M8 WP4 — the no-copy default: assignments are read DIRECTLY; the build_cohort_backend factory is the one source-mode switch. BEHAVIOR CHANGE. By default (assignment.cohort_copy.enabled: false) abk run no longer writes _ab_exposures: metric SQL joins the deduping ab_cohort_source subquery over the live assignment SQL (WP3), the SRM gate/sub-day count stream/abk plan arrival rate/report SRM chip all derive from the same validated in-memory snapshot, and read-only commands (abk plan, abk validate, abk explore, tuning RELOAD/Auto-validate, --report) see the LIVE source at invocation time instead of the last run’s frozen copy — the audit’s accepted cost/freshness tradeoff. Setting cohort_copy.enabled: true keeps today’s persisted-copy behavior end to end (full-reload write at WP4; superseded by WP5’s incremental engine — see the WP5 entry below). Every cohort reader goes through the new exposure_source.build_cohort_backend(...) factory — the binding inter-milestone contract (m8 plan §0.5(e)): copy mode stays query-free for read-only callers, direct mode renders + validates the source once (cross-variant corruption now fails loudly at every surface). The sub-day SRM bisect bucketing and the arrival-rate arithmetic moved to the shared pure abkit/core/exposure_counting.py, used by BOTH the _ab_exposures mixin and the direct-mode paths — one implementation, no drift. No ALGORITHM_VERSION bump — zero statistical numbers changed: the cross-command parity gate (tests/e2e/test_cohort_mode_parity.py) pins _ab_results/_ab_aa_runs/the baked explore payload identical across modes (+ the tuning /reload reply in tests/tuning/test_server.py), and the driver-level gates pin result rows
  • the sub-day SRM verdict stream (tests/pipeline/test_pipeline.py::TestCohortModeParity). Adversarial-review hardening in the same change: abk explore fails the house way (clean ClickException, actionable message) when the live source empties or corrupts at startup; --report on abk run/abk validate reuses the invocation’s own validated snapshot for the SRM chip (never executes the assignment source twice); and a direct-mode build_report_payload call without a manager shows honest ZERO counts instead of silently reading a stale copy-era _ab_exposures.

Added

• M8 WP6 — the copy-enabled e2e legs (tests only, no behavior change). tests/e2e/test_first_run_copy_enabled.py: the scaffolded abk init example with cohort_copy.enabled proves the CLI write path is the incremental engine end to end (first run persists through insert_exposures_incremental, a rerun is an append-only watermark resume — zero cutoffs planned, zero _ab_exposures deletes, byte-stable results), and a staggered growing-source scenario proves the true increment the single-instant scaffold seed cannot express: run 1 mid-flight persists only the closed buckets (already-visible open-bucket enrollment is withheld until it matures), run 2 appends exactly the delta (earlier buckets never re-read; the persisted rows asserted field-exact, not just as a unit set), and the two-run incremental history lands _ab_results identical to a fresh direct-mode project computed in one shot (watermark_ts — the as-of-run provenance stamp — is the one legitimately differing column). The DELETE FROM _ab_exposures statement pin in tests/database/test_sql_managers.py is re-scoped to name the resync/purge path it now serves.
• M8 WP5 — the incremental cohort copy engine + abk run --resync-cohort. With assignment.cohort_copy.enabled, abk run no longer full-reloads _ab_exposures (delete + reinsert of the whole cohort every run): the new loaders/exposure_copy.copy_exposures_incremental(...) appends only the newly matured rows — GRID-ANCHORED closed-interval buckets (grid.start_ts + k·batch_interval; the still-open bucket and rows younger than maturity_delay are withheld until they mature, and the covered boundary is a deterministic function of the clock, never of the data), watermark resume from the FINAL-deduped MAX(exposure_ts) snapped to its bucket floor (first run backfills from the experiment’s tz-snapped start; a custom update_column has no persisted cursor and re-scans from the start every run — bounding another column by the exposure watermark would silently drop rows), and batch_intervals_per_round_trip-sized round trips that re-render the assignment SQL with the batch’s bounds injected through the EXISTING {{ ab_added_filters }} hook (no new jinja surface; the hook is now REQUIRED in copy mode — config-lint and the engine both prove the reference is LIVE by rendering a sentinel filter through it, so a token parked in a comment cannot pass). The run-level whole-cohort validation (WP2) still runs every run, so cross-variant corruption fails loudly before any copy write; the persisted write path is append-only (insert_exposures_incremental, never delete_rows). abk run --resync-cohort (m8 plan §4 Q2 — a dedicated flag; --full-refresh keeps its results-window semantics) recovers a poisoned copy by deleting it and rebuilding from the experiment start THROUGH the same engine — one write path, so the rebuild honors the identical closed/matured discipline (never persists unmatured rows, never advances the watermark past routine operation) and the from-scratch re-scan is what picks up backfilled rows; a no-op in the direct (no-copy) default. KNOWN LIMITATIONS, disclosed not masked (§4 Q3, doc-only): routine runs miss a row backfilled into an already-scanned closed bucket (recover via --resync-cohort or stay on the no-copy default); on malformed multi-row-per-unit input — already loudly warned about every run — a duplicate whose rows straddle two scan windows (round trips of one run, or a prior run’s window vs a resume re-scan) resolves to the later window’s minimum instead of the full reload’s global earliest (both test-pinned). In copy mode the SRM gate/report counts deliberately measure the LIVE validated source; the persisted copy metrics join trails it by the open bucket + maturity_delay, and abk run warns when a computable cutoff exceeds the copy’s deterministic coverage (align data_lag >= maturity_delay + batch_interval). No ALGORITHM_VERSION bump — zero statistical numbers changed: the cross-mode e2e parity gate and the pipeline parity tests now exercise the incremental engine on the copy leg and stay byte-identical.
• M8 WP3 — the ab_cohort_source builtin: one cohort fragment, two source modes. The packaged assignment macro’s exposed_units() now reads its cohort through the new ab_cohort_source builtin, built in Python (query_template.build_builtins) as either the persisted _ab_exposures table (+FINAL on ClickHouse — today’s behavior, still the default, rendered byte-identically) or a deduping GROUP BY subquery wrapping the rendered assignment SQL directly (direct_source_sql — the M8 no-copy read path; MIN(exposure_ts) per (unit, variant), the same aggregation as the WP2 validation pushdown). RecomputeBackend accepts direct_source_sql/has_stratum and threads them into every render, including the CUPED pre-period covariate render; ab_exposures_table stays available for external template consumers. Call sites still construct copy-mode backends — the mode switch is centralized in WP4’s build_cohort_backend factory (docs/specs/m8-implementation-plan.md WP3, §0.5(e)). No ALGORITHM_VERSION bump — zero statistical numbers changed (direct-vs-copy load parity is test-pinned).
• M8 WP1 — the assignment.cohort_copy config block (parse-only for now). AssignmentConfig gains an opt-in cohort_copy block (enabled, update_column, batch_interval, batch_intervals_per_round_trip, maturity_delay) carrying the incremental-copy knobs for M8’s no-copy-default read-path flip (docs/specs/m8-implementation-plan.md WP1). The knobs validate at config-parse time (Interval grammar; identifier-shaped update_column when enabled) but change no behavior yet — the direct-join default and the incremental copy engine land across M8 WP2–WP5. Named cohort_copy, not copy: a pydantic field named copy shadows BaseModel.copy and warns at import (m8 plan §4 Q1, settled at WP1). No ALGORITHM_VERSION bump — zero statistical numbers changed.

[0.2.0] - 2026-07-20

M7 — validate: vectorization + iteration policy (the first polish-track release; implementation record: docs/specs/m7-implementation-plan.md). No statistical numbers changed anywhere in the milestone (no ALGORITHM_VERSION bump, goldens and both e2e matrix gates byte-identical, abkit.stats purity held): the A/A validate engine went from minutes of nested Python loops to seconds of block-streamed numpy — ~10× per validate cell, ~18× for the composed family sweep, up to ~149× on the closed-form significance kernel — behind exhaustive scalar↔vectorized parity gates, and two run-policy defaults changed (the opt-in --family-sweep and the per-cell auto-N tied to alpha; see the WP6 entries under “Changed”).

Changed

• M7 WP6 — the composed family sweep (D9) is opt-in: --family-sweep. BEHAVIOR CHANGE. abk validate no longer auto-runs the multi-metric FWER/FDR sweep whenever --metric was omitted (it silently roughly doubled every multi-metric run’s cost — REPORT item 7); pass --family-sweep (ValidateSettings.family_sweep=True) to include it. A bare multi-metric run prints a one-release migration notice naming the flag; --family-sweep combined with --metric is logged-and-skipped (one metric has no family to compose). Scripts or dashboards that relied on the __family__ sentinel row appearing in _ab_aa_runs without any flag must now pass --family-sweep. Explore’s Auto mode (POST /validate) does not opt in — the D3 calibration chip keys on per-cell rows only, so Auto runs get proportionally faster.

• M7 WP6 — default placebo iterations are tied to each cell’s effective alpha: max(2000, ⌈200/α⌉). BEHAVIOR CHANGE. The flat DEFAULT_ITERATIONS = 2000 starved tight secondary-tier alphas (at α = 0.5% a 2000-split FPR estimate carries ~±0.16pp SE against a 0.5% target — REPORT item 8); the default now resolves per cell at the cell’s effective post-correction alpha (≈4000 at the 5% main tier, ≈40000 at a 0.5% secondary tier), so a default run costs more iterations than before — cheap after the WP1–WP5/WP7 vectorization (~10× per whole cell by the WP5 perf gate, ~18× for the family sweep; individual kernels up to ~90×). -n/--iterations stays a hard override for every cell; the family sweep sizes its shared draw count at the tightest member alpha; the persisted row’s iterations column records the resolved N that actually ran. Per the m7 §4.1 maintainer call the auto-N is never hard-capped — above 100 000 the runner logs a warn-and-continue decision entry, echoed by the CLI as a yellow terminal warning, instead of silently truncating a configured alpha tier. Neither WP6 change moves a statistical number — Monte-Carlo sample size and which passes run are not method math (no ALGORITHM_VERSION bump, no statistics-changes.md entry; the exact-null FPR/power columns stay seed-deterministic at any given N, and the exit-gate e2e pins the same numbers under its explicit iterations=). This is deliberately distinct from the byte-identical WP1 hot-path fix below — do not conflate the two categories.

• M7 WP1 — scalar hot-path quick wins (hardening bucket A, A1–A8). No statistical numbers changed: the old-vs-new swap was verified bit-exact on the capture environment against a fixture frozen from the pre-change code, and the committed golden gate tests/stats/test_normal_path_golden.py re-checks the battery (extreme-z, degenerates, all six closed-form methods end-to-end) on every run — float fields at the repo’s golden relative 1e-9 (BLAS/libm builds differ across machines in the last ULP; a formula change fails by orders of magnitude), every reject/size/warning/flag field exactly. The whole stats+golden suite passes unmodified (634 passed, 1 opt-in benchmark skipped). The wins:

  • A1 — scipy.special.ndtri/ndtr replace the frozen sps.norm objects on the closed-form significance path (effects.normal_test, the z-test, sequential.se_from_ci_length), with the sf tail computed as ndtr(-z) (never 1 − ndtr(z), which drifts for extreme z). Alpha-only quantiles are now computed once per alpha (lru_cache), not per comparison. Measured: normal_test 283.8 → 1.9 µs/call (~149× on the abk validate/explore closed-form hot path).
  • A2 — statsmodels imports moved inside the power/MDE solves — import abkit.stats no longer eagerly loads statsmodels+pandas+patsy (~0.5 s cold in this env); a subprocess test pins the deferral.
  • A3 — TestResult.effect_distribution is now a LazyNormal proxy on the closed-form path — freezing the never-serialised scipy distribution is deferred to the first attribute read (delegated reads are byte-identical); the is not None truthiness contract and to_dict() behavior are pinned by a new test. (The bootstrap methods’ effect_distribution stays eager — negligible next to the resampling itself.)
  • A4 — bootstrap result-assembly dedup — per-arm stat_point values are computed once and passed into _finalize; pvalue_sign counts each side once and divides once (provably byte-identical, goldens intact).
  • A7 — shared BaseMethod._result_from_normal_test — the six closed-form methods’ copy-pasted ~20-kwarg TestResult tails now assemble in one place (field-drift risk removed), pinned field-by-field by the golden gate.
  • A8 — samples.py micro-dedups — SufficientStats.from_sample reuses the Sample’s already-computed covariate mean; from_ratio_sample computes each mean once; RatioSufficientStats gains the same m2 ≥ 0 validation SufficientStats already had.
  • A5/A6 — registry-driven contract tests + a completeness gate — the universal method contracts (dual-entry, seed-exclusion, to_dict, quarantine) are parametrized off the plugin registry so a new method is auto-swept in, and a new completeness test fails if a BaseMethod subclass is importable but silently unregistered.

Added

• M7 WP2 — the array-wise significance kernel (supports_vectorized + from_suffstats_array). Purely additive; no statistical numbers changed — every scalar path is untouched byte-for-byte. A new opt-in plugin capability (mirroring supports_sequential) lets a method expose a batch significance entry: column arrays of per-arm sufficient statistics in, a slim BatchEffectResult (effect/left_bound/right_bound/ci_length/ pvalue, one row per comparison) out, computed via numpy broadcasting with the alpha-only quantiles evaluated once. Exactly five methods opt in — t-test, z-test, cuped-t-test, paired-t-test, ratio-delta (pinned by a capability-roster test); bootstrap stays scalar-only, exercising the fallback the M7 WP4 engine will rely on. The sequential module gains the same siblings (se_from_ci_length_array, sequentialize_array). Row-level parity with the scalar from_suffstats is pinned by tests/stats/test_vectorized_parity.py + tests/stats/sequential/test_sequential_arrays.py across every guard branch (H5 denominators, degenerate variances, pooled proportions 0/1, extreme-z tails, heterogeneous 1e-4…1e4 magnitude mixes) — bit-exact for all five methods and both test types, by construction: power terms route through the same C-library pow the scalar ** uses (_libm_pow), because numpy’s own integer-exponent power is 1 ULP off libm and the cancelling delta-method variance sum amplifies that far past rel-1e-9 (found by adversarial review round 1, pinned by a cancellation regression test); only the sequential siblings’ log/exp keep the golden rel-1e-9 bound across libm/numpy builds (same-sign sums, no cancellation to amplify — measured byte-identical on the capture environment). Degenerate batch rows yield NaN (“gaps, never zeros”) instead of per-row warnings/exceptions — the one documented contract divergence (ddof-1 n < 2 rows NaN-poison where the scalar raises) has its own regression tests; mismatched per-arm row counts, 0-d/scalar columns and 2-D columns all fail loudly (SampleValidationError), never broadcast or malform; and the kernels mirror the scalar constructors’ int(n) truncation (np.trunc) so a fractional-n row cannot silently diverge (both from adversarial review round 2). Measured on the M7 reference shape (200k rows ≈ 2000 iterations × 100 cutoffs): ~120 ms batched vs ~1.4 s scalar-looped (~12×) for the relative t-test kernel, ~16 ms (~90×) for pow-free branches — the libm-pow routing deliberately trades a slice of the speedup for bit parity.

• M7 WP3 — the block-streamed vectorized placebo-resampling engine (abkit/validate/vector_resample.py). Purely additive; no statistical numbers changed — nothing consumes it yet (the M7 WP4 score_cell rewrite will) and the scalar resample.py path is untouched. placebo_mask_block produces a block of placebo masks where row i IS placebo_mask(..., derive_seed(*seed_parts, block_start + i)) — the permutation layer stays bit-identical to the scalar loop by construction. build_arm_batch then collapses a whole block’s per-arm sufficient statistics at one cutoff into one GEMM per arm (pooled-shifted one-pass co-moment columns; sample/CUPED/fraction/ratio kinds; columns keyed to feed WP2’s from_suffstats_array directly), with per-(iteration, cutoff) degenerate gap masks (MIN_ARM_UNITS, zero-trial fraction arms) whose rows are NaN-poisoned — gaps, never zeros. Blocking mirrors the bootstrap engine’s BLOCK_QUANTUM/256 MiB-cap arithmetic (block_rows/iter_blocks) with one documented divergence: mask rows are seed-independent, so a block may shrink below one quantum (down to one row), keeping the cap honored for the block-scaled working set at any population size (the per-cutoff k ≤ 5 value columns are a cap-independent 8·k·n_units fixed overhead, asserted separately by the memory tests). The block-size contract is stated honestly from measurement: masks/counts/degenerate flags are byte-identical under ANY partition; float columns are byte-reproducible under a fixed partition and ULP-class (gated rtol 1e-12) across different partitions — it was measured that no float reduction (BLAS or numpy’s own sum(axis=1)) keeps the same row bit-stable across buffers with different row counts, so the bootstrap engine’s “any cap, same bytes” promise is provably out of reach here and rel-1e-9 scalar parity (matmul-vs-.sum() reduction order) is the numeric gate, pinned per row against the scalar build_arm across all four input kinds, growing unit sets, extreme shares, offset (1e8) data and mixed degenerate blocks in tests/validate/test_vector_resample.py. Resolves m7 open question §4.4: cross-cutoff prefix sums are permanently inapplicable (the full-window re-render makes per-unit values non-appendable — refunds shrink sum(...) metrics, max(...)-shaped metrics are not additive at all), recorded in the module docstring. inject.py gains the batch mirror of the injected pass (inject_multiplicative_columns/injection_clamped_columns, bit-exact vs the scalar injection algebra per row) so the WP4 scorer’s power/coverage pass has its seam ready. Adversarial review round 1 (2 reviewers, 2 major

  • 6 minor, all fixed): the rel-1e-9 parity band is scoped and pinned at its real float64-conditioning boundary (|value|/σ ≲ 1e10; the scalar path’s rounded-arm-mean m2 inflation is what diverges past it, measured ~5e-9 at 1e12), the CUPED/ratio memory profile is asserted with the capped and fixed parts split, overflow-scale data cannot leak RuntimeWarnings or a non-degenerate NaN row unnoticed, and malformed count > nobs fraction data is pinned to flow to a NaN gap (the scalar path crashes at construction — the one documented build-level divergence). Round 2 (fresh reviewer, 2 major + 3 minor, all fixed): the hoist API rejects a mismatched (prepared, cut) pair (an equal-sized-cutoffs mixup would otherwise score silently-wrong numbers), the batch injection’s deliberate NaN-m2 divergence from the scalar max(0.0, nan) == 0.0 quirk is documented + regression-pinned (the batch keeps the gap poison), and panel arrays are float64-normalized like the scalar constructors (a float32 panel would otherwise break rel-1e-9 parity at ordinary offsets). Measured at the reference shape (2000 iterations × 100 cutoffs, CUPED, n=2000): ~1.4 s for the full suffstats aggregation vs ~20 s for the equivalent scalar build_arm loop (~15×), before the WP4 significance-side vectorization lands on top.

• M7 WP4 — score_cell now runs the vectorized engine for supports_vectorized methods, with the original scalar loop preserved verbatim as the fallback. No statistical numbers changed — the e2e validate/sequential matrix gates pass unmodified against the new default path, and per-kind smoke-parity tests pin the two engines against each other (integer tallies + count-ratio columns exactly equal; continuous means at rel-1e-9, the WP3 reduction-order budget). The vectorized engine consumes the WP2+WP3 primitives end to end: per block of iterations (iter_blocks over block_rows(n_units) — blocking is a pure function of (iterations, n_units) + module constants, so persisted A/A numbers stay byte-reproducible run-to-run under a fixed BLAS configuration, D13; a different BLAS build/thread count re-rounds the GEMM’s continuous columns at ~1e-15 rel, counts unaffected — the same scope the Poisson bootstrap engine ships with), each cutoff is one build_arm_batch GEMM + one from_suffstats_array call; the peeking first-crossing state streams per row in O(block) memory (never block × cutoffs), explicitly guarding the argmax-on-all-False footgun (regression-tested: a grid where no null split ever crosses reports peeking_fpr == 0.0, not 1.0); the always-valid D8 twin rides the same per-look (effect, SE) arrays through sequentialize_array under the unchanged scalar τ² anchor; the injected power/coverage pass reuses the held horizon batch through inject_multiplicative_columns (same one-shot saturation warning); and the reporting-only achieved-MDE loop stays scalar but strictly iterations-shaped (never iterations × cutoffs — the §WP4 risk-list regression). Methods without a batch kernel (supports_vectorized=False: the bootstrap family, any custom plugin) dispatch to _score_cell_scalar — a pure code move of the previous loop, pinned identical via a stub-method test; a plugin that declares supports_vectorized=True without a working batch kernel fails its own cell loudly (ValidateError), never aborting the whole matrix. The engine’s live allocations share ONE 256 MiB ceiling: hoisting the per-cutoff GEMM operands (prepare_cutoff) gets only what the block working set leaves of the cap (past the leftover, blocks re-prepare per cutoff — bounded memory, identical bits either way, equality-pinned by forced-non-hoist tests over single- AND multi-block partitions; the BLAS-scope boundary itself stays documented-not-CI-enforced, same as the donor bootstrap engine’s gate). Also fixed while under review (pre-existing, shared by both engines, no scorable number moved): an exactly-zero pooled ratio denominator crashed the whole matrix with an uncaught ZeroDivisionError out of _point_estimate instead of falling back to the per-iteration value_1 truth anchor as documented — now guarded like ratio_delta._arm_linearisation, regression-tested on both engines. Measured at the reference shape (2000 iterations × 100 cutoffs, CUPED, n=2000, with injection): ~2.5 s vectorized vs ~25 s scalar (~10×); the dedicated parity + perf gates land in WP5.

• M7 WP5 — the exhaustive parity gate + the executable perf gate closing the milestone’s engine chain. Zero statistical numbers changed — every existing golden reference (the validate-matrix and sequential-matrix e2e, the sequential/family parity suites) passes unmodified. tests/validate/test_vector_parity.py runs the preserved scalar engine against the vectorized default across ≥50 seeds × 8 shapes (sample / CUPED / absolute test_type / fraction / ratio, plus three adversarial stress shapes: a gap-heavy sparse shape where some splits degenerate and some don’t, CUPED at the MIN_ARM_UNITS floor, and a saturating-clamp fraction injection), ± injection, asserting exact equality on every count/decision/curve/warning field — including achieved_mde, see below — and rel-1e-9 on continuous means, with a trip-wire pinning every CellScore field to a parity class so a future field cannot dodge the gate, a multi-block streaming test (quantum 1/7/128 — cross-block accumulators and the ragged final block), and scanned-and-pinned deterministic seeds for two rare-but-reachable states the battery alone would undersample: the τ²-unanchorable cell (“always-valid column skipped”) and the no-valid-horizon cell (the milestone exit run passed at ABKIT_PARITY_SEEDS=200 — 1 600 engine-pair runs). The §0.3(3) mandatory near-boundary stress manufactures the dangerous input outright: brentq solves the injected δ that puts a split’s CI bound exactly on the significance boundary — at δ·(1±1e-9) parity stays exact (the bound sits five orders of magnitude above the engines’ ~1e-16 ULP divergence), and AT the solved root (|bound| ≲ 1e-15, inside ULP ambiguity) the measured, now-pinned honest limit is a single flipped decision confined to the stressed iteration’s power column — both roundings correct, real cells at generic positions unaffected (the e2e matrices are byte-identical); two scanned seeds whose null split is already significant keep the negative-root bracket branch live rather than dead defensive code. Fixed under adversarial review round 1 (the one engine change in this WP): the vectorized MDE seam now rebuilds each valid row’s control arm through the scalar build_arm on the row’s own mask — bit-identical _analytic_mde inputs by construction — instead of reading the GEMM columns, which diverged at a knife-edge (a 2-unit CUPED arm has metric↔covariate corr ≡ ±1; whichever engine’s reduction rounds exactly onto ±1 reports achieved_mde=None while the other reports 0.0, and the persisted column feeds the Recommended-row tie-break). achieved_mde is therefore asserted exact, not rel-1e-9, and the GEMM-column _control_stats_from_row helper (with its documented fractional-count clamp caveat) is gone. tests/validate/test_vector_perf.py asserts the REPORT reference cell (2 methods × 2000 iterations × 100 grid cutoffs × 1000 units, null + injected + sequential columns) under a generous CI-safe 10 s bound sized against the coverage-instrumented run — the CI Test job traces --cov=abkit, which roughly doubles the cell: dev-measured ~1.3–1.7 s bare / ~2.2–2.5 s under coverage (vs ~25 s scalar, same methodology), with the scalar engine monkeypatched to fail loudly if dispatch ever regresses. Adversarial review round 2 (fresh reviewer) additionally: caught and fixed the MDE-seam rebuild crashing a whole fraction cell on corrupt over-counted input (per-unit successes > trials) where the batch main pass scores it — the row’s MDE is now skipped, reporting-only stays reporting-only, with the residual scalar-fails/batch-scores divergence on such corrupt input documented in the spec §9 and pinned by a dedicated regression test; re-examined the WP2 kernel-tolerance question (§4.3) — already closed at exact (assert_array_equal, nothing to tighten); and measured the battery’s continuous-field deviations at ≤ ~2e-14 rel, keeping the rel-1e-9 assertion as the principled conditioning-band bound, not defensive slop. The spec gains the matching contract section (aa-false-positive-matrix.md §9 “Implementation note”) so the invariant lives in the spec, not only in code comments.

• M7 WP7 (stretch) — the composed family sweep (D9) runs its own block-streamed vectorized engine. Zero statistical numbers changed — the scalar loop is preserved verbatim as the fallback, and every existing family/e2e reference passes unmodified. family.py has its OWN hot loop (the §0.3(1) plan-review correction — the WP4 score_cell rewrite never touched it); sweep_family now dispatches exactly like score_cell: when EVERY member’s method opts in via supports_vectorized, blocks of shared union masks come from placebo_mask_block (row i IS the scalar union mask — bit-identical by construction), each member’s per-look arms build through one build_arm_batch GEMM + one from_suffstats_array call (the _Peek accumulator gets a block-wise mirror, _PeekBlock, with the same first-crossing/latest/min-p semantics), and the per-iteration COMPOSITION (composed_significance) stays the unchanged scalar helper applied in iteration order — so every FamilyScore column (count ratios, exact-fraction FDP sums, warnings incl. the one-shot clamp warning’s lexicographic (iteration, member) pick) is expected EXACT, not rel-1e-9. A family with any non-opted-in member (bootstrap, custom plugins) runs _sweep_family_scalar — a pure code move of the previous loop; a lying supports_vectorized=True member fails the sweep loudly as a ValidateError. The new gate tests/validate/test_family_vector_parity.py asserts exact equality on every FamilyScore field across ≥50 seeds × 5 family shapes (overlapping/disjoint cohorts, ratio+CUPED members, a persistent-gap 3-unit member with its ‘scored in 0 iterations’ disclosure, a saturating-clamp planted fraction member, bonferroni AND benjamini_hochberg, ± injection, ± sequential; exit run at ABKIT_PARITY_SEEDS=200 — 1 000 engine-pair runs), plus multi-block (quantum 1/7, every shape) and dispatch/fallback/lying-flag contracts. Measured on a reference family (3 members × 2000 iterations, sequential + injection): ~0.11 s vectorized vs ~1.96 s scalar (~18×), with byte-identical output. Two adversarial review rounds; fixed under round 1: the batch engine gained the scalar _member_marginal’s except Exception net around the batch kernels (a structural kernel raise — e.g. a programmatically-built CUPED member on a covariate-less panel — now gaps that member exactly like the scalar engine instead of crashing the sweep; NotImplementedError re-raises so the lying-flag contract stays loud), and the corrupt-input divergence class (fraction count > nobs: the scalar engine crashes the sweep, the batch engine scores it) is now spec-documented for the family surface and pinned by a dedicated regression test — the batch-flag hardening remains the same named follow-up as score_cell’s. Round 2 scoped the net honestly: under sequential=True (the runner’s only mode) a member whose τ² ANCHOR itself raises structurally crashes BOTH engines identically inside the shared, unguarded _cell_tau2 — pre-existing, symmetric, runner-isolated; the engine net applies where the anchor didn’t already fail (a degenerate-anchor member — pinned by a dedicated walk-raise parity test — or sequential=False), and guarding _cell_tau2 itself is a named follow-up since it would change both engines’ behavior at once.

• The polish track M7–M17 (0.2.0 … 0.12.0) planned into the repo — docs only, no behavior change, no statistical numbers touched: the approved (2026-07-18) track section in ROADMAP.md (milestone map + versioning + the coverage map over the data-flow audit’s 15 items and the entire post-baseline hardening backlog + the cross-cutting discipline, incl. the M7–M12 “numbers do not move” parity gates and the M8→M9 build_cohort_backend blocker contract), six as-designed contracts docs/specs/m7…m12-implementation-plan.md (from the code-verified WP breakdowns), and the verified pain audit committed as docs/research/2026-07-data-flow-audit/REPORT.md (four verification corrections recorded in its banner). M13–M17 stay contours — each opens with its own design session.

Fixed

• M7 WP0 — multi-arm Review mode dropped every verdict after the first (UI-only; no statistical number touched). abk explore’s Review mode rendered a metric’s verdict via .find(...) over payload.verdicts, which holds one block per (metric × control-vs-treatment pair) — so in a 3+-arm experiment only the first pair’s verdict showed and the rest were silently dropped (the underlying per-pair verdicts were always computed and persisted correctly). Review mode now renders one labeled verdict line per declared pair (same abk-review-verdict/abk-verdict-<word> marker classes; 2-arm rendering unchanged), with jsdom regression tests for both the 2-arm and 3-arm cases and the rebuilt committed explore.js. A new honest “Known multi-arm limitations” section in docs/guides/experiments.md names what is not k-arm-aware today: no experiment-level winner rollup (M14), abk plan sizes off the first declared pair only, and abk validate’s placebo split is two-arm (control share vs the rest pooled).

[0.1.2] - 2026-07-09

Explore-cockpit / CLI DX + reporting polish. No statistical numbers changed (no ALGORITHM_VERSION bump, goldens intact, abkit.stats purity held) — every change below is transport, logging, or presentation.

Added

• Brand logo in every generated surface — the “Diverge” mark + abkit wordmark now render in the abk run --report and abk explore headers (inline SVG, shared web/src/shared/logo.ts), not just the browser-tab favicon.
• Progress heartbeats on long-running compute so a multi-minute run is no longer a silent freeze: abk run prints a throttled LOOK i/N per computed look; abk validate streams scoring cell i/N per cell; Auto mode echoes the same to the explore terminal.

Changed

• Explore Auto button is honest when unavailable — on a --no-serve / saved-report page it now carries an actionable tooltip (“Auto needs a live server — open the printed localhost (127.0.0.1) URL, not a saved report”) + aria-disabled, and it shows a busy state while a validation is in flight.
• Removed the redundant explore “CUPED on/off” checkbox — it was a pure UI alias of the method picker (it only strip/prepended cuped- and switched the method). CUPED is now chosen directly in the method picker as the cuped-t-test variant; no functional loss, one fewer duplicate control.

Fixed

• No more BrokenPipeError tracebacks from abk explore — the stale-drop discipline (a knob turn aborting a superseded request) left the server writing to a closed socket; the transport helpers now suppress BrokenPipeError/ConnectionResetError (the latest request still computes and replies).
• No more per-split AbkitStatsWarning flood during abk validate / Auto mode — the A/A sweep re-invokes the same method over hundreds of placebo splits × looks, so the CUPED low-correlation / ratio-zero legacy guards spammed stderr thousands of times. They are now suppressed inside the scoring loop only (the single real abk run still surfaces them; also carried in TestResult.warnings). Non-numeric.

[0.1.1] - 2026-07-08

Documentation + AI-assistant-context accuracy patch (no code, no statistical numbers — a post-0.1.0 fact-check of every published doc page and every packaged abk init-claude asset against the shipped CLI/config/method surface). 15 verified findings fixed; each was independently re-verified against the code.

Fixed

• Packaged abk init-claude assets now match the shipped API (these ship in the wheel, so the fix ships in 0.1.1):
  • Metric-SQL docs no longer reference a non-existent {{ data_schema }} template built-in (rules/metrics.md, skills/abk-new-metric) — {{ data_database }} is the single data-location built-in on every dialect (on Postgres it resolves to the profile’s data_schema value); under StrictUndefined the old note would have made Postgres metric SQL fail to render.
  • skills/abk-explore no longer lists the multiple-comparison correction among the identity params that orphan an _ab_results series — the correction (like alpha) is experiment-level and never enters method_config_id; changing it re-arms the calibration chip but does not orphan results.
  • skills/abk-validate quotes the actual Recommended-row rationale (“highest power among methods with FPR within budget”, tiebreak: tightest achieved MDE), not a CI-width criterion the selector never uses.
  • rules/explore.md marks Segment mode as a deferred placeholder (not an available 0.1.0 cockpit mode).
  • rules/project.md describes env-var interpolation correctly (an unresolved placeholder is kept verbatim, not raised as an error; only channel secrets are actively rejected); skills/abk-setup-project scopes the “no database: key” note to ClickHouse (it is optional for MySQL, required for Postgres).
• abk test-report + notification_channels are now covered for the AI assistant and in the docs (they shipped in 0.1.0 but were undocumented): a test-report command entry + ## abk test-report section (rules/cli.md, docs/reference/cli.md), a notification_channels block in rules/project.md, a routing entry in CLAUDE.section.md, and the command added to the enumerated command surface in docs/README.md / docs/getting-started/installation.md.
• abk plan runtime/ASN documented as shipped: docs/reference/cli.md no longer says runtime/ASN “is not part of this command”, and the --arrival-rate flag is added to the abk plan option/flag tables in docs/reference/cli.md, rules/plan.md, and skills/abk-plan.
• The CI install-smoke version gate now compares abk --version against abkit.__version__ dynamically (no longer hard-pinned to a literal), and the abk --version sample in installation.md tracks the release.

No abkit.stats change; no ALGORITHM_VERSION moved; goldens untouched at rel-1e-9.

[0.1.0] - 2026-07-08

The first tagged public release — milestones M1–M6. The pure numpy statistical core, the declarative YAML+SQL config / DB layer / recompute pipeline, the explore cockpit + self-contained reports, abk validate (the A/A false-positive matrix), opt-in sequential analysis + abk plan, and the M6 DX layer (abk init-claude, abk test-report, the docs site, Prefect scaffolding). No statistical numbers changed across M2–M6 (goldens intact at rel-1e-9; no ALGORITHM_VERSION moved).

Added

• M6 WP10 — the M6 exit gate: release-readiness e2e, ≥2 adversarial review rounds, and the coordinated milestone-header sync. New tests/e2e/test_release_readiness.py proves the whole first-release journey offline and byte-reproducibly — abk --version reports the real (non-placeholder) release, abk init → abk run --select lands a real verdict-bearing _ab_results row, abk run --report bakes a self-contained zero-network readout, abk init-claude materializes the managed CLAUDE.md block + the 9 rules + 7 skills (idempotently), and the committed renderer bundles are self-contained (offline, no external host). The wheel-packaging DoD — a built wheel shipping both bundles + every abkit/cli/assets/claude/** asset and resolving in a clean venv — is owned authoritatively by the CI lint wheel-namelist gate + the install-smoke job (across the Python matrix), which this fully-offline e2e complements deterministically. The as-built docs are flipped to one story now that M6 is shipped: the status headers in CLAUDE.md, .claude/rules/architecture.md (including __version__ 0.0.1.dev0 → 0.1.0 in the banner), .claude/rules/contributing.md (the release checklist names the three single-source bodies + the packaging DoD), and ROADMAP.md (M6 ✅ SHIPPED; the sole alpha_spending/group-sequential deferral pointed at the future with no version promise). The exit-gate review (≥2 full rounds, refute-by-default, a second independent verifier per finding) is recorded in docs/specs/m6-implementation-plan.md §5. No abkit.stats change; no ALGORITHM_VERSION moved; goldens untouched at rel-1e-9; abkit.stats purity intact.
• M6 WP9 — release engineering (prep only; the tagged publish is a separate, maintainer-gated step). Bumped __version__ 0.0.1.dev0 → 0.1.0 (the first real version must exceed the reserved placeholder or PyPI rejects the upload) and the packaging classifier to Development Status :: 3 - Alpha. Cut this [Unreleased] history into the dated [0.1.0] section. Hardened the release DoD with three new gates: (1) the CI wheel-namelist gate now also asserts the wheel ships every abkit/cli/assets/claude/** file (the 17 abk init-claude assets — the highest-risk packaging miss, since a bad wheel can’t be re-uploaded under the same version), alongside the existing report.js/explore.js bundle check; (2) a new pip install DoD smoke job installs the built wheel (not -e .) into a clean venv on Python 3.10/3.11/3.12 and proves abk --version reports 0.1.0 and abk init-claude -d <tmp> materializes the managed CLAUDE.md block + the 9 rules
  • the 7 skills from importlib.resources at install time; (3) the WP8-promised cross-body docs single-source drift gate (tests/docs/test_docs_single_source.py) asserts every packaged operator rule in abkit/cli/assets/claude/rules/ has a corresponding published docs/ page — so a new rule cannot ship without a user-doc home. The mypy abkit strict gate stays continue-on-error (aspirational) for 0.1.0: the ~124 tracked strict-mode errors live in numeric hot paths (recompute.py/readout.py) and clearing them is a post-0.1.0 quality pass, not a release blocker (§7 Q9 decision). No abkit.stats change; no ALGORITHM_VERSION moved; goldens untouched.

Changed

• M6 WP8 — named-deferrals hygiene: the shipped code, packaged assistant assets, docs, and specs now tell one true story about what is and isn’t implemented (no behavior change). Every “planned for M6 / deferred to M6 / M6 follow-up” string that pointed at a feature which actually shipped in M6 is flipped to shipped — abk plan runtime/ASN (WP-A) and the A/A sequential × composed sweep (WP-B) across abkit/planning/__init__.py, the packaged abk init-claude assets (rules/plan.md, rules/validate.md, skills/abk-plan), docs/guides/plan.md, docs/specs/cli-and-dx.md, docs/specs/aa-false-positive-matrix.md, ROADMAP.md, .claude/rules/architecture.md, and this repo’s CLAUDE.md. The one genuinely unshipped item — alpha_spending / group-sequential — is re-pointed everywhere from “M6” to a future item with no version promise (the user-facing config error already refuses it cleanly). Three spec-reconciliations bring the as-built into line with the prose: the single-source docs model is documented as three separately-authored bodies kept consistent by human review (not machine cross-generation; a CI drift gate lands in WP9), the BI deliverable as tool-agnostic reference SQL + one Grafana dashboard (not a per-tool importable dashboard for each of the four), and project-level error notification as a post-M6 item (with abk test-report the shipped connectivity smoke). No abkit.stats change; no ALGORITHM_VERSION moved; goldens untouched.
• M6 WP7b — the self-contained abk run --report + abk explore surfaces now render in the finalized Iris brand. The one brand-token layer (web/src/shared/chart.ts TOKEN_FALLBACKS) was frozen from placeholder values to the real Iris palette (docs/design/brand-tokens.md): a warm-paper light page (#f5f1e8/#1b1916), a dark chart panel, the iris-family series slots (#c9a6f0/#8e76e0), and the five verdict/status tokens (WIN #1e9e6a … SRM #b23a6b). Both page shells (abkit/reporting/html_report.py, abkit/tuning/html.py) now carry the abkit “Diverge” brand mark as their favicon (iris tile + paper strokes) and open on warm paper — still fully self-contained (no network, no webfonts; system-font fallback). report.js + explore.js rebuilt. The CI token-sync gate is promoted to a hard value check for the theme-independent tokens now that the palette is frozen (per-theme surface tokens stay value-skipped). WCAG-AA contrast recorded for the reskinned text surfaces (body/muted/accent on paper and the dark chart panel all pass AA). No abkit.stats change; no ALGORITHM_VERSION moved.

Added

• M6 WP7b — an interactive stabilization demo on the landing page. The marketing hero now mounts a live #abk-demo widget (dial true effect / noise / traffic, watch the cumulative effect + CI converge past the decision horizon and the WIN/LOSE/FLAT/INCONCLUSIVE verdict get called). Its compute core (website/src/scripts/demo/stats.ts) is a TypeScript re-derivation of abkit.stats, golden-parity-gated in CI (check-demo-parity.mjs, rel-1e-6), painted through the shared framework-free renderer core (web/src/shared/chart.ts). Everything is client-side and dependency-free. The Notification channels guide (docs/guides/notification-channels.md, WP5) is now wired into the docs site (sidebar + sync-docs PAGES).

Fixed

• abk init prod-profile env placeholders were double-wrapped (latent scaffold bug). The prod: profiles in the three abk init templates used quadruple-brace {{{{ env_var('ABKIT_*') }}}} placeholders, but profiles.yml is written raw (never .format()-ed, unlike abkit_project.yml), so a set env var resolved to {{value}} (wrapped in stray braces) instead of value. Corrected to standard double-brace {{ env_var('ABKIT_*') }}. Latent because abk init’s scaffold self-check runs with the vars unset (a preserved placeholder validates fine either way). Surfaced while adding the WP5 notification_channels: seed block.
• M6 WP1 — tooling debt root-caused + partly cleared (no behavior change). The long-standing “mypy fails on clean HEAD” was not a numpy issue: a stray comment # type: (required, optional) in abkit/config/metric_config.py was parsed by mypy as a PEP-484 type comment (Invalid syntax), making it bail before type-checking anything. Reworded the comment; raised [tool.mypy] python_version to 3.12 (clears the secondary numpy 2.5 PEP-695 stub error); added yaml.* to ignore_missing_imports. mypy abkit now runs to completion (it reports ~124 real strict-mode errors, still continue-on-error — tracked debt, they live in numeric hot paths). Pinned [dev] black==24.4.2 and mypy==1.10.0 to the pre-commit revs so CI and local pre-commit cannot diverge (zero reformat churn). No runtime code changed; goldens untouched; no ALGORITHM_VERSION moved.

Added

• M6 WP5 — abk test-report + a minimal notification-channel layer (abkit/notify/). A new command sends a synthetic mock readout through every channel in a new profiles.yml notification_channels: block and prints a per-channel ✓/✗ — a connectivity + formatting smoke test (no lock, no warehouse read, no statistics). Five channels ported and reshaped from detectkit’s alerting channels — Slack, Mattermost, a generic webhook, Telegram, email — keeping the transport/envelope but dropping every alerting semantic (no severity / recovery / no-data / detector / quorum / consecutive machinery; abkit has no alerting). The message is experiment-primary: a verdict (WIN/LOSE/FLAT/INCONCLUSIVE, SRM-gate overriding), effect + CI, p-value, the effective post-correction alpha, and the weekly-cycle representativeness, colored by the five brand verdict tokens. Secrets come only from env interpolation (${VAR} / {{ env_var(…) }}) and an unresolved placeholder is refused with a clear error. notification_channels: is a new typed field on ProfilesConfig (NotificationChannelConfig, additive — existing profiles.yml files are unaffected); a commented example ships in the abk init seed. The command exits non-zero on any send failure / misconfiguration (the CLI-is-the- automation-unit convention). Pure Python, no new dependency (requests was already a dependency); abkit.stats untouched, no ALGORITHM_VERSION moved. Covered by tests/notify/test_channels.py + tests/cli/test_test_report_command.py.
• M6 WP-A — abk plan gains runtime + ASN (read-only, no stats-core change). Given a unit-arrival rate — derived read-only from _ab_exposures (new get_arrival_rate: distinct units per observed day, whole-cohort window, split to the control arm) or supplied via the new --arrival-rate <units/day> flag — each sizable comparison now also reports runtime (days-to-required-N = required_n / rate + the planned horizon) and, for a sequential.enabled sequential-eligible design, the always-valid ASN (average sample number): the expected control-arm N at which the confidence sequence first excludes zero under the true effect (H1) and the null (H0). ASN is a deterministic fixed-seed Monte-Carlo estimate over the canonical information-time process, crossing the exact shipped CS boundary (abkit.stats.sequential) — it adds no estimator and moves no ALGORITHM_VERSION; abkit.stats stays pure and byte-identical. No arrival data ⇒ runtime is SKIPPED with a reason (never guessed); a fixed-horizon/resampling design ⇒ ASN n/a. Honest framing: the always-valid design’s sample requirement (N to reach a given power) is larger than the fixed required-N (the Robbins mixture CI is wider by design — the price of unlimited peeking), so the CS never lets you design for fewer units at the same power. The reported ASN is a different quantity — the expected stopping N, horizon-capped — guaranteed only against the horizon (ASN_H1 ≪ horizon-N; ASN_H0 ≈ horizon-N; monotone in effect); vs required-N it is regime-dependent (can dip below in the underpowered/horizon-capped case, which the CLI line flags). The Monte-Carlo estimate is cross-validated against an independent scalar first-passage simulation in the tests.
• M6 WP-B — the A/A composed sweep gains its always-valid (peeking) twin (no behavior change to the shipped single-look family). abk validate’s composed multi-metric family sweep now mirrors the per-cell D8 trio at the family level: alongside the unchanged single-look fwer/fdr, it composes a matched peeking pair over the same shared placebo assignments — fwer_peeking/fdr_peeking (each member’s fixed CI peeked across every look: the composed optional-stopping hazard, inflated) and fwer_sequential/fdr_sequential (the always-valid twin via the identical D8 estimator: controlled, ≈ the single-look rate). Gated on a sequential-eligible family (≥1 member has a frozen τ²); an ineligible member (bootstrap — unscorable from suffstats) is a full gap in every family, disclosed by the existing “scored in 0 iterations” warning. The numbers persist additively in the _ab_aa_runs sentinel row’s details.family (no new schema column); the report’s composed band renders a “peeking → always-valid” recovery stat. This is a validate-layer MODE transform reusing the M5 D8 estimator verbatim — no ALGORITHM_VERSION bump, no stats-core number changed, the single-look family byte-stable (sequential defaults off). Closes the last non-alpha_spending A/A deferral (aa-false-positive-matrix.md §8.1). Pinned by the D8×D9 headline tests in tests/validate/test_family_sweep.py + the sequential-matrix e2e.
• M6 WP7a — the abkit docs + marketing website (website/, Astro + Starlight). A single-source site built from the docs/ body via sync-docs.mjs, on the real Iris brand (brand.css, light+dark, name-locked to the bundles’ --abk-* token layer), with the “Diverge” logo/favicon, a landing page, and an interactive stabilization-chart demo whose JS compute path is golden-pinned to abkit.stats (hard demo-parity CI gate). web/ and website/ are now an npm workspace (single root lockfile); a Docker-free website CI job runs sync + astro check + build + demo-parity. The live deploy (Dockerfile → GHCR → abkit.pipelab.dev) is a separate gated step. Renderer bundles unchanged.
• M6 — user-facing docs body + brand source-of-truth. The docs/ guide/reference tree (WP3) and the finalized Claude Design brand deliverables under docs/design/ (brand-tokens.md, logo SVGs, mockups) that the site and surfaces build on.
• M6 WP2 — abk init-claude + packaged Claude Code context. New command that installs AI-assistant context into a user’s abkit project (idempotent, version-stamped, re-runnable after upgrade): a marker-delimited managed block in CLAUDE.md (existing content preserved; a stale versioned marker is refreshed in place), the 9 reference rules under .claude/rules/ab-analysis-kit/ (overview, cli, project, experiments, metrics, methods, explore, validate, plan), and the 7 abk-* skills under .claude/skills/ (setup-project, new-experiment, new-metric, explore, validate, plan, feedback). The source tree ships in the wheel (abkit/cli/assets/claude/**) and is read via importlib.resources. Ported from the detectkit donor (cli-and-dx.md §5); mechanism domain-agnostic, content authored for A/B analysis and fact-checked against the M5 as-built engine.
• M6 WP4 — BI reference queries + dashboards (docs/examples/bi/). Connect Grafana / Lightdash / Metabase / Superset to the _ab_results contract table: queries.sql (8 tool-agnostic recipes — headline scoreboard, the effect+CI stabilization chart, raw/CUPED arm values, significance-vs-effective-alpha, MDE/power, cross-experiment board, freshness, config-drift detector), srm_panel.sql (the SRM validity guard), one importable grafana_dashboard.json (ClickHouse), and a README documenting the five hard invariants (read FINAL; group by method_config_id; compare to the row’s two-tier alpha not 0.05; respect the pre-horizon peeking guard via is_horizon/ci_kind; handle NULLs). Guarded by tests/reporting/test_bi_examples.py, which fails if a recipe drifts from the real _ab_results schema. Docs/SQL only — no runtime code.
• M6 WP6 — Prefect deployment scaffold. abk init now also scaffolds runners/prefect.yaml (a Prefect 3 project-deploy config — prefect deploy --all schedules the daily abk run) beside the existing runners/prefect_flow.py. Documents the tag:actual convention the daily job relies on (tag live experiments actual; the demo is tagged example so the schedule skips it) and pins the targeted Prefect major. The [orchestration]/[all] extras now require prefect>=3.0 to match the scaffolded syntax (abkit still never imports prefect). Scaffold test asserts the deployment is valid YAML and the flow parses.
• M5 — sequential analysis, the always-valid CI, abk plan, composed corrections. Opt-in (sequential: {enabled: true}, default off — the fixed-horizon series is byte-identical, no ALGORITHM_VERSION bump, goldens untouched). Landed so far (implementation record: m5-implementation-plan.md; math: statistics-changes.md §4.1):
  • The always-valid confidence sequence (abkit/stats/sequential/) — an asymptotic Gaussian confidence sequence (Waudby-Smith & Ramdas normal mixture) computed as a pure experiment-level MODE transform over the fixed (effect, SE), never a method plugin. SE recovered by CI-inversion (preserving the delta-method covariance); the mixing variance τ² is anchored to the first usable look (stable across runs, computable live). Rows carry ci_kind='always_valid'.
  • The A/A matrix’s sequential side-by-side column (D8) — abk validate now measures the always-valid peeking FPR, power, and CI-width beside the fixed ones: where the fixed peeking FPR breaks budget, the always-valid twin returns to ≈α (the honest completion of the peeking story). Surfaced in the matrix report (a “peeking (AV)” column + a second curve) and the live explore calibration chip.
  • Pipeline activation — a plain abk run on a sequential-enabled experiment emits always-valid rows. scheme: alpha_spending (group-sequential) is a clear “planned M6” config error.
  • The toggle self-invalidates (B4) — flipping sequential.enabled on an existing experiment now re-plans the affected series in place on a bare abk run (no --full-refresh needed): sequential.enabled is deliberately not in method_config_id, so the planner compares the persisted per-pair ci_kind against the mode this run stamps and forces a full recompute on a mismatch — idempotent (a steady sequential experiment still plans zero) and robust to the first-usable-look τ² anchor legitimately leaving a later-usable pair fixed.
  • Explore threading (B5) — the live explore recompute now mirrors the baked per-pair CI vocabulary so the cockpit never mixes fixed & always-valid on one chart. A pair is widened live iff its persisted rows are already always-valid (a read-view of what abk run stored — so the multi-pair case where the anchor left a late-usable pair fixed, and a not-yet-applied config toggle, both stay consistent); each widened point uses the same first-usable-look τ² (the configured knob state reproduces the baked always-valid bounds — exactly for the closed-form families). α-inversion cannot honestly widen an already-widened persisted CI, so under the mode those cutoffs are dropped with a Reload hint rather than shown as a silent fixed CI; a switch to a sequential-ineligible method (bootstrap) turns the mode off. Server-only — no bundle change (the client draws whatever bounds the reply carries).
  • The readout reads always-valid rows early (WP4) — the pre-horizon withholding that refuses WIN/LOSE/FLAT before the planned horizon now lifts for a row whose persisted ci_kind is always_valid (a fixed row is still withheld). An early decisive verdict names its own justification (“called before the planned horizon under an always-valid confidence sequence — peeking-safe by construction”). The “covers X% of a weekly cycle” representativeness caveat on a sub-week verdict is promoted from a caveat bullet to a structured weekly_cycle_pct rendered as a chip on the HTML report’s verdict card. The daily-SRM posture under sequential is settled (plan D9): daily & coarser keep the χ² gate (bounded looks on a ~3.3σ hard gate ⇒ negligible peeking inflation); only sub-day (a follow-up) swaps to the anytime-valid multinomial test.
  • Sub-day anytime-valid SRM (WP5) — below 1d cadence the SRM gate swaps from χ² to an anytime-valid Dirichlet-multinomial e-process (Lindon & Malek 2022; statistics-changes.md §4.2): a dense sub-day cadence would peek the χ² hard gate dozens of times a day → false alarms, whereas the e-process is valid at every look by construction. Dispatched on experiment.is_sub_day() (daily & coarser are unchanged). One verdict per look, stamped from the cumulative as-of exposure counts (get_exposure_count_stream) — the truthful as-of series the M2 whole-cohort broadcast deferred — and it runs even on demoted rows. Default prior is the paper’s uniform Dir(1,…,1); the anytime false-alarm rate holds ≤ α for any fixed prior. It is an additive gate, not a registered method: no ALGORITHM_VERSION bump, goldens untouched, no schema change (reuses srm_flag/srm_pvalue).
  • abk plan — the read-only pre-launch power/sizing planner (WP6) — abk plan --select <exp> [--metric <m>] [--mde <pct>] [--power] [--alpha] [--baseline] reports, per comparison, the required sample size to detect a target MDE, the achievable MDE at the current size, and the achieved power — at the effective two-tier alpha — plus the projected look count and cost shape from the same generate_grid enumeration run/config-lint use. Baseline moments come from the latest persisted _ab_results per-arm stats (a --baseline metric:mean=..,std=..,n=.. override sizes a greenfield experiment); the target MDE defaults to the comparison’s min_effect. Strictly read-only — no lock, no _ab_* writes. Refuses what it cannot size honestly: ratio and bootstrap/resampling methods have no versioned power formula (SKIPPED, never invented math), and CUPED is sized on the raw persisted variance (ρ is not persisted per row) as a flagged conservative upper bound. Runtime / ASN (days-to-N from an arrival rate + the sequential design’s average sample number) are a named M6 deferral.
  • The composed multi-metric FWER/FDR family sweep (D9, WP7+WP8) — M4 validated only the per-cell peeking FPR at the correct two-tier alphas; D9 closes the family-level loop. The read-time composed rule (two-tier Bonferroni ∘ Benjamini-Hochberg) is extracted from the readout’s inline _build_sig_map into one shared pure helper (stats.correction.composed_significance, WP7) that the readout and the sweep both apply — a behavior-preserving refactor (goldens untouched, verdict-snapshot pinned). abk validate then runs the sweep: each iteration draws one unit→arm assignment over the union of the metrics’ cohorts (the real single-assignment semantics; no imputation — a unit absent from a metric doesn’t contribute), scores every metric at its horizon, and tallies the empirical family-wise error rate (any false rejection) and false-discovery rate (mean false fraction among rejections). On the placebo (complete) null FWER and FDR coincide by construction, at the composed rule’s nominal rate (≈α per tier, so ≈2α whole-family under the default two-tier Bonferroni); the budget is anchored to that nominal rate so “over budget” flags a miscalibrated method (clustering), not a loose correction. A planted true effect in one metric leaves the null metrics’ family error controlled. Persisted as one sentinel _ab_aa_runs row (metric='__family__', numbers in details) — no schema change, never lights the per-cell calibration chip — and surfaced as a composed-family band above the report’s A/A matrix (report.js rebuilt). Fixed-horizon only; sequential × composed is a named M6 follow-up.
• M4 — abk validate, the A/A false-positive matrix. The trust artifact that answers “is this method actually calibrated on this data, or does it lie about its α?” (docs/specs/aa-false-positive-matrix.md; the implementation record is m4-implementation-plan.md):
  • abk validate --select <exp> [--method <m>] [--metric <m>] [--iterations N] [--inject-effect <pct>] [--scoring fpr|power|mde] [--report] [--force] — draws N deterministic placebo A/A splits over the experiment’s own pooled cohort (label-permutation, an exact null by construction), scores each declared method’s empirical single-look FPR, cumulative-peeking FPR, power @ MDE, achieved MDE, CI coverage, and effect-exaggeration-at-stop, and persists one _ab_aa_runs audit row per cell at the effective per-comparison alpha. Its own out-of-band lock (process_type='validate', abk unlock-clearable); non-zero exit on failure; stages LOAD → RESAMPLE → SCORE → PERSIST (distinct copy from abk run’s config-lint VALIDATE).
  • Honest peeking FPR — the naive optional-stopping hazard (CI-excludes-zero at any look, pre-horizon refusal off), reported beside the single-look FPR so the jump is visible, with the per-look cumulative curve. Deliberately not the readout’s stabilized verdict (that is the defense); pipeline/readout.py is untouched.
  • The matrix UX — budget-band-colored FPR cells, an explicit Recommended row (FPR-closest-to-nominal, max-power) with a truthful one-line rationale, plain-language per-method verdicts, and the “nominal α 5%, real peeking FPR X%” headline. Rendered by abk validate --report reusing the committed report bundle (no third JS bundle) and surfaced live by the explore calibration chip.
  • Auto mode — a real server-side POST /validate (was a 501 stub) runs a reduced validate, refreshes session.aa_rows in place so the D3 chip greens without an explore restart, and re-seeds the knobs to the recommended config. The Apply gate is unchanged (an uncalibrated Apply still confirms).
  • metric.aa_fpr_budget (a fraction in (0,1]) completes the budget resolver (metric → project → α×1.5); added to the §8 validation matrix.
  • No statistical numbers changed — validate reads the existing from_suffstats methods; the goldens are untouched and no ALGORITHM_VERSION was bumped.

Fixed

• M3 milestone review closure (the WP10 exit gate: 7 lenses / 17 raw findings, verified + inline-triaged — 13 real, all fixed; the full record is m3-implementation-plan.md §5):

  • Apply writes are atomic: the final YAML overwrite goes through temp + os.replace (+fsync) — an ENOSPC/kill mid-write can no longer leave the live config torn while the reply claims nothing was written.
  • Guardrail regression is correction-independent: judged from the STORED CI bounds per D5(c) — BH adjustment can no longer un-flag a stored-significant harm and un-block a WIN (known-answer test added).
  • SRM stays loud over an empty main series: the summary scans ALL comparisons’ series, so the state an explore Apply produces (main series empty under its new id, flagged rows elsewhere) no longer renders a green “SRM ok” chip.
  • The D3 Apply gate keys role flips at the PROSPECTIVE alphas: posted is_main/is_guardrail flips overlay the prospective experiment before effective_alphas, closing the under-gating latent behind the empty _ab_aa_runs (server + regression test).
  • Ctrl-C cannot swallow a successful Apply: serve_explore returns the applied config even when SIGINT races the post-Apply self-shutdown window — the orphan/re-run epilogue always prints.
  • Stale mid-series horizons render honestly: both charts corroborate a stored hz=1 row against the CURRENT config horizon, so an end_date extension no longer paints later cutoffs as decision-grade solid CIs (§4).
  • Cockpit dirty-state fidelity: edited keeps FULL params (an edit back to a spec default no longer silently reverts to the configured value on a rail rebuild; wire bodies are minimalized at send time), and the confirm box’s “Apply anyway” runs the same preflight as the Apply button (a pending Tier-R edit can no longer ride into the YAML).
  • Orphan warnings survive unbindable legacy method blocks; the client remembers a completed covariate /reload (no redundant re-renders); the explore bake test asserts https:// too; build.mjs fails on </script/<!-- tokenizer hazards inside a bundle; header period timestamps are labeled UTC next to the experiment-tz name.

• M3 WP5/WP6/WP8 review-closure (adversarial review, 4 lenses / 25 raw findings; the verify fleet was limit-truncated, findings triaged inline — 7 real after dedup):

  • The D3 calibration gate lost its side doors: correction-only and role-flip-only Applies now gate too (a correction edit re-keys every comparison; a role flip moves comparisons across the two Bonferroni tiers), and the gate keys by the prospective EFFECTIVE per-comparison alpha (effective_alphas over the applied alpha/correction), not the raw body alpha — restoring the mechanically testable “every Apply takes the confirm path” DoD. Params carrying a riding "name" key are keyed exactly as the writer strips them; unbindable params gate conservatively instead of silently skipping the check.
  • Handler-thread hardening: a malformed Content-Length header and a non-numeric alpha in the /apply body are clean 400s (previously a dead thread with no HTTP reply); /apply is serialized under the request lock (two tabs cannot race the archive/rewrite seam or the shared CLI-thread DB manager) and a second Apply after a successful one is a 409; the self-shutdown thread now spawns in a finally, so a client that vanishes mid-reply can no longer leave the server alive with the YAML already rewritten (Ctrl-C would then have lied “experiment unchanged”).
  • /reload refuses on a budget-degraded (suffstats-only) session instead of silently growing a shadow cache the replies keep contradicting, and keeps session.cache_values accounting exact when replacing entries.
  • The HTTP comparisons parser preserves an ABSENT params key as None (the writer’s “a method switch must carry the full param set” guard was bypassable with a fake {}); the provenance header sanitizes newlines (no comment-escape injection into the emitted YAML); the WP5 role-flip test now proves the promised per-comparison alpha shift on a three-comparison fixture (0.05 → 0.025), not a structural equality.

Added

• M4 WP5 — the A/A calibration matrix report + payload block + metric budget (per docs/specs/m4-implementation-plan.md WP5/D10/D12): abk validate --report now bakes a self-contained matrix page by reusing the committed report bundle (no third JS bundle) — the report/ explore payload’s reserved calibration block is filled from the latest _ab_aa_runs invocation (abkit/reporting/calibration.py), so the offline readout and the live explore chip both surface the “nominal α X%, real peeking FPR Y%” headline, the per-method matrix (FPR coloured against the aa_fpr_budget band, the Recommended row + rationale, plain-language verdicts), and the recommended cell’s cumulative peeking-FPR-vs-looks curve. The scorer now emits that monotone peeking_curve (one point per grid look, ending at the reported peeking FPR — the “peeking is the product” visual). Adds MetricConfig.aa_fpr_budget (a fraction in (0, 1]) completing the resolve_fpr_budget chain (metric → project → α × 1.5). No payload version bump; no statistical-number change (goldens untouched). The standalone WP4 matrix template is retired in favour of the shared bundle.

• M3 WP7 — the explore cockpit client (per docs/specs/m3-implementation-plan.md WP7; data-contract §5.1 as amended by D9/D12): the browser half of abk explore, ported from the detectkit tune.ts skeleton to web/src/explore/ and committed as the wheel-shipped abkit/tuning/assets/explore.js (replacing the WP6 placeholder). The windshield: the stabilization chart with D1-tier-styled live segments (solid exact, hatched “approx (α-only)”, the persisted baseline always visible), §4 dashed pre-horizon CIs, greyed insufficient spans, run breaks at server-refused cutoffs, an off-scale indicator, and pinned chips (lift, ±CI, p, power, the D3 calibration chip incl. the alpha-mismatch downgrade, the red SRM gate, the sub-day look counter) re-keyed from every /recompute reply. The side rail is auto-derived from param_specs (Basic = method/CUPED/test_type/alpha; an Advanced disclosure for the rest + correction; identity ⚠ and Tier-R ↻ badges; the donor’s slider identity hazard ported). Tier-R edits route through a per-metric confirm → POST /reload; Apply follows the dirty-slot discipline (role-only entries carry no method key; minimal params) behind the uncalibrated-cost confirm mirroring the server gate, with the archive/orphan/abk clean epilogue. The donor’s stale-drop discipline is re-expressed over HTTP: a monotonic request_id seeded from Date.now() (re-seeded after a two-tab 409), AbortController kill-not-queue, stale replies never clear the spinner, the 130 ms debounce with the flush-before-switch trap. The client resolves raw alpha + correction to the effective per-comparison alpha by mirroring analyze.effective_alphas over the new payload["explore"]["experiment"] block (raw alpha, correction + choices, groups_count, non_main_count). Toolchain: a second build.mjs bundle entry (marker-gated), --abk-explore-accent joins the brand-token layer, the CI hex loop covers tuning/html.py, the wheel gate asserts explore.js, a jsdom smoke suite drives the live half through a fake fetch, and tests/tuning/test_explore_bundle.py pins the bundle packaging + the alpha-mirror substrate. Reviewed: 11 findings fixed pre-merge (stale cached-reply adoption on metric switch, surfaced-subset non_main_count, two-tab 409 lockout, reload-pending Apply bypass, chart listener leak, and six more).

• M3 WP10 — the e2e exit gate (per the plan WP10): tests/e2e/test_first_report.py (scaffold → abk run --report → a verdict-bearing, self-contained readout with the baked payload asserted structurally; re-run byte-stable modulo generated_at; a builder crash yellow-skips) and tests/e2e/test_explore_session.py (the real explore server over live HTTP: persisted numbers reproduced at rel-1e-9, Tier-E alpha recompute + α-inversion on a suffstats-only CUPED series, the stale 409, the Apply gate → .history archive → orphan block → self-shutdown).

• M3 WP8 — abk explore (per docs/specs/m3-implementation-plan.md WP8; cli-and-dx §1): the cockpit shell — abk explore --select <exp> [--metric <m>] [--no-serve] [--no-open] [--profile]. Registered per the house pattern (eager stanza, lazy command body — abk --version stays instant). Resolves exactly ONE experiment (selection errors name the namespace), guards a never-run project with the friendly “run abk run first” noop (D2), prints the startup orphan warning (the same list_method_config_ids scan the driver and abk clean use), streams the session load through the house StageLogRenderer, then serves the WP6 cockpit — or, with --no-serve, atomically writes the static reports/<experiment>__explore.html snapshot (null endpoints — the preview badge, Apply disabled). --metric narrows the opened comparison (default: the main metric). The Apply epilogue echoes the archive path, updated/preserved comparisons, the orphan warning + abk clean hint, and the “re-run abk run --select <exp>” reminder; Ctrl-C cancels with the experiment unchanged. All failures exit non-zero (the house rule).

• M3 WP6 — the explore localhost server + page + payload (per docs/specs/m3-implementation-plan.md WP6/D1/D3):

  • abkit.tuning.server: build_explore_server / serve_explore — the donor’s exact interaction contract on 127.0.0.1:0 with a one-shot token: GET serves ONE pre-rendered page on any path (the token gates only POSTs); POST /recompute answers knob states from the in-memory session — repeatable, advisory, lock-serialized, stale-dropping (outdated request_ids get 409 {stale} before AND after the compute lock — debounced knob drags never queue behind an in-flight bootstrap) and silent; POST /reload executes the confirmed Tier-R actions with its OWN manager inside the serialized handler (re-rendering cached cutoffs under the requested lookback — the session tracks per-entry render lookbacks so the refreshed cache serves subsequent /recomputes) and streams a run-log through server.echo; POST /validate is the reserved M4 slot (501); POST /apply is the only terminal action — the server-side calibration gate (D3: confirm_uncalibrated required while the applied (metric, method_config_id, alpha) keys are not green — with _ab_aa_runs empty until M4 every Apply takes the confirm path), the WP5 seam, the orphaned block + warning echoed in the reply, then self-shutdown from a daemon thread. Invalid configs return 400 and KEEP serving; error detail travels in the UTF-8 body (never the latin-1 status line); oversized bodies drain-then-413; no pipeline lock is ever taken.
  • abkit.tuning.html: render_explore_html — the WP3-hardened template mechanics verbatim (one-pass regex substitution, every < in the baked JSON escaped, no webfonts, abk-explore mount, __ABK_EXPLORE__ global). Ships with a committed placeholder assets/explore.js (honest pending note) until the WP7 cockpit bundle replaces it — the wheel packaging contract was pre-wired in WP3.
  • abkit.tuning.payload: build_explore_payload — the WP2 report payload riding verbatim + the explore block (knob surfaces from param_specs, per-metric initial calibration chip state keyed by the configured (method_config_id, alpha), session-cache facts, ms-epoch cutoffs) and the four endpoint slots (None = the static --no-serve preview badge).

• M3 WP5 — Apply, .history, orphan detection (per docs/specs/m3-implementation-plan.md WP5/D4/D9):

  • abkit.tuning.config_writer: apply_tuned_config — the ONLY mutation seam of abk explore, donor-disciplined validate → archive → re-emit: per-comparison method blocks (matched by metric; a merely-viewed comparison is never written — the dirty-slot discipline), Review-mode is_main_metric/is_guardrail flips (marking only, D9), and experiment-level alpha/correction, merged into the parsed document and validated as a whole (create_method per touched method + ExperimentConfig.model_validate) before ANY filesystem write. Tunability is registry-derived (paired designs and cross-kind methods refused — never a hardcoded name set); identity-excluded params (seed, max_block_bytes) carry over from the slot being retuned via the specs.
  • The previous YAML is archived byte-verbatim (comments included) to <dir>/.history/<experiment>/<experiment>-<stamp>.yml before overwrite — repeated Applies each archive, same-second Applies de-collide, and discovery never picks archives up as live configs. Comments die on re-emit (owner-ratified D4); re-emission is isolated behind the ONE _reemit_yaml strategy function so a comment-preserving ruamel backend can swap in later without contract changes.
  • Orphan detection (NEW vs the donor): old-vs-new method_config_id per touched comparison through the single hashing path; an identity edit over a series with persisted rows yields the orphaned block + the driver-identical warning (abk clean + abk run --select hints) in the result, and the provenance header. Apply never auto-cleans or auto-runs; alpha-only edits and role flips are orphan-free by construction.

• M3 WP4 — the explore recompute engine (per docs/specs/m3-implementation-plan.md WP4/D1/D3/D11/D12):

  • abkit.tuning.session: load_session — the one warehouse load pass at explore start (D2): the persisted per-comparison series plus the bounded Tier-S per-unit cache (latest cutoffs first, older newest-first under a ~2×10⁷-value budget; over-budget degrades honestly to a suffstats-only session with a reason string, never a silent partial cache).
  • abkit.tuning.recompute: RecomputeEngine — one knob state answered entirely in memory (D1, “no warehouse round-trip per knob change”): Tier E exact suffstats reconstruction across the whole grid for the closed-form families (t-test m2 = std²·n; z-test nobs inverted from the persisted SE — never from the one-row-per-unit size_i; ratio-delta via the exact denominator≡1 surrogate; CUPED→t-test “CUPED off” rides the persisted ORIGINAL per-arm mean/std), Tier α alpha-inversion for closed-form rows (symmetric normal CIs only — resampling families are declaratively excluded), Tier S from_samples over the session cache (bootstrap knobs, the stratify toggle, CUPED param edits) with the per-row seed re-derived by the persisted convention so unchanged knobs reproduce stored rows byte-exactly, and Tier R classification for CUPED off→on / covariate_lookback edits (the serialized /reload executes them, WP6). Per-pair points carry an exact/approx/baseline tier; windshield chips (lift, CI half-width, p-value, achieved power at min_effect with honest capability notes); the live method_config_id hashed only through the bound-probe path; knob metadata auto-derived from param_specs (nothing special-cases a method name; a supplied seed is ignored with a warning); QuarantinedMethodError surfaces verbatim.
  • find_calibration + resolve_fpr_budget (D3): the calibration chip lookup keyed by (metric, method_config_id, **alpha**) against the as-built _ab_aa_runs (status='failed'/FPR-less rows never count; alpha edits downgrade to alpha_mismatch; identity edits flip to uncalibrated — that IS the staleness semantics); budget resolves metric-seam → project aa_fpr_budget → α × 1.5.
  • pipeline.analyze.build_container is now public (shared by the engine’s Tier-S path — byte-identical containers to the pipeline); InternalTablesManager.aa_runs_table_exists() guards chip reads on a never-validated project. Sidedness + winsorization stay OFF the knob surface (D12) — deferred to M4 under change control (ROADMAP note).

Fixed

• M3 WP4 review-closure (adversarial review, 4 lenses / 15 findings, the blocker empirically reproduced by an independent verifier):
  • RecomputeEngine.recompute gained the analyze_cutoff-parity gate: a paired or cross-kind knob state (e.g. t-test on a fraction series, whose persisted std_i is the SE, not a sample std) now raises MethodParamError instead of returning a silently ~nobs-fold-collapsed CI labeled tier="exact" (the confirmed major).
  • Tier E now refuses rows whose per-arm columns don’t carry mean/std semantics: a resampling series with a non-mean stat (e.g. median bootstrap) persists the bootstrapped statistic in value_i — such rows recompute only through the Tier-S cache (correct) or stay gaps, never “exact” numbers off the median. Unknown/quarantined legacy row methods are likewise never reconstructed.
  • New declarative BaseMethod.requires_covariate capability flag (CUPED + post-normed families): the Tier-S cache gate reads it instead of guessing from param names, so post-normed-bootstrap — which needs cov_array but has no covariate_lookback param — yields an honest gap on a covariate-less cache instead of an unhandled SampleValidationError.
  • Demoted (insufficient_data) and NULLed (H5) rows now pass through the reply untouched as flagged baseline points (NULL test columns, real sizes) instead of vanishing; the windshield chips read the latest point with inference, so a demoted latest cutoff no longer blanks or shifts them silently.
  • Point size_i keeps the persisted unit-count semantics across every tier (a fraction result’s round(nobs) no longer makes sizes jump between tiers of one series; the method sizes stay on the raw result); the fraction power chip solves on trial counts (nobs) from the reconstruction, falling back to SE-inversion.
  • The session load clamps the cache during the latest-cutoffs pass, bounding the transient peak near the budget in the exact scenario the clamp exists for; knob_surface additionally exposes needs_covariate per method, the correction_tier (correction resolves to the effective alpha upstream — the WP4 DoD’s experiment-level-knob classification), and the cache’s covariate_cutoffs (the WP7 ↻-badge substrate).

Changed

• D11 — canonical unit order in load_metric (M3 WP4; recorded in statistics-changes.md §8; a pipeline-level input-assembly fix, NO ALGORITHM_VERSION bump): every variant’s per-unit arrays are sorted by unit key after fetch, making order-dependent bootstrap replicates reproducible across physical warehouse read orders (ClickHouse guarantees none). Bootstrap rows persisted before the sort may differ from re-computed ones on backends that happened to return a different order; closed-form results are order-invariant.

Added

• M3 WP3 — the self-contained HTML readout + abk run --report (per docs/specs/m3-implementation-plan.md WP3/D7/D8):
  • abkit.reporting.html_report: render_report_html(payload) — one offline HTML per experiment (baked payload + the inlined committed assets/report.js bundle; framework-free, zero network requests, no webfonts — the donor’s Google-Fonts links are deliberately dropped). Template mechanics per the donor (escaped title; data-URI favicon; never .format), hardened past it after the WP3 adversarial review: the baked JSON escapes every < as \u003c (escaping only </ leaves the HTML tokenizer’s <!--+<script double-escaped state able to swallow the real terminator), placeholders substitute in one regex pass (a payload string containing __REPORT_JS__ can no longer be clobbered), and the CLI writes the file atomically (temp + os.replace) so a mid-write failure never truncates a previous good report.
  • web/ — the dev-only bundle toolchain (D7): web/src/shared/payload.ts (the §5.3 contract in documented lockstep with builder.py), web/src/shared/chart.ts (canvas primitives + the one placeholder brand-token layer per branding-and-site.md §3), web/src/report/report.ts (the experiment-primary renderer: verdict banners with rationale/caveats/ guardrails, the stabilization chart — effect + CI vs elapsed_days, zero line, horizon marker, wheel-zoom/drag-pan/hover — four one-axis small multiples (variant means incl. CUPED covariate, pair MDE vs min_effect, p-value vs α, client-derived avg group size), a results/audit table, the red SRM gate chip, the calibration empty state “uncalibrated — run abk validate (M4)”, and the sub-day look counter). Built by web/build.mjs (esbuild, IIFE, es2019) into the committed, wheel-packaged abkit/reporting/assets/report.js.
  • Peeking honesty rendered per data-contract §4 with stable machine-checkable markers: pre-horizon fixed CIs dashed/de-emphasized (abk-prehorizon), insufficient_data cutoffs greyed with counts+SRM only (abk-insufficient), the SRM chip (abk-srm-fail); asserted by the build script, the Python suite, the jsdom smoke suite, and a new CI bundle job that rebuilds web/ and diffs the committed assets (freshness gate).
  • abk run --report (D8, the donor’s tri-state flag): bare → reports/<experiment>.html, a directory → <dir>/<experiment>.html, a .html value → that exact file. Emitted per experiment after its pipeline best-effort — a report failure yellow-skips and never fails the run (the one recorded exception to the CLI exit-non-zero contract) — and even with zero pending cutoffs (the re-run-to-report path). --report with --steps validate is rejected; one .html file with multiple selected experiments is rejected. cli-and-dx §1’s never-wired readout --steps token is amended away (D8).
  • Payload series points gain per-arm keys v1/v2/sd1/sd2/cv1/cv2 (stored value/std/CUPED covariate means) — additive, no schema v-bump — feeding the §5.2 variant-means/lift view; §5.3 amended, payload.ts lockstep.

Fixed

• MDE solve crash + report cost (M3 WP2 review-closure, adversarial re-verification): abkit.stats.power — statsmodels’ solve_power returns a shape-(1,) ndarray from its fsolve fallback for a data-dependent few-percent of ordinary (nobs, ratio) inputs (e.g. n=139, ratio=1.0); under numpy ≥ 2.0 float(ndarray) raised, crashing the readout verdict and report MDE paths. _as_scalar now extracts the value (value-preserving — golden tests unchanged, zero statistical numbers changed). And the report payload’s per-point mde reads the stored mde_1/2 columns only (null when the row did not compute MDE) instead of a read-time statsmodels solve per point — the read-time D5(b) fallback stays verdict-level (one solve per pair on the latest cutoff). A worst-case sub-day payload dropped from ~40–100 s (and a hard crash) to milliseconds; data-contract §5.3 amended.
• Payload consistency (M3 WP2 sweep-closure, second review pass):
  • per-point mde now honours the D5(b) both-present guard — a half-present stored pair (one arm’s MDE solved to inf and was NULLed by enrich) shows null, never the finite arm alone (which would fake adequate power and contradict the verdict on the same cutoff; review finding).
  • srm.observed is the whole-cohort count even under a pinned-end replay, so it stays coherent with the whole-run srm.flag/pvalue the driver computes once and broadcasts (the until= pin is dropped; per-cutoff SRM lands with M5 sequential). §5.3 amended.
• SRM chip loudness under replay (M3 WP2 final-gate, third review pass): the payload srm block is now window-independent (current experiment health) — flag/pvalue come from the latest persisted row overall via a new readout.srm_summary, not the latest charted row. A pinned or empty replay window therefore never silences a failing SRM gate (§6 must-fix) and the flag/pvalue stay coherent with the whole-cohort observed; the chart and verdict remain as-of the window. readout’s experiment-level SRM aggregation is extracted into srm_summary (no behavior change to evaluate). §5.3 amended.

Added

• M3 WP2 — the experiment-primary report payload (per docs/specs/m3-implementation-plan.md D6):
  • abkit.reporting.builder: build_report_payload(experiment, tables, ...) — one versioned JSON-serializable payload per experiment from persisted _ab_results rows, the shared contract for the WP3 readout renderer and the WP6/WP7 explore shell: WP1 verdict block, experiment-level SRM block (driver-mirrored zero-filled exposure counts), M4-shaped calibration: null, look: {n, planned} from the one-enumeration planner grid, terse ms-epoch series points, NaN and ±inf → null, provenance projection (rendered SQL never enters the payload; one metric_query per metric), metric descriptions from the metric YAML, caller-supplied generated_at, inclusive start/end window pinning (historical readout replay), a global point budget with trailing-window clipping + a loud payload warning, and the full-key empty-experiment contract. Zero statistical numbers changed.
  • InternalTablesManager: results_table_exists() / exposures_table_exists() — the never-run-project guards for read-only surfaces (reporting never creates schema). (A short-lived until= bound on get_exposure_counts was added here and then removed in the review passes below — the SRM block is whole-cohort/window-independent; see the Fixed entries.)
  • Review-driven consistency rules (adversarial review, 4 lenses): rows for variant pairs outside the declared arms are excluded from every payload surface with a loud warning (never silently mixed into look/period/BH); the driver’s orphaned-method_config_id scan is surfaced as a payload warning on the read path too.
  • Specs amended (data-contract-and-reporting.md §5: subsections numbered 5.1/5.2, the D2 explore data-source rewording, new §5.3 payload contract; §2 metric-description sourcing note).
• M3 WP1 — the readout decision core (per docs/specs/m3-implementation-plan.md D5):
  • abkit.pipeline.readout: pure read-time WIN/LOSE/FLAT/INCONCLUSIVE verdicts over persisted _ab_results rows — SRM hard gate; pre-horizon withholding (extends to FLAT); elapsed-time stabilization over the trailing readout.stabilization_days (default 7, floored at 3 informative cutoffs); FLAT gated on min_effect vs the pair MDE with a read-time MDE fallback for t-test/z-test rows (the z-test nobs inverted from the persisted SE, never the unit count); guardrail regression under the owner-ratified guardrail_policy: block | warn; read-time Benjamini-Hochberg rescoring (pulled forward from the M5 roadmap line — compute-time BH rows carry the raw alpha); orphaned/unconfigured row filtering with warnings. Verdicts are read-time only, never persisted. Zero statistical numbers changed.
  • Experiment config: readout: {stabilization_days, guardrail_policy} and per-comparison min_effect / desired_direction (read-time only — never part of method_config_id); specs amended (data-contract-and-reporting.md §1, declarative-config.md §2).
• M2 — declarative config + DB layer + the recompute pipeline (per docs/specs/m2-implementation-plan.md):
  • abkit.core: duration parser (N{s,m,h,d,w}), TableModel/ColumnDefinition (+max_length for MySQL key budgets), and period_planner — ONE pure grid enumeration (scalar + dense-early schedule cadence, experiment-tz midnight snapping, DST-safe, horizon always flagged) consumed by BOTH the validator’s look gates and the planner anti-join; data_lag: 0 + half-open windows reproduce *_wo_curr_day exactly.
  • abkit.database: generic CH/PG/MySQL managers with the quorum atomic lock primitive (PG single-statement INSERT…ON CONFLICT…DO UPDATE…WHERE; MySQL row-alias upsert with the claim verdict latched into a session variable; ClickHouse advisory claim with a deterministic read-back tie-break) and the greenfield _ab_* schema: _ab_experiments, _ab_exposures (persisted cohort), _ab_unit_state (replace-not-sum, keyed per source-table+column-set+unit+day; twice-run invariant tested), _ab_results (the BI contract incl. new warnings/diagnostics JSON columns — spec §2 amended), _ab_aa_runs, _ab_tasks; strictly-monotonic distinct created_at via next_version_ts().
  • abkit.config: pydantic Experiment (primary entity; cadence duration-or-schedule union; sub-day gates) / Metric (type + column roles) / Method (delegates validation AND method_config_id to the stats factory — one hashing path; quarantined branches fail at validate time) / Project (statistical defaults + max_looks/warn_looks/min_units_per_arm) / Profiles (env-interpolated, lazy driver imports); the full declarative-config §8 level-2 validation matrix incl. the macro-usage lint and the peeking warnings; project-root discovery + the two-level selector.
  • abkit.loaders: StrictUndefined Jinja with the authoritative ab_* built-ins and the packaged assignment macro (ab.exposed_units() — dialect-aware cohort dedup, both window predicates, exposure filter); exposure loader (idempotent per experiment; unit-in-two-variants is a hard error) and metric loader (one-row-per-unit REJECTED on violation with the GROUP BY hint).
  • abkit.pipeline + abkit.compute: the v1 full-window recompute pipeline — lock → catalog → exposures once → SRM gate (blocking-but-non-dropping, broadcast to every row) → per-comparison anti-join plan (Python-computed watermark) → analyze (declarative input_kind/is_paired dispatch; two-tier Bonferroni; deterministic per-row bootstrap seeds; insufficient_data demotion) → enrich (the full contract row) → LWW persist; worker pool across experiments; backlog + orphaned-series warnings.
  • abk CLI: run (validate/plan/load/compute steps, --full-refresh --from/--to, the inspectable effective-alphas echo, the red SRM FAILED gate line), unlock, clean (method_config_id drift GC + orphaned experiments; dry-run default), and init — a runnable example (z-test fraction + CUPED sample metrics, assignment SQL, a deterministic ClickHouse seed dataset, Prefect flow example) that round-trips through the real config classes and the L2 validator at scaffold time.
  • Tests: 905 (incl. an in-memory SQL-semantics fake backend, a synthetic warehouse that aggregates a real event log per rendered window, the machine-independent first-run e2e mirroring the seed generation rule, and a testcontainers ClickHouse e2e gate that runs where Docker is available).
• M2 stats-core additions (zero number changes; goldens untouched): COVARIATE_LOOKBACK_PARAM on the two CUPED methods (the lookback is identity-bearing — a different pre-period is a different covariate series); declarative BaseMethod.input_kind/is_paired capability attributes.

Changed (M2 recorded deviations — no statistical numbers changed)

• Jinja precedence flip vs the detectkit donor: ab_* built-ins WIN over caller context; a colliding context key raises instead of silently moving the analysis window.

• CLI exit codes: every abk command exits non-zero on failure (the donor echoed and returned 0) — the CLI is the Prefect unit of automation.

• CUPED covariate mechanics (declarative-config §3/§4 amended): the covariate comes from a SECOND render of the same metric SQL over the fixed pre-period window with the exposure filter dropped (legacy semantics — the covariate is the same metric pre-period); the original ab.covariate_window() conditional-aggregate sketch is superseded (its own spec example would have double-counted the pre-period under plain sum()).

• _ab_results gains nullable warnings/diagnostics canonical-JSON columns (plan R7) — the stats core’s human-readable failure signal is persisted, not lost to stderr; data-contract-and-reporting.md §2 amended in the same change.

• M1 — the pure statistical core abkit.stats (importable standalone; numpy/scipy/statsmodels only). Data model: Sample / Fraction / RatioSample, sufficient statistics with the exact legacy mixed-ddof convention (np.var→ddof=0, np.cov→ddof=1), JointMoments, PairedSufficientStats, Welford/Chan-stable merges (accumulate). Plugin method registry + factory + canonical method_config_id (sha256 over registry name + sorted non-default identity params, version appended only when >1; byte-exact-tested; seed identity-excluded). Closed-form methods (t-test, paired-t-test, z-test, cuped-t-test, paired-cuped-t-test, the new ratio-delta) with dual entry (from_samples ≡ from_suffstats); bootstrap family (bootstrap, paired-bootstrap, poisson-bootstrap, paired-poisson-bootstrap, post-normed-bootstrap, paired-post-normed-bootstrap) on a vectorised block-streaming engine with deterministic per-seed draws. Power/MDE (t-test, CUPED-deflated, proportions), Bonferroni (incl. the legacy two-tier scheme) + read-time Benjamini-Hochberg, SRM chi-square gate, deterministic seed derivation (rng.derive_seed).

• Tests (760+): golden tests vs an independent transcription of the legacy engine at rel-1e-9 (incl. the CUPED θ golden and a heavy-tailed sparse-revenue fixture), byte-exact identity-hash tests, bootstrap byte-stability / block-invariance tests, quarantine and known-answer tests (ratio-delta ≡ t-test at denominator ≡ 1), A/A calibration smoke.

Changed

• Engine-hygiene fixes H1–H10 applied per statistics-changes.md §7 (M1 implementation record): Generator-based RNG + deterministic per-row seeds, baseline-faithful sign p-value default with the H4 plug-in as opt-in pvalue_kind: plugin, Hamilton stratum apportionment (quorum-mandated), Poisson mean-only guard, H5 zero-denominator NaN+warning policy, H9 point-estimate effect convention, named-stat registry (register_stat) replacing raw stat_func callables; broken legacy ratio methods quarantined (never silently substituted).
• Adversarial post-M1 review (8 finder angles → 30 verified findings) applied: registry alias-shadowing guard + reload-safe re-registration; param range / finiteness validation at construction (power, n_samples, max_block_bytes); weight_method removed from Poisson schemas and rejected without stratify (a no-op value could fork method_config_id); two-tier Bonferroni supports main-metric-only experiments; paired methods drive through the uniform compare() (a sequence of PairedSufficientStats is a list of ready comparisons); bootstrap memory cap accounts for index matrices + fancy-indexing temporaries; Poisson engine reuses one float64 weight buffer; stratified planning is a single np.unique pass; power/MDE effect-size solves are LRU-cached; TestResult.to_dict derives from dataclass fields; purity of abkit.stats enforced by test.
• Project initiation contract. Architecture synthesized from the legacy ab_testing engine (statistical baseline) and detectkit (architecture / DX), validated by a 5-lens adversarial subagent quorum (all approve-with-changes). See the master plan docs/ru/project-initiation-spec.md and the specs index: architecture, statistics baseline + changes + legacy method catalogue, cumulative-intervals/compute strategy, declarative config, data contract & reporting, A/A false-positive matrix, CLI & DX, branding & site, and the quorum must-fix gate.
• Development scaffolding (this session): packaging (pyproject.toml, setup.py, MANIFEST.in, requirements.txt), pre-commit, GitHub workflows (CI, publish-to-PyPI on tags, website), a minimal importable abkit package with a working abk CLI entry point (abk --version), and smoke tests.

Decisions

• Sub-day cumulative intervals (abk-intervals, 2026-07). cadence is a true duration with schedule support (dense-early grids first-class); NO hard time floor — the hard gate is max_looks (look count is the dangerous variable, not the time unit); data_lag completeness watermark required below 1d; window contract keyed on exclusive UTC end_ts with derived end_date (daily parity byte-clean); fixed-horizon sub-day = monitoring mode (readout still refuses pre-horizon WIN/LOSE), sequential: always_valid is the sanctioned early-decision path; early rows demoted via insufficient_data, never hidden; anytime-valid sequential SRM below 1d; A/A peeking-FPR runs the actual cadence grid + gains an exaggeration-at-stop column; unit-state stays day-grained (sub-day reads = closed-day state + current-day tail). Full record: docs/specs/cumulative-intervals.md §6.
• CUPED covariate window resolved to fixed lookback (whole days, cadence- independent) — the legacy growing window is incoherent at sub-day grain. Record: docs/specs/statistics-changes.md §5.

Locked decisions

• Greenfield storage (legacy dashboard is reference only); statistical math preserved as a baseline then improved deliberately.
• Fixed-horizon CI by default with honest cumulative-peeking FPR in abk validate; sequential (always-valid) CIs opt-in.
• ClickHouse-first; PostgreSQL/MySQL supported. Read-only exposures.

This section was authored pre-release and is cut into the [0.1.0] heading above — the first tagged PyPI release (M1–M6). Roadmap: ROADMAP.md.