Skip to content

Internal tables (_ab_*)

abkit keeps all of its own state in a small set of greenfield tables prefixed _ab_. They are created and maintained for you — you never write DDL by hand — but you will read them: _ab_results is the BI contract your dashboards query, and the rest are useful when you need to understand what a run did, debug a stuck lock, or audit an A/A calibration.

This page documents the schema as it exists in code (abkit/database/internal_tables/ + abkit/database/tables.py). It is a reference, not a tutorial — for how results are produced see data-contract-and-reporting; for the config that drives it see declarative-config.

By founding decision, abkit does not carry over the legacy marts.exp_comparison_results layout or any of its storage internals (data-contract-and-reporting §2). The legacy Grafana dashboard is a reference only — it told us what an analyst needs to see and how they decide. The _ab_* schema is designed from scratch to support that decision logic with plain SQL, owing nothing to the old table shapes.

There are seven internal tables:

TableEnginePrimary keyWhat it is
_ab_resultsReplacingMergeTree(created_at)(experiment, metric, name_1, name_2, method_config_id, end_ts)The BI contract — one row per comparison per cutoff.
_ab_exposuresReplacingMergeTree(loaded_at)(experiment, unit_id)The persisted assignment cohort — optional, copy-mode only (assignment.cohort_copy.enabled: true); absent by default.
_ab_tasksMergeTree(experiment, scope, process_type)Run locks + idempotency.
_ab_aa_runsReplacingMergeTree(created_at)(experiment, run_id)abk validate A/A audit trail.
_ab_experimentsMergeTree(experiment)Informational experiment catalog.
_ab_notify_statesReplacingMergeTree(updated_at)(experiment, metric, name_1, name_2, method_config_id)What --notify last announced — the verdict-change dedup, plus the recurring signals’ memory.
_ab_unit_stateReplacingMergeTree(version)(source_table, column_set_id, unit_id, day)Per-(unit, day) cumulative moments — written by the state stage, read only under compute.incremental_reads.

The tables live in the internal location you configure in profiles.yml, separate from your source data:

  • ClickHouse / MySQL: internal_database
  • PostgreSQL: internal_schema

Fully qualified, that is <internal_location>.<table> — e.g. abkit_internal._ab_results. Your fact/event tables live in the data location (data_database, or data_schema on PostgreSQL) and are only ever read.

The tables are created on demand and idempotently: every CLI invocation that needs them calls ensure_tables(), which creates any _ab_* table that does not yet exist, additively adds any column a newer abkit version has introduced to an existing table (ALTER TABLE … ADD COLUMN; never drops or renames), and is safe to call repeatedly. On a migrated table the added columns sit at the physical end (PostgreSQL has no positional ADD COLUMN), so storage order may differ from a fresh install’s — harmless: every abkit read is column-name-keyed. Read-only surfaces (a report on a never-run project, the explore cockpit, the calibration chip) deliberately do not create schema — they guard with existence checks (results_table_exists, exposures_table_exists, aa_runs_table_exists) so reading never mutates your warehouse.

Column types below are given in ClickHouse spelling (DateTime64(3, 'UTC'), Nullable(Float64), UInt64, Bool, …). The PostgreSQL and MySQL backends map these to their native equivalents through the generic manager; the semantics are the same. All timestamps are UTC and are normalised to naive-UTC on the read path.

Last-writer-wins dedup — read with the grain

Section titled “Last-writer-wins dedup — read with the grain”

Five of the tables use ReplacingMergeTree(<version_column>). On ClickHouse a background merge collapses rows that share a primary key, keeping the one with the largest version — but that merge is asynchronous, so a naive SELECT * can transiently return more than one row per key. Every version stamp comes from a single strictly-increasing, distinct source (advancing at least 1 ms past the previous write) so the “latest” is always unambiguous.

  • abkit’s own reads append FINAL on ClickHouse (no-op elsewhere) to dedup.
  • Your BI queries must dedup too: take the row with the max version per key (argMax(...) / LIMIT 1 BY <pk> on ClickHouse, or a window row_number() filter on PostgreSQL/MySQL). Do not assume one physical row per key.
  • PostgreSQL / MySQL get the same last-writer-wins effect via a version-aware upsert that mirrors ReplacingMergeTree, so the physical table already holds one row per key there.

The version column per table: created_at (_ab_results, _ab_aa_runs), loaded_at (_ab_exposures), version (_ab_unit_state), updated_at (_ab_notify_states). The plain MergeTree tables (_ab_tasks, _ab_experiments) maintain uniqueness through their write protocol (atomic claim / synchronous upsert), not a version merge.


The one table you will query for dashboards. One row per (experiment, metric, variant-pair, method_config_id, end_ts), where end_ts is the canonical exclusive cutoff of a cumulative window measured from the experiment start. Idempotency is last-writer-wins on the primary key via the strictly-monotonic created_at version.

A comparison series is the set of rows sharing (experiment, metric, name_1, name_2, method_config_id) ordered by end_ts; each point is cumulative from start_ts. method_config_id pins the exact method + identity params, so editing an identity param starts a new series rather than mutating the old one.

Test columns are nullable because a small-sample cutoff can be demoted (insufficient_data = true): the row is still written — counts, SRM and window stay visible — but inference (pvalue, effect, bounds, reject) is withheld (cumulative-intervals §6.1.4).

Engine ReplacingMergeTree(created_at) · PK (experiment, metric, name_1, name_2, method_config_id, end_ts)

ColumnTypePurpose
experimentStringExperiment name.
metricStringMetric name.
is_main_metricBoolDrives the verdict; sets the alpha tier.
is_guardrailBoolChecked for regression, never for a WIN.
method_nameStringRegistered method (e.g. t-test, cuped-t-test, ratio-delta).
method_paramsStringCanonical JSON of the method params (single json_dumps_sorted path — exact-string BI filters never split a series).
method_config_idStringStable hash of method + identity params; the series key.
name_1, name_2StringThe variant pair (control vs treatment).
ColumnTypePurpose
start_tsDateTime64(3,'UTC')Window start (experiment start).
end_tsDateTime64(3,'UTC')Cutoff — exclusive half-open edge, the canonical key.
window_secondsInt64Window length in seconds.
elapsed_daysFloat64Fractional elapsed days — the chart x-axis.

The window is a pair of instants; there are no start_date/end_date columns (they were dropped in 0.5.0 — nothing read them). Group and order by end_ts, which is exact at every cadence including sub-day.

When a dashboard genuinely wants a calendar day, derive it — but mind the two traps that make the naive form wrong. end_ts is exclusive, so a daily cutoff carries the next day’s midnight; and it is stored in UTC, while the day a stakeholder means is a day in the experiment’s timezone. Subtract one microsecond, then read in that timezone:

BackendColumn typeExpression
ClickHouseDateTime64(3,'UTC')toDate(end_ts - toIntervalMicrosecond(1), 'Europe/Moscow')
PostgreSQLTIMESTAMP(3)((end_ts - interval '1 microsecond') AT TIME ZONE 'UTC' AT TIME ZONE 'Europe/Moscow')::date
MySQLDATETIME(3)DATE(CONVERT_TZ(end_ts - INTERVAL 1 MICROSECOND, '+00:00', 'Europe/Moscow'))

Substitute the experiment’s own timezone. Two backend notes: the PostgreSQL parentheses are load-bearing (the column is timestamp without time zone holding naive UTC, so the first AT TIME ZONE interprets it as UTC and the second converts it — and ::date binds tighter than AT TIME ZONE, so the outer pair is required too); MySQL’s named-zone CONVERT_TZ returns NULL unless the server’s mysql.time_zone% tables are populated (mysql_tzinfo_to_sql), so use a fixed '+03:00'-style offset if they are not and the zone has no DST.

toDate(end_ts) alone is wrong on both counts: a Moscow experiment’s 2024-07-02 00:00 MSK cutoff is stored as 2024-07-01 21:00 UTC and would read as July 1 by luck, while a UTC experiment would read as July 2 — the day after the one it measures.

start_ts needs no such care (it is inclusive), but the same timezone shift applies: toDate(start_ts, '<experiment timezone>').

ColumnTypePurpose
value_1, value_2Nullable(Float64)Per-arm metric value (arm 1 / arm 2).
std_1, std_2Nullable(Float64)Per-arm standard deviation.
cov_value_1, cov_value_2Nullable(Float64)Per-arm covariate value (CUPED / ratio context).
cov_std_1, cov_std_2Nullable(Float64)Per-arm covariate standard deviation (cuped-t-test only; NULL otherwise and on pre-0.4.0 rows).
corr_coef_1, corr_coef_2Nullable(Float64)Per-arm value↔covariate correlation (cuped-t-test only; NULL otherwise, on pre-0.4.0 rows, and when degenerate).
size_1, size_2UInt64Per-arm unit counts (always populated).
ColumnTypePurpose
alphaFloat64Effective post-correction per-comparison alpha stamped at compute time (see note below).
pvalueNullable(Float64)Test p-value.
effectNullable(Float64)Point estimate of the effect.
left_bound, right_boundNullable(Float64)(1−α) confidence-interval bounds.
ci_lengthNullable(Float64)Interval width.
rejectNullable(Bool)Whether this one comparison rejects at its own stored alpha (≡ CI excludes zero) — a pre-family flag. Under correction: benjamini_hochberg / holm the decision is recomputed over the family at read time and can differ; the readout (report / dashboard / notifications) is the authority there.
mde_1, mde_2Nullable(Float64)Minimum detectable effect per arm.
ColumnTypePurpose
srm_flagBoolSample-ratio-mismatch tripped.
srm_pvalueNullable(Float64)SRM chi-square p-value.
decision_blockedBoolVerdict withheld (SRM gate is blocking-but-non-dropping — rows are still written).
insufficient_dataBoolSmall-n demotion — inference withheld, counts/SRM kept.
ColumnTypePurpose
ci_kindStringfixed (default fixed-horizon) or always_valid (opt-in sequential — statistics-changes §4).
is_horizonBoolThis cutoff is the planned horizon.
ColumnTypePurpose
warningsNullable(String)Canonical-JSON array of warnings routed from the stats core (e.g. H5 zero-denominator explanations).
diagnosticsNullable(String)Canonical-JSON object of context (θ, bootstrap diagnostics, …).
ColumnTypePurpose
metric_queryStringThe metric SQL as authored.
metric_rendered_queryStringThe SQL after Jinja rendering for this window.
watermark_tsDateTime64(3,'UTC')Completeness boundary in force for this row.
created_atDateTime64(3,'UTC')Strictly-monotonic LWW version.

BI notes. avg_group_size = (size_1 + size_2)/2 and the zero_effect = 0 reference line are derived in your query, not stored. Metric descriptions are not stored here — they live in _ab_experiments / metric YAML and are joined by BI, so there is one source of truth. Corrections that are applied at read time — Benjamini-Hochberg and Holm, and the verdict WIN/LOSE/FLAT/ INCONCLUSIVE logic — are not persisted: compute-time rows deliberately carry the raw effective alpha, and the verdict is recomputed at render. Two-tier Bonferroni is reflected here: main metrics and secondary metrics land at different alpha values.


_ab_exposures — the assignment cohort copy (optional)

Section titled “_ab_exposures — the assignment cohort copy (optional)”

The persisted per-unit assignment cohort — optional, copy-mode only: it exists only when the experiment sets assignment.cohort_copy.enabled: true. By default (no-copy, the M8 default) abkit never creates or writes this table: every metric query joins a deduping subquery over your live assignment SQL instead (the ab_cohort_source builtin behind the packaged ab.exposed_units(...) macro), re-rendered and re-validated on every invocation. Either way the cohort is resolved once per run, never re-derived per interval, and stays read-only for compute: the pipeline never writes back into it and never randomizes. The SRM gate always measures the live validated assignment source, not this table.

With copy mode on, the table is written incrementally and append-only (insert_exposures_incremental): each run appends only the newly-matured, grid-anchored closed-interval batches since the last watermark (MAX(exposure_ts), FINAL-deduped, snapped down to its bucket floor) — a routine run never deletes rows. The one exception is abk run --resync-cohort, which deletes the experiment’s copy and rebuilds it from the experiment start through the same engine — the recovery for rows backfilled into an already-scanned closed bucket, which the watermark alone silently misses (the documented copy-mode limitation).

Engine ReplacingMergeTree(loaded_at) · PK (experiment, unit_id)

ColumnTypePurpose
experimentStringExperiment name.
unit_idStringThe randomization unit (user, session, …).
variantStringAssigned arm.
exposure_tsDateTime64(3,'UTC')First-exposure timestamp (drives cumulative counts and the sub-day SRM stream).
stratumNullable(String)Optional stratum label.
loaded_atDateTime64(3,'UTC')LWW version (stamped at load).

Run locks + idempotency. Each pipeline run claims a lock row before doing any work and releases it on exit; a second run against the same experiment finds the lock held and no-ops. The lock grain is (experiment, scope, process_type):

  • abk run uses scope pipeline, process_type run.
  • abk validate uses scope pipeline, process_type validate (its own out-of-band lock, so a validation and a run can’t clobber each other).

A running row whose age exceeds its stored timeout_seconds is treated as stale and can be overridden — so a process that died mid-run (or a database restart) never blocks future runs forever. Release is ownership-checked: a run whose lock aged out and was legitimately stolen will not wipe the new owner’s live row on exit. locked_by is a per-claim owner token (host:pid:nonce), which also makes a held lock human-attributable.

Engine MergeTree · PK (experiment, scope, process_type)

ColumnTypePurpose
experimentStringExperiment name.
scopeStringLock grain — pipeline today (the key shape reserves per-metric scopes for later parallelism).
process_typeStringrun or validate.
statusStringrunning | completed | failed.
started_atDateTime64(3,'UTC')Claim time (staleness is now - started_at vs timeout_seconds).
updated_atDateTime64(3,'UTC')Last update.
locked_byStringOwner token host:pid:nonce.
error_messageNullable(String)Failure detail, recorded before the error propagates.
timeout_secondsInt32Staleness horizon for this claim.

If a run is killed uncleanly and leaves a running row behind, clear it with abk unlock — it force-releases the lock (ignoring the age check) and marks the task completed so future runs proceed without --force.


_ab_aa_runs — the A/A validation audit trail

Section titled “_ab_aa_runs — the A/A validation audit trail”

Written by abk validate (the A/A false-positive matrix — a placebo label-permutation experiment, not a linter). One row per scored (experiment, metric, method) cell: empirical false-positive rate vs the nominal alpha, the honest cumulative-peeking FPR over the real cadence grid, power / achieved-MDE under injected effects, CI coverage, effect exaggeration at stop, plus a plain-language verdict. When sequential is in play the same measurements appear again in the *_sequential columns, side by side with the fixed-horizon ones.

This is an audit trail: it is informational, never read by the run pipeline, and deliberately not pruned by abk clean — it is kept forever. The rows also drive the explore cockpit’s calibration chip: the persisted alpha is the same effective post-correction alpha the chip and Apply seam use, so a matching cell lights the chip as calibrated. run_id is {run_stamp}:{cell_hash} — one row per cell, no version collapse across cells.

Engine ReplacingMergeTree(created_at) · PK (experiment, run_id)

ColumnTypePurpose
experimentStringExperiment name.
run_idString{run_stamp}:{cell_hash} — one row per scored cell.
metricStringMetric name.
method_nameStringMethod scored.
method_paramsStringCanonical JSON of the method params.
method_config_idStringMethod + identity-param hash.
modeStringRecommended-row objective: fpr | power | mde.
iterationsInt32Number of placebo iterations.
alphaFloat64Effective post-correction alpha (matches the run / chip).
injected_effectNullable(Float64)Effect injected for power/MDE modes.
fprNullable(Float64)Single-look false-positive rate (horizon only).
peeking_fprNullable(Float64)Cumulative-peeking FPR across all looks.
powerNullable(Float64)Power under the injected effect.
achieved_mdeNullable(Float64)Achieved minimum detectable effect.
coverageNullable(Float64)CI coverage.
effect_exaggerationNullable(Float64)Effect exaggeration at stop.
tau2Nullable(Float64)Frozen mixture variance (sequential).
fpr_sequentialNullable(Float64)Single-look FPR under the always-valid CI.
peeking_fpr_sequentialNullable(Float64)Peeking FPR under the always-valid CI.
power_sequentialNullable(Float64)Power under the always-valid CI.
coverage_sequentialNullable(Float64)Coverage under the always-valid CI.
effect_exaggeration_sequentialNullable(Float64)Exaggeration under the always-valid CI.
ci_widthNullable(Float64)Fixed-horizon CI width.
ci_width_sequentialNullable(Float64)Always-valid CI width.
verdictStringPlain-language verdict for the cell.
detailsStringCanonical JSON of supporting detail.
statusStringsuccess | failed.
error_messageNullable(String)Failure detail when status = failed.
created_atDateTime64(3,'UTC')LWW version.

An informational catalog: one row per experiment carrying its resolved metadata (dates, variants, split, cadence, alpha/correction, sequential settings, tags, config path). The run pipeline never reads it back for a decision — it exists so BI can join human-readable metadata (descriptions, tags, the config path) to _ab_results from one source of truth. It is upserted once per run (delete + insert), preserving the first-seen created_at.

Engine MergeTree · PK (experiment)

ColumnTypePurpose
experimentStringExperiment name.
descriptionNullable(String)Free-text description.
statusStringdesign | running | concluded | archived.
is_actualBoolWhether this config is the current one.
start_ts, horizon_tsDateTime64(3)Experiment window, RESOLVED to naive UTC (horizon_ts exclusive) — the same frame as _ab_results.start_ts, so a join lines up; use timezone to read them back in local time.
unit_keyStringRandomization unit key.
cadenceStringCanonical JSON — scalar or schedule.
interval_anchorStringmidnight | start | an ISO instant — where the cutoff lattice sits. Informational: a date and a midnight timestamp resolve alike but stringify differently.
data_lag_secondsInt64Completeness watermark lag.
timezoneStringExperiment timezone.
variantsStringCanonical JSON array (config order).
expected_splitStringCanonical JSON object.
alphaNullable(Float64)Effective alpha.
correctionNullable(String)Correction method.
contrastsStringall_pairs | vs_control — the family the alphas were divided by (m13 STAT-1b). all_pairs divides by C(variants, 2), vs_control by variants − 1 and writes only the control-vs-treatment pairs, so deriving the divisor from variants alone is wrong for a vs_control experiment. Added additively in 0.8.0 with DEFAULT 'all_pairs', which is what every pre-0.8.0 experiment’s family was — an upgraded install picks it up on the next run, with no recreate.
sequential_enabledBoolSequential opt-in flag.
sequential_schemeStringSequential scheme name.
comparisonsStringCanonical JSON comparison summary.
pathStringexperiments/<name>.yml.
tagsStringCanonical JSON array.
created_at, updated_atDateTime64(3,'UTC')First-seen / last-write times.

_ab_notify_states — what each comparison last announced

Section titled “_ab_notify_states — what each comparison last announced”

The memory behind abk run --notify: one row per comparison recording the last message that actually went out, so a scheduled run every hour is not a message every hour. Nothing reads it except the notification dispatcher — it is not part of the BI contract and no number in it is a statistic.

The dedup rule it serves: a change always sends, an unchanged value never re-sends. “Value” is the pair (last_verdict, last_srm_flag), not the verdict word alone — a pair that is already INCONCLUSIVE (the normal state before the horizon) keeps that exact word when its sample-ratio gate breaks, and deduping on the word would swallow the SRM alarm.

A row is written only after a channel accepted the message. If every channel was down, nothing is recorded and the next run tries again — an announcement nobody received must not become history.

The same table also holds the two RECURRING signals’ memory (abk run’s stale and abk validate’s calibration_red), one row per experiment per kind under the sentinel key metric='__stale__' / '__calibration_red__' with an empty arm pair — which is what keeps it disjoint from every comparison row, since a variant name cannot be empty. There last_verdict holds the condition’s signature (the sorted metric names that were behind, the sorted red cells) and an EMPTY signature is written when the condition clears, so the same backlog recurring months later is announced again instead of deduping against a stale row. These are also the only kinds notify.cooldown_seconds applies to.

Engine ReplacingMergeTree(updated_at) · PK (experiment, metric, name_1, name_2, method_config_id)

ColumnTypePurpose
experiment, metric, name_1, name_2StringThe comparison and its arm pair.
method_config_idStringIn the key on purpose: a re-tuned comparison is a different measurement, so it starts a fresh announcement history instead of inheriting the old method’s.
last_verdictNullable(String)The verdict last announced — or, on a sentinel notice row, the condition’s signature ("" once it cleared).
last_srm_flagBoolWhether that message carried a failed SRM gate.
last_notified_atNullable(DateTime64(3,'UTC'))When it was delivered.
notify_countInt32How many messages this comparison has produced.
updated_atDateTime64(3,'UTC')Version column.

Deleting a row (or purging the experiment with abk clean --orphaned-experiments) resets the dedup: the next notified run announces the current verdict as if it were new.


The incremental compute path’s storage, not part of the BI contract. It holds cumulative per-unit statistical moments, day-bucketed, keyed by (source_table, column_set_id, unit_id, day).

  • Written by the pipeline’s state stage (between load and compute; in the abk run default steps): every closed local day of every STATE-eligible metric — closed-form (unseeded) comparison, non-stratified, no explicit columns.covariate role, SQL free of ab_cov_* — is rendered once and replaced in (replace-not-sum, so re-running a day leaves aggregates unchanged; cumulative-intervals §5.2). The series is strictly contiguous: every day up to the last materialized one exists, and any failure truncates the tail rather than leaving stale rows.
  • Read only with compute.incremental_reads: true (opt-in; scaffolded on by abk init, library default off): eligible comparisons then load each cutoff as one additive per-unit SUM over closed days plus, for sub-day cadence, a fact scan of just the current-day tail — instead of re-scanning the whole cumulative window. Any gap in the materialized series falls back to full recompute for that cutoff (with a warning), so a missing day can never become a silent undercount.

In v1 the source_table key is scoped per (experiment, metric) — the per-day render is cohort-filtered, so the “co-located metrics share one set of moments” ideal is deliberately deferred. Because no experiment owns these rows in the _ab_* housekeeping sense, abk clean deliberately leaves them alone (as it does _ab_aa_runs); a metric-SQL or cohort-config edit orphans its series, which the next run sweeps.

Engine ReplacingMergeTree(version) · PK (source_table, column_set_id, unit_id, day)

ColumnTypePurpose
source_tableStringFact source table.
column_set_idStringIdentifies the column-role set (value/covariate/ratio).
unit_idStringRandomization unit.
dayDateDay bucket.
nUInt64Unit-day observation count.
sum_value, sum_value_sqFloat64Mean / t-test moments.
sum_cov, sum_cov_sq, sum_value_covFloat64CUPED co-moments.
sum_denominator, sum_denominator_sq, sum_value_denominatorFloat64Ratio moments.
versionDateTime64(3,'UTC')LWW version (replace-not-sum: re-running a day leaves aggregates unchanged).

Moment columns unused by a given column set stay 0.


CommandEffect on _ab_*
abk runClaims/releases the _ab_tasks lock; upserts _ab_experiments; writes _ab_results. With --notify, reads and writes _ab_notify_states (the verdict-change dedup). Reads _ab_results for the planner anti-join. With assignment.cohort_copy.enabled: true only, appends incrementally to _ab_exposures (or rebuilds it with --resync-cohort); the no-copy default never touches that table.
abk validateClaims its own _ab_tasks lock (process_type=validate); writes _ab_aa_runs. With --notify, reads and writes its sentinel row in _ab_notify_states. Never writes _ab_exposures — a placebo split is in-memory only.
abk exploreRead-only over _ab_results; reads _ab_aa_runs for the calibration chip (Auto mode can write _ab_aa_runs).
abk planRead-only pre-launch sizing — no internal-table writes.
abk cleanPrunes orphaned result series (delete_results) and, with --orphaned-experiments, purges every experiment-keyed table (_ab_experiments, _ab_exposures, _ab_results, _ab_tasks, _ab_notify_states). Never touches _ab_aa_runs or _ab_unit_state.
abk unlockForce-clears a stale/held _ab_tasks lock.
abk init / abk init-claudeScaffold files only — no database access.