Configuration

An abkit project is a directory of declarative files — no Python required to run an experiment. Two files live at the project root and configure the whole project; the rest are the reusable library of experiments and metrics you author.

File	Role
`abkit_project.yml`	Project settings + statistical defaults
`profiles.yml`	Database connections (secrets via env vars)
`experiments/<name>.yml`	The primary entity — one experiment per file
`metrics/<name>.yml` (+ `sql/`)	The reusable metric library, referenced by experiments

A directory becomes an abkit project the moment it contains abkit_project.yml. abk init <name> --db-type {clickhouse,postgres,mysql} scaffolds all of the above with a runnable seed example (ClickHouse is the default backend). This page covers the two project-level files and how experiments and metrics are discovered; for the experiment and metric YAML themselves see the experiments and metrics guides.

Project layout

abk init writes a tree like this (paths are configurable — see paths below):

my_ab_project/
  abkit_project.yml     # project config
  profiles.yml          # database connections
  experiments/          # one experiment YAML per file (the primary entity)
  metrics/              # the reusable metric library (YAML + SQL)
  sql/                  # SQL referenced by `query_file:`

Every command finds the project by walking up from the current directory looking for abkit_project.yml (up to 10 levels), so you can run abk from any subdirectory of the project.

`abkit_project.yml`

The only two required keys are name and default_profile; everything else has a default. abk init writes those two, a compute: block (see below — the scaffold decides incremental_reads explicitly rather than leaving it unset), and the rest commented. The fields below are the authoritative pydantic model (abkit/config/project_config.py).

name: my_ab_project              # required — identifier (alphanumeric / _ / - / space)
version: "1.0"                   # optional (default "1.0")
default_profile: dev             # required — a profile name from profiles.yml

paths:                           # optional — directory names, relative to the project root
  experiments: experiments       # experiment YAMLs (default: experiments)
  metrics: metrics               # metric YAMLs (default: metrics)
  sql: sql                       # query_file SQL (default: sql)

statistics:                      # project-wide defaults; an experiment overrides any of these
  alpha: 0.05                    # significance level, a fraction in (0,1) (default 0.05)
  test_type: relative            # relative | absolute (default relative)
  correction: bonferroni         # none | bonferroni | benjamini_hochberg | holm (default bonferroni)
  power: 0.8                     # target power for MDE / `abk plan` (0,1) (default 0.8)
  aa_fpr_budget: null            # optional — fraction in (0,1] the `abk validate` matrix colours against

limits:                          # look-count & small-sample gates
  max_looks: 5000                # planned cutoffs above this = a config error (the one hard cadence gate)
  warn_looks: 100                # looks above this WITHOUT sequential = peeking warning
  min_units_per_arm: 100         # below this a row is demoted to insufficient_data (written, inference withheld)

timeouts:                        # per-step, in seconds (each 1..86400)
  load: 3600                     # load step (default 3600)
  compute: 7200                  # compute step; also the run-lock staleness threshold (default 7200)

compute:
  mode: recompute                # v1 ships full-window recompute only — the only accepted value today
  incremental_reads: true        # opt-in (m9): eligible comparisons read _ab_unit_state day moments
                                 # instead of re-scanning the fact window; any gap falls back to
                                 # recompute. Changes HOW a number is computed, never the number.
                                 # `abk init` writes `true`; the library default (an ABSENT key) is
                                 # false, and `abk run` says so once the series is long enough to
                                 # matter. Writing it explicitly — either way — records the decision
                                 # and silences that.

The `compute` block

Why the library default is false when the fast path is verified. It is not doubt about the numbers: abk verify-incremental reconciles both read paths at rel-1e-9 across the whole computed series, and any gap in day state falls back to full recompute on its own. The flag guards exactly one thing — an event backfilled later than your data_lag freezes in already-materialized day state, where the recompute path would pick it up at the next look. That is a property of your ingestion SLA, not of abkit, so abkit will not assume it. Set data_lag to your real delay, run abk verify-incremental, and then decide; the scaffolded example project (whose seed data never backfills) ships true.

The `statistics` block

These are project-wide defaults; the corresponding fields on an experiment override them. Nothing in this block enters a method’s method_config_id, so changing alpha, correction, or power never orphans a results series (declarative-config §7).

Field	Meaning
`alpha`	Experiment-level significance, pre-correction. The per-comparison post-correction alpha is derived (see below), never set here. Must be in `(0,1)`.
`test_type`	`relative` (percent lift) or `absolute` (raw difference) — the units the persisted `effect` and any `min_effect` live in.
`correction`	Multiple-testing correction across a comparison family: `bonferroni` (the compute-time two-tier scheme), `benjamini_hochberg` (read-time FDR), `holm` (read-time FWER), or `none`. See below.
`power`	Target power for MDE reporting and `abk plan` sizing. Must be in `(0,1)`.
`aa_fpr_budget`	Tuning-only band for the `abk validate` matrix: a fraction in `(0,1]`; a cell whose measured false-positive rate exceeds it colours red. A per-metric `aa_fpr_budget` overrides it (declarative-config §8). Never touches the pipeline math.

Why the effective alpha isn’t the alpha you set. With correction: bonferroni, abkit applies the legacy two-tier scheme keyed off each comparison’s is_main_metric flag: the main metric gets alpha / C (where C is the number of comparisons the experiment declares — C(variants, 2) by default, or variants − 1 under the experiment’s contrasts: vs_control, see Experiments), and secondary metrics split a stricter alpha / (C × n_secondary) budget (declarative-config §6). Do not compute this by hand — abk run, abk validate, abk plan, and the HTML report all echo the effective per-comparison alpha and the C × metrics divisor, and all four use the same resolver so an A/A cell calibrated for a metric matches what the pipeline actually applied.

`correction`

Two of the four schemes are compute-time and two are read-time, and the difference is visible in the stored rows:

Value	Kind	Controls	The persisted `alpha`
`none`	compute-time	nothing	the raw alpha
`bonferroni` (default)	compute-time	FWER (per tier; see below)	the two-tier effective alpha
`benjamini_hochberg`	read-time	FDR	the raw alpha
`holm`	read-time	FWER ≤ α, under arbitrary dependence	the raw alpha

A read-time scheme decides over the whole family at once (one cutoff’s metrics × declared arm pairs), so there is no per-comparison level to stamp on a row: whether one comparison is rejected depends on the others’ p-values. abk run therefore stores the raw alpha and the readout — the HTML report, abk dashboard, notifications — recomputes the decision every time it is read.

Two consequences worth knowing before you switch:

The verdict and the interval beside it may legitimately disagree. The stored CI is one comparison at its own alpha; the verdict is the family’s. The disagreement is one-directional — you can see an interval excluding zero under a verdict that declines to call it, never the reverse — and the readout attaches an explicit caveat to that pair when it happens.
reject in _ab_results is the per-comparison flag, not the composed decision (which is not persisted at all). BI charts reading the table directly see the pre-family reading; the report is the authority.

holm is uniformly more powerful than dividing alpha by the family size, at the same guarantee. It is not uniformly more powerful than the default two-tier scheme, whose main tier is deliberately loose (it protects the main and the secondary tiers at α each — see statistics-changes §4.3). Switching schemes is opt-in and moves no number until you switch.

The `limits` block (cadence & small-sample gates)

An experiment’s cadence enumerates the cumulative cutoffs (“looks”) over the window. These gates bound that (declarative-config §8, cumulative-intervals §6):

max_looks — the single hard gate. A plan that would produce more looks than this is a config error at validate time. (There is deliberately no minimum-interval floor.)
warn_looks — softer: past it, an experiment without sequential.enabled gets a peeking warning, because fixed-horizon CIs are not valid under repeated looks. Turn on sequential CIs to silence it legitimately.
min_units_per_arm — the small-sample floor. A cutoff with fewer units per arm is still written to _ab_results, but demoted to insufficient_data with inference withheld.

The `tables` block

The seven internal tables (_ab_experiments, _ab_exposures, _ab_unit_state, _ab_results, _ab_aa_runs, _ab_tasks, _ab_notify_states) have a tables: block for forward compatibility, but it rejects any override today — the _ab_* names are canonical. You do not need to set it.

`profiles.yml`

Connections live in profiles.yml, keyed by name, with a top-level default_profile selecting one. ClickHouse, PostgreSQL, and MySQL are all supported. The connection fields are the ProfileConfig model (abkit/config/profile.py).

Two locations, always separate. abkit reads your fact tables from a data location and writes its own _ab_* state to an internal location. Keep them apart so the internal tables don’t clutter shared analytics schemas. ClickHouse and MySQL name these as two databases (internal_database / data_database); PostgreSQL connects to one database and names two schemas (internal_schema / data_schema).

ClickHouse (native protocol; no database: field):

default_profile: dev

profiles:
  dev:
    type: clickhouse
    host: localhost
    port: 9000
    user: default                       # optional (default "default")
    password: ""                        # optional (default "")
    internal_database: abkit_internal   # required — where the _ab_* tables live
    data_database: analytics            # required — your fact tables (queries read here)
    settings:                           # optional — extra ClickHouse settings
      max_execution_time: 600

  prod:
    type: clickhouse
    host: "{{ env_var('ABKIT_CH_HOST') }}"
    port: 9000
    user: "{{ env_var('ABKIT_CH_USER') }}"
    password: "{{ env_var('ABKIT_CH_PASSWORD') }}"
    internal_database: abkit_internal
    data_database: analytics

PostgreSQL (connect to one database; internal/data are schemas inside it):

profiles:
  prod:
    type: postgres
    host: localhost
    port: 5432
    database: analytics                 # required — the database to connect to
    user: postgres
    password: "{{ env_var('ABKIT_PG_PASSWORD') }}"
    internal_schema: abkit              # required — the _ab_* tables
    data_schema: public                 # required — data queries read here
    settings: {}                        # optional — extra psycopg2.connect kwargs

MySQL (8.0+; two databases):

profiles:
  prod:
    type: mysql
    host: localhost
    port: 3306
    user: root
    password: "{{ env_var('ABKIT_MYSQL_PASSWORD') }}"
    internal_database: abkit            # required — the _ab_* tables
    data_database: analytics            # required — data queries read here
    database: analytics                 # optional — default db for the connection
    settings: {}                        # optional — extra pymysql.connect kwargs

Connection field reference

Field	Applies to	Meaning
`type`	required, all	`clickhouse` \| `postgres` \| `mysql`
`host`	all	default `localhost`
`port`	required, all	integer 1–65535
`user`	all	default `default`
`password`	all	default empty; put secrets behind an env var
`database`	required PostgreSQL, optional MySQL, unused ClickHouse	the database to connect to
`internal_database`	ClickHouse / MySQL	where the `_ab_*` tables live
`internal_schema`	PostgreSQL	schema for the `_ab_*` tables
`data_database`	ClickHouse / MySQL	where metric/assignment SQL reads from
`data_schema`	PostgreSQL	schema for data queries
`settings`	all	extra backend driver kwargs (a mapping)

Select a profile per command with abk run --profile prod (also on explore, validate, plan, unlock, clean); without it, abk uses profiles.yml’s own default_profile.

Two default_profile keys. Both abkit_project.yml and profiles.yml carry a default_profile. At runtime the profiles.yml one selects the connection when you omit --profile; the project-file one is not read for connection selection. Keep them in sync to avoid confusion.

Keeping secrets out of YAML

Any string value in abkit_project.yml or profiles.yml may contain an environment-variable placeholder, resolved before the config is validated (abkit/utils/env_interpolation.py). Two syntaxes are accepted:

{{ env_var('VAR_NAME') }} — dbt-style.
${VAR_NAME} — shell-style.

Interpolation walks nested mappings and lists, so it works anywhere in the file. An unresolved placeholder (the variable is not set) is left intact rather than replaced with an empty string — so a missing secret fails loudly (an invalid port or a refused connection) instead of silently connecting with blank credentials. Export the variables before running:

export ABKIT_CH_PASSWORD='…'
abk run --profile prod --select my_experiment

How experiments and metrics are discovered

Experiments and metrics are selected with two separate flags — --select for experiments, --metric for a comparison — but their names live in one shared namespace. Files are discovered recursively under the paths.experiments and paths.metrics directories; a hidden .history/ subdirectory (where abk explore archives pre-tune versions) is always excluded. Every name must be globally unique across the whole project — names are database keys, and the validator rejects a duplicate, including an experiment and a metric that share a name (declarative-config §8).

abk run (and explore, validate, plan, clean) selects experiments with --select (abk run additionally supports --exclude), which accept four selector forms:

Selector	Matches
`signup_test`	an experiment by file name, then by its `name:` field
`experiments/growth/*.yml`	a path glob (relative to the project root)
`tag:actual`	every experiment whose YAML `tags:` list contains `actual`
`*`	everything (also the default when no `--select` is given)

--select resolves the experiment namespace only. Commands that narrow to one comparison (run, explore, validate, plan, verify-incremental) take a separate --metric flag. A selector that matches nothing produces a warning that reminds you of this split.

Tags are the idiom for orchestration: tag your live experiments (e.g. actual) and schedule abk run --select tag:actual. The scaffolded example is tagged example on purpose so a daily job doesn’t pick it up.

Validate the config without a database

Every config change should round-trip through the validator before you touch the warehouse. abk run --steps validate runs the full parse, cross-reference resolution, method-param checks, cadence/looks gates, and a StrictUndefined SQL render smoke-test without connecting to any database — safe to run in CI (declarative-config §8):

abk run --steps validate                # config-lint only, no DB
abk run --select my_experiment          # the real pipeline (validate → plan → load → compute)

Note that abk run --steps validate is a config lint — it is unrelated to the abk validate command, which runs the A/A false-positive matrix against live data. The CLI exits non-zero on any failure, so both are cron/Prefect-safe.

Configuration

Project layout

abkit_project.yml

The compute block

The statistics block

correction

The limits block (cadence & small-sample gates)

The tables block

profiles.yml