Experiments

The test ledger is a durable record of hypotheses your agent is testing against your data: what you expect to change, for which subjects, by when, and how big a move counts. It is observational, not a controlled experiment platform - there is no holdout, no traffic split, and Kreel changes nothing on any ad or email platform. The four tools below freeze a hypothesis, compute its verdict once the review date arrives, let you read it back, and archive it when you’re done.

A decided experiment can link to a learning automatically, so a test’s outcome becomes durable brand context instead of a one-off result you have to remember.

record_experiment

Freezes a hypothesis: a homogeneous set of typed subjects, a metric expectation, and an inclusive observation window. Review becomes eligible the day after end_date on the brand’s reporting calendar.

record_experiment(definition: ExperimentCreate, brand: str | None = None) -> dict

definition is an object:

| Field | Type | Description | |---|---|---| | hypothesis | string | Free text, 1-4000 characters. What you expect and why. | | subjects | list of objects | 1-100 entries. See Subject shape below. All subjects must share the same entity_type. | | expectation | object | {metric, direction, threshold, threshold_unit}. See Expectation shape below. | | start_date | string (date) | Inclusive observation start. | | end_date | string (date) | Inclusive observation end. Must be on or after start_date. | | baseline | string | "prior_period" (default) or "control". | | controls | list of objects | Same shape as subjects. Required (and only allowed) when baseline="control". | | origin_reference | string | Optional free-text pointer to where this hypothesis came from (a report, a brief), up to 2000 characters. | | brand | string | Brand name or id. Required when you can reach more than one brand. |

Subject shape: {platform, entity_type, account, ids, selector}.

• platform: meta, google, tiktok, or klaviyo.
• entity_type: creative, campaign, flow, or selector.
• account: the connected account name or id the subjects belong to.
• ids: up to 10,000 explicit entity ids. Required (non-empty, no duplicates) for creative/campaign/flow subjects; must be empty for a selector subject.
• selector: a matching rule instead of explicit ids (for example, a future naming-convention match). Required for a selector subject; must be omitted otherwise.

Expectation shape:

| Field | Values | |---|---| | metric | spend, revenue, conversions, roas, cpa, ctr, open_rate, click_rate | | direction | increase or decrease | | threshold | A non-negative number - the size of move that counts as a win. | | threshold_unit | absolute or relative_pct |

Returns: {experiment, cap} - the created experiment (status: "running") and the same active/cap counts get_experiments reports.

review_experiment

Computes the first verdict once the review date has arrived, and links a learning to the decided result. An annotation you pass cannot override the computed numbers - it’s context alongside them, not a substitute. Calling this again after a verdict exists just returns the saved result; review never re-runs.

review_experiment(experiment_id: str, annotation: str | None = None, brand: str | None = None) -> dict

| Parameter | Type | Description | |---|---|---| | experiment_id | string | The experiment’s id. | | annotation | string | Optional free-text note attached alongside the computed verdict. | | brand | string | Brand name or id. |

Returns: the experiment’s computation (verdict: win, loss, or inconclusive, with observed, baseline, change, relative_change_pct, and a reason), plus the observed/baseline snapshots each carry the window, resolved subjects, base measures, row/account counts, and source references the verdict was computed from.

get_experiments

Reads a saved experiment, or a bounded page of them. Reads never compute or trigger a verdict.

get_experiments(
    experiment_id: str | None = None,
    include_archived: bool = False,
    offset: int = 0,
    limit: int = 50,
    brand: str | None = None,
) -> dict

| Parameter | Type | Description | |---|---|---| | experiment_id | string | Omit to list; pass an id to read one experiment. | | include_archived | boolean | Include archived experiments in a list call. Default excludes them. | | offset / limit | integer | Pagination for a list call. | | brand | string | Brand name or id. |

Returns (list call): {items, total, offset, limit, truncated, active_count, cap, cap_warning}. Returns (single call): {experiment}, or {"error": "not_found"}.

archive_experiment

Soft-archives a decided experiment - it drops out of the default list view, but its result and any linked learning remain available.

archive_experiment(experiment_id: str, brand: str | None = None) -> dict

| Parameter | Type | Description | |---|---|---| | experiment_id | string | The experiment’s id. | | brand | string | Brand name or id. |

Need help?

Email support@kreel.ai.