Rules of the Growth Engine
How experiments get registered, and what counts as a real win when you don't have big-company traffic. Full detail:
architecture spec.
Stages a product moves through
- Demand Validated — people want the offer (a demand test passed). Not the same as PMF.
- PMF — retention proven: Ellis ≥40% "very disappointed" AND a cohort that plateaus (doesn't trend to zero).
- S3 Channel Discovery — probe channels to find the ONE that works. S4 Compound — go all-in on it, turn the crank. S5 Expand — second channel + pricing.
Demand can be validated before PMF — GDA is that case today (demand ✓, PMF ✗) even while it runs a channel engine.
Registering a test (any Claude session, any project)
If a change is a falsifiable bet on a product metric (a title-format change to lift CTR = yes; a typo fix = no), pre-register it before it goes live:
POST /api/experiments/register (header X-Internal-Auth: $MARKETING_BOT_API_KEY)
{ product, metric, direction, hypothesis, treatment, window_days, ... }
Register before the treatment ships (walk-forward) so the readout is honest, not hindsight. Metric is <product>.<source>.<name> where source is one the engine reads (ga4, organic/GSC, ai_referrals, crank). The treatment field is the "what we changed" shown on the Growth Engine board.
What counts as a winning test — the low-traffic doctrine
We will never have the traffic for classic A/B significance (p<0.05). We don't chase it. Instead:
- The test is XmR, not a p-value. A metric "moved" only if it clears its XmR limit and holds for 3 consecutive periods — one spike is noise.
- Pre-registration is the rigor. Direction + window + baseline declared before the change. At low traffic this replaces statistical power and stops hindsight storytelling.
- Pool to get signal. Read out at the cohort level (all price-guide pages as one series), never per-page — aggregation is the biggest lever for visible signal.
- Replication is our significance. The same win across 2–3 pages / properties / periods beats one big reading.
- Decision-relevance filter. Big enough to act on, not merely detectable.
- "Underpowered" is a valid verdict. If the traffic can't show an effect of a plausible size, the engine says so ("you'd need +Xpp to see anything — pool or skip") instead of forcing a fake call.
WIN = pre-registered · XmR signal in the predicted direction · held 3 periods · at a cohort with enough traffic to have shown it · ideally echoed elsewhere.
How a metric move gets explained (attribution)
When a metric fires an XmR signal, the engine cross-references the build log (what shipped) against live experiments in a 3–14 day window and names the likely cause — and maps it to the term in the product's growth model. That's why build and measure live together.