Journey Optimizer · experiments and holdouts
Did the campaign work?
Your sale campaign has an audience of 100,000 people. Subject B gets more opens, but fewer purchases than A. Change the metric to see which subject you would choose, then compare with people who receive no email to estimate additional purchases.
Example assumptions
Example campaign, rates and results. The intervals on this page use a normal approximation; the Journey Optimizer report uses its own statistical methods.
Reading
Which subject to choose and what the campaign adds
Comparing A and B helps you choose the subject. Comparing with people who receive no email estimates additional purchases.
Results
Opens and purchases by group
With an approximate 95% interval. The holdout does not receive the email, so it only counts for purchases.
The recommendation
Choose by purchases and check what the campaign adds
One subject line beating another does not prove that the campaign generates additional purchases. This example separates the questions with a content experiment and a group receiving no email.
- 01
A business metric
Purchases or revenue, not opens. Opens are inflated by privacy protection in some mail clients and say nothing about impact.
- 02
A valid comparison
Here, a randomly assigned group receives no email so additional purchases can be estimated. Other designs can estimate incremental impact, with their own assumptions and limits.
- 03
Enough people
A small group increases uncertainty, but can detect large differences. If the interval includes zero, these data do not demonstrate additional purchases; they do not prove that the effect is zero either.
- 04
Decide with confidence
Wait until the report shows enough confidence before declaring a winner. Journey Optimizer uses methods designed for reading results while the experiment runs.