In plain words
A split test, or A/B test, shows different versions of an offer to different visitors at the same time. Each visitor is assigned to a version at random and keeps seeing it. The versions are then compared on what they earned.
Randomness and the shared period are what make the comparison fair. Both versions get the same mix of traffic, the same weather, the same ad campaigns. Whatever differs in the result is down to the offer, or to chance, and with enough buyers chance can be ruled out.
Formula
- Profit per visitor of an arm = the profit of its orders ÷ the visitors assigned to it
- Result = the difference between arms, with an interval around it
Worked example
The control ships the $40.00 bag free and keeps $15.36 per order. The variant charges $7.00 for shipping and keeps $21.81. The control converts 3.0% of its visitors and the variant 2.3%.
The variant converts worse and wins: $0.50 per visitor against $0.46. By the engine's figure, the control would need 8.9% more buyers to catch up.
The example order
A $40.00 bag of coffee, shipped free, every cost counted
2 declared kinds cost nothing on this option
| Step | Grain | Amount |
|---|---|---|
| Revenue | – | $40.00 |
| Cost of goods | per unit | −$12.00 |
| Pick and pack | per order | −$1.18 |
| Packaging | per parcel | −$0.50 |
| Shipping label | per parcel | −$7.50 |
| Card fee per order | per order | −$0.30 |
| Refunds | percent | −$2.00 |
| Card processing | percent | −$1.16 |
| Contribution margin | – | $15.36 |
| 2 kinds costing nothing | – | $0.00 |
Computed by the margin engine in Offer Suite's own code, in whole cents, from a $40.00 bag of coffee with a $12.00 unit cost.
A before-and-after comparison is not a split test
Changing an offer and comparing this month with last month mixes the change with everything else that moved: season, ad spend, stock. Only showing both versions over the same days, to visitors divided at random, removes that.
Using it
Decide the metric before you start, and make it profit per visitor. Run both arms for whole weeks, so each sees the same weekdays. Do not stop the moment one arm is ahead: early leads reverse. And test offers that could plausibly win. A version that needs 40% more buyers to break even is not worth the traffic.
In Offer Suite
An Offer Suite experiment splits an offer's visitors across arms, where each arm is one version of the offer and the control is the live one. Arms are compared on realized profit per visitor, net of refunds.
Source: Offer Suite's experiment design.