Creative that adapts to every customer segment

The holdout test: the 10 percent you never mail, and what it proves about the other 90 percent

The 10 percent you never mail is the control group that proves the other 90 percent was worth mailing.

Ten percent of your list never gets an email. Not because they unsubscribed, but because you held them back on purpose. That is the holdout, and it is the only honest way to know whether a segment is worth mailing at all. Without it, every open rate is a guess, and every revenue claim is a story.

What the holdout actually measures

A holdout measures incremental revenue: the purchases the segment produced that would not have happened without the emails. The mailed group buys at some rate. The holdout buys at some rate. The difference is the segment's contribution, and everything else is noise. Without a holdout, the mailed group's purchases are attributed to the campaign by default, and the attribution is a guess dressed as a report.

Klaviyo's 2024 benchmarks say segmented campaigns earn 46 percent higher opens and 101 percent higher clicks than unsegmented ones. Those are real numbers, and they still do not prove a segment is worth mailing, because opens and clicks are not revenue. The holdout is the test that connects the campaign to the money.

How to set one up without breaking anything

Pick a segment, split off 10 percent at random, and exclude that slice from the segment's campaigns. The split has to be random, not by signup date or by value, because a holdout chosen by any other rule is not a control group. The holdout stays out for a full campaign cycle, 30 to 90 days, and the comparison is purchases per person, not opens or clicks.

The cost is the revenue the holdout would have produced if mailed. For a healthy segment that cost is small, because the segment's lift is real and the holdout is only 10 percent. For a segment with no lift, the cost is zero, because the holdout buys just as much without the emails. The holdout pays for itself by finding the segments that are not worth the sends.

Reading the result without fooling yourself

Read purchases per person, not total revenue, because the mailed group is nine times larger than the holdout and the totals will always favor it. Read it over the full cycle, not after the first send, because a single campaign is noise. And read it with a threshold: a lift under 5 percent is probably random variation, and a lift over 20 percent is a segment worth keeping.

The hardest result to accept is a zero lift. It means the segment is a label on people who would have bought anyway, and the campaigns are cost without incremental revenue. The right response is to retire the segment, not to shrink the holdout until the numbers look better.

Which segments deserve a holdout first

Start with the segments that cost the most: the largest ones, the most frequently mailed ones, and the ones whose revenue claims justify the most sends. A holdout on a segment mailed twice a week pays for itself in a month. A holdout on a segment mailed twice a year is a curiosity, not a control.

Then hold out the segments you are most sure about. The sure ones are the ones most likely to be wrong, because nobody tests a segment they are confident in, and confidence without a control group is a story, not a measurement.

Keep reading

Frequently asked questions

What is a holdout test, in plain terms?

A holdout is a randomly selected slice of a segment, usually 10 percent, that never receives the segment's campaigns. Their behavior is the baseline. The difference between the holdout and the mailed group is the incremental revenue the segment actually produced.

How big should a holdout be?

Ten percent is the usual size: large enough to measure, small enough that the revenue cost is acceptable. A smaller holdout is noisy. A larger one costs more revenue than the answer is worth. For a segment under a thousand people, use a fixed count, not a percentage, or the holdout is too small to read.

How long do you run a holdout?

At least a full campaign cycle, usually 30 to 90 days. A holdout measured over a week measures the week, not the segment. The comparison needs enough sends to separate a real lift from random variation.

What does it mean if the holdout buys as much as the mailed group?

When the holdout buys as much as the mailed group, the segment is not doing anything. The emails are not driving the purchases. The segment is a label on people who would have bought anyway, and the campaigns are cost without incremental revenue.