Your email dashboard says your abandoned-cart and welcome automations generated revenue last month. But how much of that revenue would customers have spent anyway? If you rely only on orders attributed to email, it is hard to tell whether a flow created new sales or simply received credit for purchases already likely to happen.
A holdout test helps you answer that question. You randomly withhold a specific automation from a small, comparable group of eligible customers, then compare their purchasing behavior with customers who received the usual experience. The difference is your estimate of incremental revenue from the email automation. Here is how to run that test without letting overlapping campaigns, mismatched groups, or reporting windows distort the result.
What does a holdout test measure?
A holdout test divides eligible customers into two groups at random:
- Treatment group: Customers remain eligible to receive the automation as usual.
- Holdout group: Customers do not receive the automation being tested.
You then measure all qualifying store revenue from each group during the same defined period—not just orders that an email platform attributes to a click or open. If the groups are comparable and the treatment group spends more, that difference is evidence that the automation contributed incremental revenue.
Holdouts answer a different question from email attribution. Attribution asks, “Which orders followed an email interaction?” A holdout asks, “What happened because this automation was available?” Both can be useful, but only the second directly tests the automation’s added effect.
How to set up an ecommerce email holdout test
1. Choose one automation and one business question
Start with a flow that matters to your store, such as a welcome series, abandoned-cart sequence, or post-purchase cross-sell. Define exactly what you are testing. For example: “Does our abandoned-cart email sequence increase revenue from shoppers who become eligible for it?”
Do not change the subject lines, timing, offer, or audience rules midway through the test. If several things change at once, you will not know which experience produced the result.
2. Define eligibility before you randomize
Write down the event that makes someone eligible, such as entering the welcome flow or meeting your abandoned-cart criteria. Apply the same eligibility rules to both groups. Include only people you can legitimately contact under your applicable consent and marketing rules; a holdout should not be used to compare contactable customers with people who were never eligible to receive marketing.
Randomize at the customer level whenever you can. If the same shopper triggers a flow more than once, keep them in their assigned group for the test. Switching a person between treatment and holdout makes the comparison harder to interpret.
3. Decide what the holdout will not receive
For a test of one automation, suppress that automation for the holdout while keeping other marketing activity as consistent as possible across both groups. A storewide “no email” holdout asks a broader question: What is the combined effect of your email program? It does not isolate the value of one flow.
Check whether customers can enter other automations during the test. If an abandoned-cart holdout receives a similar browse-abandonment message, the measured difference may understate the effect of cart recovery messaging overall. You may choose to suppress overlapping flows or leave them running, but document the choice and interpret the result accordingly.
4. Set a measurement window that fits the buying cycle
Choose a fixed period after each customer becomes eligible—for example, seven days for a short cart-recovery sequence or a longer period for a considered purchase. Use the same window for both groups, and allow every enrolled customer to complete it before finalizing the result.
Define your revenue rules in advance. Will you count orders placed during the window, net of cancellations and refunds recorded by a specified cutoff? Will you include shipping and tax? Consistency matters more than choosing a universally “correct” revenue definition. If returns commonly arrive later, report the initial result as provisional and revisit it when those adjustments are available.
5. Run the test long enough to make the result useful
A handful of orders can produce a large-looking percentage difference by chance. Before launching, estimate how many eligible customers the flow receives and what size of revenue lift would be meaningful to your business. If you have access to an experiment calculator or analytics support, use it to plan sample size and assess uncertainty.
For a lower-volume store, the practical choice may be to test a higher-traffic flow, extend the enrollment period, or accept that the result will be directional rather than conclusive. Avoid repeatedly checking the dashboard and ending the test the first day the result looks favorable.
How to calculate incremental revenue from an email automation
Compare revenue per eligible customer, not total revenue, because random groups may differ slightly in size.
Incremental revenue per eligible customer = (treatment revenue ÷ treatment customers) − (holdout revenue ÷ holdout customers)
Suppose 1,000 eligible shoppers are assigned to each group. Over the same seven-day window, the treatment group produces $60,000 in qualifying revenue and the holdout produces $54,000:
- Treatment revenue per eligible customer: $60,000 ÷ 1,000 = $60
- Holdout revenue per eligible customer: $54,000 ÷ 1,000 = $54
- Estimated incremental revenue: $60 − $54 = $6 per eligible customer
In this example, the observed incremental revenue across the 1,000 treatment customers is $6,000. That is not necessarily incremental profit. If the automation offers a discount, account for the discount, product costs, and other relevant costs before deciding whether the flow is financially worthwhile.
Also inspect the uncertainty around the estimate. An observed $6 difference does not prove the true effect is exactly $6; it could partly reflect chance, particularly when order values vary widely. Report the group sizes, revenue definition, test dates, and measurement window alongside the result.
Common mistakes that make holdout results misleading
Counting only email-attributed orders
A customer may read an email and buy later through a direct visit, or buy without seeing the email at all. Comparing all qualifying orders between randomly assigned groups captures these outcomes more fairly than comparing click-attributed revenue alone.
Comparing recipients with non-recipients
Customers who received an email may differ from those who did not because of consent, deliverability, or the rules that triggered the flow. That is not a reliable holdout. Compare customers assigned to treatment and holdout under the same eligibility rules, even if some treatment customers never open—or never receive—the message. This measures the impact of running the automation as it operates in practice.
Changing other marketing during the test
A major sale, new discount, or campaign sent unevenly across groups can cloud the result. You do not need to stop normal trading, but both groups should have the same opportunity to encounter other promotions. Keep a record of unusual events that might affect interpretation.
Extrapolating too confidently
You can multiply an estimated lift per eligible customer by expected future eligibility volume to build a forecast. Treat that number as a scenario, not a guaranteed annual return. Customer mix, seasonality, inventory, and promotions can all change the automation’s effect.
What should you do with the result?
If the treatment group clearly outperforms the holdout, keep the automation running and look for ways to improve it without sacrificing the value you measured. Test one meaningful change at a time, such as the offer, timing, or message sequence.
If the groups perform similarly, do not assume the flow is useless based on one small test. Check whether the test ran long enough, whether another message reached the holdout, and whether the measurement window suited the product. If the result remains flat after a sound test, consider simplifying the flow or using that effort elsewhere.
A practical first step is to choose your highest-volume automation and write a one-page test plan: who qualifies, how customers are assigned, what the holdout misses, which revenue counts, and when you will review the outcome. That gives you a decision you can trust more than an attributed-revenue dashboard alone.
Veelgestelde vragen
How large should an email holdout group be?
There is no single percentage that works for every store. A larger holdout can produce a clearer comparison but means withholding the automation from more customers. Base the split and test duration on your eligible audience size, typical purchase rate, and the smallest effect worth detecting.
Can I run a holdout test on an abandoned-cart flow?
Yes. Randomly assign shoppers when they meet the flow’s eligibility criteria, suppress the cart sequence for the holdout, and compare qualifying store revenue over the same window after eligibility. Account for other recovery messages that either group might receive.
Should I measure revenue or profit?
Measure revenue if your immediate question is whether the automation generates additional sales. For a decision about discounts or investment, also estimate incremental contribution after relevant costs. A flow can increase sales while delivering less financial value than its revenue figure suggests.
How is a holdout different from an A/B test?
A typical email A/B test compares two versions of a message, such as different subject lines. A holdout compares access to the automation with no access to that automation. Use an A/B test to improve a flow’s execution; use a holdout to estimate whether the flow adds value at all.