How to Prove Your DCO Actually Works (The Holdout Test Most Vendors Avoid)

The short answer: Run a controlled 50/50 holdout — same budget, same period, one group on your current setup and one on the new creative — and measure a real business outcome like first-time buyers, not platform CTR. The gap between the two groups is the only honest measure of impact. AI-native DCO is built to be tested this way; most “AI creative” tools quietly avoid it.

If a creative vendor won’t agree to a holdout, that tells you something. Here’s how to run one, and what to measure.

Why before-and-after lies to you

The most common way teams “prove” a new creative tool is before-and-after: turn it on, watch the numbers, take credit. The problem is everything else moved too — seasonality, promotions, audience fatigue, platform algorithm shifts. You can’t separate the tool’s effect from the noise. Before-and-after isn’t measurement; it’s a story.

The fix: a controlled 50/50 holdout

Split one ad account’s budget in half, running both arms at the same time:

• Control group: your existing setup and creative, untouched.
• Exposed group: the same budget, same period, same audiences — running the new (AI-native) creative.

Because both run in parallel under the same conditions, the difference between them is the incremental impact of the creative — not seasonality, not luck.

What to actually measure

Vanity metrics (impressions, even CTR) can move without moving the business. Measure the chain that ends in revenue:

1. CTR — the first signal that the message matches the audience.
2. Conversion rate — intent and availability (is the product actually in stock?).
3. Cost per acquisition / cost per first-time buyer — efficiency.
4. ROAS — the bottom line.

Make the primary metric a real business outcome — first-time buyers, new customers, qualified leads — and treat platform metrics as supporting evidence. A creative system that lifts new-customer acquisition at a lower cost is winning, full stop.

Expect a learning curve (and don’t kill it early)

Here’s the part teams get wrong: an AI-native system learns, so for the first week or two it may trail a mature, already-trained setup while it builds its model. If you end the test in week two, you’ll “prove” it doesn’t work — right before it crosses over. Run a real window (six to eight weeks), and watch the trend, not just the first data point. The honest story is usually: learning curve, crossover, then sustained lead.

Why AI-native DCO welcomes the test

Legacy DCO and generic GenAI tools tend to avoid holdouts because their value is hard to isolate. AI-native DCO is the opposite — it’s designed to be measured, because the model optimizes against the exact outcome you’re testing. Continuum has run this controlled, head-to-head structure against platform-native automation, with first-time buyers as the scorecard — not a demo, a measured test.

If you’re evaluating any creative-AI vendor, make the holdout your first question. Book a demo and we’ll design the test with you.

FAQ

What is a DCO incrementality test? A controlled experiment — usually a 50/50 budget split running in parallel — that isolates how much a creative system adds versus what would have happened anyway, by comparing an exposed group to an untouched control.

How long should a DCO holdout run? Long enough for an AI model to get past its learning phase and reach steady state — typically six to eight weeks. Judging it in the first week or two penalizes the system right before it pulls ahead.

What’s the right primary metric? A real business outcome (first-time buyers, new customers, qualified leads), with CTR, conversion rate, CPA, and ROAS as supporting evidence.