How to test Facebook ad creatives so the winner means something

Most creative tests never produce an answer. Not because the creatives were bad: because the test could not answer anything. Four things changed at once between the variants, the result was read on day three, and the winner was picked on click-through rate.

Decide what the test is for before you build it

Two different jobs get called creative testing. Restocking: the current ad is fatiguing, you need something running by Friday, and why the replacement works does not matter. Learning: an answer that carries over to the next ten ads. Learning is the expensive one, and it pays off only if exactly one thing differs between the variants.

One hypothesis at a time is what makes an answer reusable. Change image, headline and offer together and the winner teaches you nothing: you would have to run it again to find out which part did the work. Name the variable in a sentence. If that sentence needs the word "and", you have two tests. Then pick which layer to change. The order below is by effect size, which is the same as by how cheap the answer is to detect.

1. 1.The offer. What the person gets and what it costs them. This changes who is interested at all, so the effect can be large enough to see on a small count.
2. 2.The angle. Same offer, different reason to care: price, speed, risk, one specific problem. Answers here carry across formats, which is what makes them worth paying for.
3. 3.The format and the hook. Video against static, the first three seconds, a talking head against a screen recording. Narrower, because what is being sold has not changed.
4. 4.The execution. Font, color, button wording, which stock photo. If the effect here is smaller than the gap your conversion count can resolve, no budget you would sensibly spend can measure it.

The noise floor decides what you can test

Conversions arrive as counts, and counts wobble. A variant that produced 25 conversions sits on a true rate that could as easily have produced 20 or 30, with nothing about the ad changing.

The relative noise on a count of n is roughly 1 divided by the square root of n: 20% at 25 conversions, 10% at 100. Comparing two counts is noisier than either one alone, by about the square root of two, and a gap worth acting on runs to roughly twice that combined figure. That gives a table nobody enjoys.

These are approximations from counting noise, not a significance test, and they assume both variants got a fair share of delivery. Ten conversions against eight is the same number twice.

What an honest test costs, in dollars and days

The budget is calculated, not chosen: conversions per variant from the table, times the number of variants, times what a conversion costs you today. Two variants at 50 conversions each, at a cost per lead of $20, is $2,000 for one question. If the answer is uncomfortable, cut variants or optimize for a cheaper event, never the sample.

Time works the same way: you test until a count, not until a date. Days only tell you whether the count is reachable.

• –Run at least a full week. Weekday and weekend behavior differ, and a test running Thursday to Monday compares two different mixes of days.
• –Add your attribution window before reading. On a 7-day click setting, conversions keep landing against older clicks, so a variant that attracts slower buyers looks worse on day three than on day ten.
• –Do not read it daily. At small counts one variant is almost always ahead by day two, and that lead is usually just the order the conversions arrived in. Every look builds the case for an edit that ends the test.

Why CTR is the wrong scoreboard

Click-through rate is tempting because it settles first. Every conversion sits behind a click and only some clicks convert, so the click count is always the larger of the two and steadies while the conversion column is still noise. The metric that looks decidable is the one that does not answer the question.

• –CTR measures the ability to earn a click, not a customer. A curiosity hook collects clicks from people who only wanted to know what it meant.
• –The creative changes who clicks, so it changes every rate after the click. Two ads with the same CTR can convert at different rates on the same page, and the two metrics can rank the same pair differently.

Judge on cost per result, or on cost per acquisition if your CRM records which leads closed. When the two disagree, the money is on cost per result. CTR has a later job: once a variant loses, it says whether the ad failed at the impression or after the click. Use link CTR for that, not CTR (all).

Structure the test, then stop touching it

Adding a new ad to a running ad set restarts that ad set's learning, not just the new ad's. So testing inside a campaign you were happy with costs a stretch of unstable delivery on the part that already worked. Two structures handle that differently.

• –All variants in one new ad set. Cheap and fast, and Meta decides. Delivery concentrates on whatever gets early traction, so one variant can finish the week with a readable count and another with almost nothing. A running winner, not a comparison, which is right for restocking.
• –One variant per cell with an even split, through Meta's A/B test tool. The audience is divided so nobody lands in both cells, and every cell has to reach the count on its own. Slower and more expensive. This is the structure that produces a lesson.

Whichever you pick, freeze everything else for the duration: no budget changes, no audience edits, no fifth ad on Wednesday. If changes are queued, put them in together at the start.

A variant that took most of the delivery did not beat anything. Before comparing two costs per result, check that the impressions and the spend behind them are in the same neighborhood.

Keep a control, and fix the read date before launch

A control is the incumbent creative, left running unchanged for the length of the test. It costs budget and it is the only thing separating your result from the weather: CPM moves with the season and with whoever else bids for your audience, so replacing everything at once and watching cost per result improve tells you nothing about which of the two moved. A test week against last month is not a comparison.

Write down the date you may read the result and the count each variant needs by then. Both are easy to set while nothing is at stake, and impossible to set fairly once one variant is ahead.

• –Cut early only on a floor set in advance: some multiple of your target cost per result spent with zero results. Two or three times is a working rule, not a number Meta publishes. It stops spend, it does not judge the creative.
• –Otherwise wait for the count, not the date. A test that reaches its read date with 12 conversions per variant has not failed, it has not finished. Extend it, or accept that this question is out of budget.
• –After cutting the loser, leave the survivor alone for a few days. Pausing an ad pushes its delivery onto whatever remains, so the survivor's numbers move for reasons that have nothing to do with the creative.

Two gates decide whether a comparison is worth reading: whether the variants ran on the same terms, and whether the counts behind them are large enough. Aevin's variant view applies both, comparing creatives only inside one ad set, and declining to name a winner while the ad set is under 30 leads or the gap between best and worst cost per lead is under about 1.25x. Working thresholds, not laws.