Your ad test is probably reporting winners that do not exist

Responsive search ads mean most ad groups are running an unsupervised multi-variant test whether you set one up or not. When a tool tells you variant B won, it is worth asking what "won" means.

The arithmetic

A significance test at the conventional 5% threshold is a statement about one comparison: if the two arms are genuinely identical, there is a 5% chance of seeing a difference this large by luck.

Now run five variants and report the best one. You are no longer making one comparison — you are making several, and then selecting the most extreme result. The probability that at least one of them clears the threshold by chance alone, when all five are truly identical, is far higher than 5%.

For five independent arms, the probability of at least one false positive at a 5% per-comparison threshold is:

1 - (0.95)^k

At k = 5 that is about 23%. Nearly a quarter of your "winners" across identical ads are noise. In practice the arms are not independent and the selection is adaptive, which makes it worse, not better.

Apply this across a few hundred ad groups every month and you are systematically pausing ads for no reason, then reporting the churn as optimisation work.

Three other ways ad tests go wrong

Rounding at the boundary. Math.round(0.5) is 1 in most languages. If a confidence calculation rounds at the decision boundary, two genuinely identical arms can be reported as significantly different. This is a one-line bug that produces confident, wrong answers at scale.

One-sided tests presented as two-sided. If you are asking "is B different from A", that is a two-sided question. Using a one-sided threshold roughly doubles your false positive rate, silently.

Volume dressed up as confidence. Some tools display a "confidence" figure that is really a function of impression volume. High volume and a real difference are not the same thing, and a number labelled confidence should mean confidence.

What a defensible test looks like

• Correct the threshold for the number of variants actually being compared.
• Use a two-sided test unless you have a genuine directional hypothesis stated in advance.
• Report inconclusive as inconclusive. "Not enough evidence yet" is a real, useful answer and it is the correct answer most of the time.
• Never round at the decision boundary.

The uncomfortable implication is that a rigorous ad testing tool will declare far fewer winners than a sloppy one. That is the point. A tool that finds a winner in every ad group every month is not finding winners.