How Many Ad Variations Do You Actually Need to Test

How many creative variations a video campaign genuinely needs, which variables are worth varying, and how to test without wasting the budget on noise.

Creative testing is the part of video advertising where the largest performance gains are available and the smallest share of the budget is usually spent. Most campaigns produce one film, run it, and conclude something about the audience from a result that was actually determined by a single creative choice nobody tested. The question of how many variations to make has a practical answer, and it depends on which variables genuinely move performance.

The variable that moves performance most is the opening. In a scrolling feed the first two to three seconds determine whether anything else is seen, and differences between openings are routinely a factor of two on early retention. This is why the highest return testing structure is not several different films but one film with four different openings. The production cost of an alternative opening is small, and the performance spread is large.

The second variable is the format and aspect ratio, and it is frequently confounded with creative quality. A campaign that runs a horizontal film vertically and concludes the creative underperformed has measured its own cropping decision. Testing formats requires each to be composed natively, which is a production decision made at storyboard stage rather than an export setting chosen at the end.

The third variable is the call to action and the framing of the offer, which is cheap to vary because it usually lives in the last few seconds and in the on screen text. The fourth is the audio bed, which is worth testing less often but occasionally produces surprising differences. Beyond these, variations tend to produce noise rather than information, because the differences between them are smaller than the natural variance in delivery.

A workable structure for a campaign with a normal budget is four openings against one body, run simultaneously, judged on three second retention rather than on conversions, for long enough to gather meaningful data. The winning opening then runs with two or three variations of the closing. This produces genuine learning from roughly six to seven assets, most of which share the majority of their production cost.

The evidence on format sensitivity supports testing rather than assuming. Frade et al. (2023) found that in stream ad placement and format materially affect visual attention and effectiveness, and Davtyan et al. (2025) compared skippable ads, non skippable ads and brand placements on YouTube and documented differences between these strategies. These are not marginal effects, and they are not reliably predictable in advance for a specific brand and audience.

Yin et al. (2023) add an important caution: skippable advertising influences advertising avoidance intention, meaning that a poorly matched creative does not simply underperform, it can generate active avoidance that carries over. This is the argument against running a large number of untested variations at scale. Testing should happen at low spend, and only the winners should receive significant delivery.

Dong et al. (2026), examining engagement toward short form videos on Douyin, found that engagement drivers vary by brand type, which is a useful corrective to universal creative formulas. What works for a consumer brand with high emotional engagement will not transfer to a technical B2B product, and a testing programme is how a specific brand discovers its own pattern rather than inheriting someone else's.

The measurement discipline matters as much as the variation count. Judging creative on final conversions requires far more data than most campaigns have, and it conflates creative performance with targeting, landing page and offer. Judging on early retention isolates the creative variable and produces usable answers quickly. Kim et al. (2025) found that playback interaction behaviour carries information that aggregate view counts obscure, which supports looking at the retention curve rather than the headline number.

The most common practical mistake is testing too many things at once with too little spend behind each, which produces a set of statistically meaningless results that are then treated as findings. Fewer variations, each with enough delivery to be judged, tested on one variable at a time, produces knowledge that compounds across campaigns. A brand that has run this properly for a year knows what its opening should look like, which is worth considerably more than any individual campaign result.

References

Frade, J. L. H., Oliveira, J. H. C. de, & Giraldi, J. de M. E. (2023). Skippable or non-skippable? Pre-roll or mid-roll? Visual attention and effectiveness of in-stream ads. International Journal of Advertising, 42(8), 1242–1266. https://doi.org/10.1080/02650487.2022.2153529

Davtyan, D., Tashchian, A., & Thomas, M. L. (2025). A comparative analysis of skippable ads, non-skippable ads, and brand placements: Evaluating YouTube advertising strategies. Journal of Advertising Research, 65(3), 464–478. https://doi.org/10.1080/00218499.2025.2464276

Yin, S., Li, B., & Zhou, Q. (2023). The impact of skippable advertising on advertising avoidance intention in China. Marketing Intelligence & Planning, 41(8), 1121–1137. https://doi.org/10.1108/MIP-07-2022-0298

Dong, X., Xie, J., Xi, N., Liao, J., & Xie, S. (2026). What leads to consumer engagement towards short-form videos on Douyin? The moderating role of brand type. International Journal of Advertising. Advance online publication. https://doi.org/10.1080/02650487.2026.2670858

Kim, E., Oh, S., & Park, S. (2025). An empirical study of user playback interactions and engagement in mobile video viewing. IEEE Access, 13, 78272–78289. https://doi.org/10.1109/ACCESS.2025.3566402