Glossary
What is A/B Testing?
A/B testing is a controlled experiment comparing two versions of an ad, page, or message that differ in exactly one variable, run simultaneously to determine which performs better against a defined metric.
The discipline in A/B testing is isolating a single variable — one headline, one image, one call-to-action — so that whatever difference shows up in the results can be attributed to that one change rather than several changes happening at once. Testing a new image and new copy in the same variant tells you the pair won or lost together, not which part did the work.
A valid test also needs a large enough sample and long enough runtime to reach statistical significance before a winner is declared — stopping early because one variant is ahead after a few dozen conversions is one of the most common ways tests produce false winners. Ad platforms' delivery algorithms add another wrinkle: if two variants sit in the same ad set, the delivery system can start favoring one before enough data exists to know if that's genuinely the better performer, which is why isolating variants into separate ad sets is standard practice for a clean test.
Outside of ads, the same logic applies to landing pages, email subject lines, and pricing pages — anywhere a measurable action (click, purchase, signup) can be compared between two versions shown to comparable audiences.