Question 1

How long should an ecommerce A/B test run?

Accepted Answer

An ecommerce A/B test should run until it reaches statistical significance (typically 95% confidence) AND for at least 7 days to capture day-of-week variation. For a store with 1,000 daily visitors and a 2.5% conversion rate testing a 10% improvement, you need roughly 14 days.

Question 2

What sample size do I need for an A/B test?

Accepted Answer

Sample size depends on your baseline conversion rate, minimum detectable effect, and desired statistical significance. For a typical ecommerce test (2.5% conversion, 10% MDE, 95% confidence), you need approximately 7,000-8,000 visitors per variant.

Question 3

What is minimum detectable effect (MDE) in A/B testing?

Accepted Answer

MDE is the smallest relative improvement you want your test to reliably detect. A 10% MDE means you want to detect if the variant improves conversions by at least 10% (e.g., from 2.5% to 2.75%). Smaller MDEs require much larger sample sizes and longer test durations.

A/B Test Duration Calculator

Your test parameters

Get your A/B testing playbook