Shopify A/B Tests: How Much Traffic?

20,000 store visits a month sounds promising. But if just a quarter of them reach the product page you're changing, the test has a much smaller audience. Plan around eligible, uniquely assigned visitors.

A worked example

Suppose 2.0% of visitors to the page currently order. Reaching 2.4% would make the change worth keeping. That's 0.4 percentage points, or a 20% relative lift. Assume two equal groups, a two-sided comparison of proportions, 5% significance and 80% power.

A standard normal approximation calls for about 21,100 people per group, or 42,200 overall. With 1,000 eligible new visitors a day, that's roughly six weeks. With 100 a day, it's over a year. This is an illustration, not a forecast for your store.

What can change the estimate?

A returning visitor mustn't count as a new person each time. A sale or a tracking change mid-test complicates the analysis. Detecting a smaller difference requires more data. Also check that the actual allocation roughly matches the split you planned.

What will you do with the result?

Write down in advance what you'll do with a positive, negative or inconclusive result. If the experiment would run too long, observe customers trying the same task instead. If a link is broken, fix it.

The SWISSpur test shows how a cart signal can differ from a sales result. See how I plan tests for your own store.