Figured out my A/B test sample sizes were way off after 3 failed campaigns

I ran 3 Facebook ad tests with only 200 people each and got totally useless results, then a friend pointed out I needed at least 1,000 visitors per variation to actually see a real difference has anyone else made this mistake with their split testing?

2 comments

2 Comments

drew_park29d ago

Oh man, that hurts to read because I've been there. I remember reading some marketing blog that said you need a certain number of people to make the data actually mean something, but I ignored it and ran a split test with like 150 people per group. Surprise, surprise, the results were all over the place and I couldn't tell if the green button actually worked better or if it was just random luck. I think the rule of thumb I saw was something like 300-500 people for strong results on a basic test, but for ads or landing pages where the difference might be small, you need way more. It's a bummer because it means you have to spend more time and money upfront just to know if your test is even valid.

milestaylor28d ago

And that's the thing people don't realize about split testing... it's not really testing if you don't have enough data for the math to work. I read somewhere that the sample size you need depends on how small the difference is you're trying to catch, so if you want to know if a 5% lift is real you need way more people than if you're testing a 50% change. Most small tests are just measuring random noise instead of actual results... like flipping a coin 20 times and thinking heads came up more because it's lucky. The worst part is spending all that time setting up tests and then having to throw the results away because you can't trust them.