Wednesday 22 January 2014

Why you must watch this video before doing A/B Tests

Posted by Unknown at 1/22/2014 Labels: , , ,
When it comes to A/B testing, there is a lot of content on the interweb. Like everything else about 90% of this is just total shit, from people that don't really know what they are doing, and totally miss the point.

Step in Jason Cohen!

Jason does a brilliant job of describing why so many business get it wrong when it comes to A/B split testing (if you don't know what A/B testing is check out this article).

If you are doing A/B testing, and you do only one thing today, watch this video!!! Yes it is almost 1 hour long. However, 1 hour is less time than 1 year, which is the time you will probably waste doing tests, especially if you are new to this stuff. 

After watching it clock-off work, go home, put on your slippers and get a cup of hot chocolate... Hell, put on a Michael Buble CD and relax. You have just saved yourself about 12 months of pointless work. 

Here is a quick summary of probably one of the most informative talks on the internet. To be fair nothing Jason says is new, its just simple statistical analysis. However, too many "growth-hacking" hipsters forget to take Data Science 101 and so they are miss something important in their mysterious and elusive hack-kit.

1. Have a testable hypothesis,
2. Keep A/B tests simple,
3. Avoid 47 Shades of Blue  - see point 1.
4. Ensure you have a decent enough sample size - no 100 visitors is not enough.
5. Have a high "confidence" level. 95% confidence is acceptable, anything less is not.
6. Small gains (1% - 15 %) are not enough to justify change especially if you are a low volume (< 1million visits/month) /a start up. Look for changes that double something or increase by 1/3 etc.
7. Check for statistical significance using this method.
8. Run the test for an appropriate amount of time. 
9. Find 1 - 3 business critical "metrics" to measure and focus on them. Jason gives a broken down example. 

Watch the video here:

If you like Jason's talk then you can find more at his blog:
http://blog.asmartbear.com/
Twitter: @asmartbear

No comments:

Post a Comment

Subscribe to: Post Comments (Atom)