Product Collective - 😕  Failing AB tests

Is it possible to fail an A/B Test?

We’re trained from our early days in school to want to pass tests and avoid failing a test. That makes sense when the purpose of a test is to measure your knowledge. But that’s not what A/B tests are for. They help you determine which change to select. With that in mind, what does it mean to fail an A/B test? Is that even possible? Yes, if you consider a failed A/B test as one that doesn’t help you make an informed decision.

5 Reasons your A/B test fails (and how not to). According to a VWO study, 1 in 7 A/B tests will bring you great conversions. That doesn’t mean you should stop doing A/B tests. It means you should look at how you perform your A/B test and identify ways to change your approach. Aditi Gupta explains 5 common reasons your A/B tests fail and how to run a winning A/B test.
(via @VWO)

12 A/B Testing mistakes. A/B testing is fun. With so many easy-to-use tools, anyone can—and should—do it. However, there’s more to it than just setting up a test. Tons of companies are wasting their time and money. Peep Laja discusses the 12 A/B test mistakes that people frequently make. Read through the article to find out how many you’re making.
(via @cxldotcom/)

Continues below...

It's OK to fail as long as you're learning

If there’s anything I’ve ever learned in life, it’s that you’re going to make lots of mistakes. I make a lot of mistakes – just ask my wife. And while I’d like to think that the more experience I’ve accumulated has made me wiser – whether it’s as a product person, an entrepreneur, or heck, even a dad and husband – I still continue to make mistakes in almost every category to this day. 

And I’m OK with that.

It’s not whether or not a person’s made mistakes that matters – it’s whether they’ve learned anything from those mistakes. The best entrepreneurs and product people will be the first to tell you that they’ve taken big risks… and not all of them have paid off. But the fact that they weren’t afraid to take those risks in the first place is a key trait that likely helps them stand out among their peers. This is probably why Steve Jobs once said, “Sometimes when you innovate, you make mistakes. It is best to admit them quickly, and get on with improving your other innovations.” In fact, even Jobs has taken some very big risks and made some big mistakes throughout his career. 

When scientists conduct experiments, failure isn’t necessarily failure. It’s disproving a hypothesis. And scientists see a lot of value in failure, as there’s so much to learn from it. 

So when you’re thinking about the experiments you’re trying to make as a product person – don’t be afraid to be bold. Don’t be afraid to do something that pushes the envelope. It’s okay to experiment and fail, but be sure that you’re learning along the way. It’s that knowledge that will help you lead to real innovation. 

INDUSTRY: The Product Conference is back with a bang! The in-person edition returns on September 19-21 in Cleveland, Ohio, reviving the top independent product conference anywhere. Build your network and find inspiration as you'll learn from some of the best -- like David Cancel (CEO at Drift), Sarah Butterfass (CPO at FanDuel), and even Mark Duplass (award-winning filmmaker and co-star of Apple TV's The Morning Show).

What to do when there’s not enough traffic for a/b testing. So, you are dying to run an A/B test on that one site/product/app/feature you think you can improve. But the user traffic is so low that it seems to be statistically unfeasible to run the test. An A/B test needs a minimum number of participants — or traffic — in order to achieve reliable results. Otherwise, you may end up drawing conclusions based merely on chance. Rafael Damasceno explains what you can do when you try to run an A/B test with a low number of users.
(via @SeerInteractive)

When A/B Testing Doesn’t Tell You the Whole Story. In order to prevent churn, marketers traditionally start by identifying which customers are most likely to churn, and then run A/B tests to determine whether a proposed retention intervention will be effective at retaining those high-risk customers. While this strategy can be effective, Eva Ascarza shares new research that suggests it isn’t always the best way to maximize ROI on marketing spend. Instead, Eva argues firms should use A/B test data alongside customers’ behavioral and demographic data to determine which subgroup of customers will be most sensitive to the specific intervention that’s being considered. 
(via @HarvardBiz)

The top 3 mistakes that make your A/B test results invalid. When it comes to testing, too many companies worry about the “what”, or the design of their variations, and not enough to worry about the “how”, the execution of their experiments. James Flory shares the biggest mistakes you can make within each step of the testing process: the design, launch, and analysis of an experiment, and how to avoid them.
(via @widerfunnel)

The Customer Analytics Experience

Let's face it — making your way into the world of product management probably wasn't so easy. Very few of us went to school for product management — and most of us probably feel like we simply "fell into" the profession. But what if it could be easier for those breaking into product? As product people today, what can we do to play our part in helping pave the way for others? And once we've helped people find their way to our own organizations… what else can we do to help empower them?

We'll dive into all of this and more with Seth Hutcheson, Senior Product Manager at insightsoftware — with plenty of time for you to ask questions of your own as well!

This webinar is sponsored by our friends at Logi Analytics.

1. A newsletter each and every Friday centered around an important product topic, like roadmapping, strategy, and analytics.
2. Access to a thriving Slack group with thousands of others to share advice, tips and get feedback on your work.
3. Plus, invitations to exclusive video chats, special member privileges at the conferences we hold in the United States and Europe, and more!