Product Collective - ⚖️ Running product experiments

Running Product Experiments

Let’s say you’re trying to optimize your product to increase the retention of existing customers. You could have endless rounds of conversation amongst your product team, or you could prompt your team by saying “I don’t know, let’s go find out.” That call to action is a great way to get your team into experimentation mode.

3 lessons from running experiments as a Product Manager. Experiments are an important part of the build-measure-learn loop. After making assumptions about how our users behave, finding out the truth can be incredibly humbling. Ivan Teong has learned so much from shipping experiments to Jira Software in the last few years. He believes that this learning journey will continue for many years to come and he wants to share his learnings with you, hoping it will help you experiment.

(via Ivan Teong)

6 In-Product experiment types for continuous discovery. What comes to mind when you think about experiments? Is it a professor in a lab playing with test tubes and Petri dishes? Maybe it’s creating potions as a kid with the contents of your parents’ bathroom cabinet? Or is it product teams testing feature changes to find the best solution? These answers are valid, but Ray Slater Berry talked about experiments to optimize your product. They enable you to test new feature designs and ideas to determine their effectiveness. So, put down your Petri dish and step away from that cabinet—it’s time to talk product experiments (lab coats optional)

(via Chameleon)

Continues below...

Hold on to your keyboards, folks, because Twitter is in for a shake-up! Elon Musk just dropped a bombshell tweet announcing that he’s finally hired a new CEO for Twitter. He mentioned that she starts in less than 6 weeks, so we’ll see a new Twitter chief take the reins this summer! 

But who will it be? Well, we have our guesses…

First up on our speculative list is Linda Yaccarino, NBCUniversal's head of advertising. In fact, this seems to be the runaway favorite as she’s already stepped down from her current post. Yaccarino is known for her strategic vision and no-nonsense approach. With her expertise in advertising, she could bring a fresh perspective to Twitter.

Of course, Musk has been known to surprise people – so if we’re going to explore any surprise moves here, then Laura Desmond, former CEO of Starcom Mediavest Group, could be another intriguing choice. Desmond's extensive experience in media and advertising would certainly be of interest to Twitter. She certainly has the gravitas, too, serving on major Boards at places like Adobe and DoubleVerify. That said, she just became CEO at Smartly last year… so it may not be likely to see her make a change this soon. 

And if we were really looking for a head-turner, could Sheryl Sandberg, Facebook's Chief Operating Officer, be poached away from the social media giant? Sandberg is known for her influential role in shaping Facebook's success. With her expertise, she could bring stability and strategic direction to Twitter.

In all likelihood, it seems like it’s a foregone conclusion that Yaccarino is the person who we’ll see become the head of Twitter in the next few weeks. Regardless, as the countdown begins to the new CEO's arrival, I’m sure we’ll see the Twitterverse buzzing with anticipation. And who knows… maybe we’ll all be surprised after all!

Pitfalls in product experimentation. We all know product experimentation is important, and its benefits have largely been proven by organizations, enabling data-driven decisions on products, features, and processes. However, product experiments, like any other statistical testing or experimentation, are prone to pitfalls. These are design and/or execution flaws, which you may not think of while you’re experimenting. Olivia Tanuwidjaja explains that it’s important to guardrail experimentation execution and analysis to get reliable results. And hence it is important to understand the common pitfalls and how to treat them, as they might mislead your analysis results and conclusion.

(Via Olivia Tanuwidjaja)

How product teams can build empathy through experimentation. Travis Brooks, Netflix Product Manager for Experimentation Platform, and George Khachatryan, OfferFit CEO sat down at George’s office (virtually) to extend their conversations about the philosophy of experimentation. They had a fun conversation and realized that some parts of it might make a good blog post as well. So they jointly edited a bit for length and clarity. 

(via Travis Brooks)

Are you running your product experiments with the right mindset? Running experiments to learn and make improvements in product development has become a common approach adopted by most of the teams Yuan Ng has worked with at Xero, and something that offers enormous value. It’s a phenomenon that is very much influenced by the increased availability of data, which allows the teams at Xero to make decisions by combining this data-driven insight, with customer insight. What this means is that they’re now able to see what actions people are taking in the product experience, and their correlation with Xero’s business goals. However, sometimes it’s easy to fall into the trap of wanting to “test and win” instead of “test and learn” when running experiments. Having done multiple mistakes in experiments as a product designer, one thing that Yuan learned is to conduct experiments with the right mindset from the get go, so you can achieve the right outcome: to learn and improve.

(via Yuan Ng)