Dear subscribers,

Today, I want to talk about how you can run experiments that don’t suck.

Eric Metelka is head of product for Eppo where he has a broad view of how fast growing companies run experiments. I spoke to Eric about:

1. When to run experiments

2. How to run experiments that don’t suck

3. How to introduce experimentation to your company

This post is brought to you by…Eppo

I've run 100s of experiments in my PM career so I know how painful it is to build a great experimentation platform in-house.

Eppo is the only 3rd party platform that's built on top of your company's data warehouse. This lets you track critical server-side metrics in real-time without having to share data externally or implement client events. Give it a try for free below!

Try Eppo for Free

When to run experiments

Welcome Eric! How do you decide if you should run an experiment or not?

You should run experiments if you need to…

1. Prove causality. We’ve all attended meetings where people question if a product moved a metric. An experiment removes this uncertainty.

2. Measure impact. For example, you may run an experiment that grows engagement by 3% but hurts revenue by 1%. Measure impact to make decisions.

You should not run experiments if you need to…

1. Make a decision quickly. Experiments usually take time and work to run. They shouldn’t be used to make every product decision.

2. Bet on a 0-1 product. If you’re building something new, you might have too few users or not enough data to run an experiment.

3. Do a big launch. If you need to make a big public launch, it’s hard to experiment.

How do you run an experiment?

There are 5 steps to running an experiment:

1. Craft a hypothesis. e.g., If we do this, then this metric will grow by X%.

2. Define output, input, and guardrail metrics.

3. Design the experiment.

   1. Eligibility: e.g., Users who visit the onboarding flow

   2. Test group: e.g., New onboarding experience

   3. Control group: e.g., Existing experience

   4. Duration: e.g., 2-4 weeks 

4. Run the experiment.

5. Share your findings. e.g., Metric had X% lift so we will roll out to 100%.

What are output, input, and guardrail metrics?

Your output is the key metric that you want to move (e.g., nights booked for Airbnb). Your inputs are leading indicators of the output (e.g., percent that complete user onboarding). Your guardrails are high level metrics or adverse effects to monitor (e.g., overall revenue and number of support tickets).

Track all 3 types of metrics to make sure your experiment is having the right impact.

How to run experiments that don’t suck

What are some best practices for running experiments?

Three come to mind:

1. Be precise about your outputs, inputs, and guardrails. Don’t try to move too many output metrics and be sure to include inputs and guardrails.

2. Estimate how much the output metric will move. This helps you determine if you have enough users to achieve the desired impact (e.g., sample size calculator).

3. Check your experiment's health. Check how many users are in the test and control groups and whether your numbers make sense before and after launch. A bad experiment can easily burn a month of your team’s time.

What are some effects that may dilute your experiment results?

There are two in particular:

1. Novelty effect is when users engage more with a product in the beginning. To avoid this, run the experiment at least 2-3 weeks before drawing any conclusions.

2. Interaction effect is when one experiment starts impacting the results of another. To avoid this, test different hypotheses and try to space your experiments out.

Some teams maintain a long-term holdout (users that don’t get any product updates) to avoid the novelty effect. Is this necessary?

Holdouts work best at large companies with mature data practices. 

They take a lot of work and your user experience could also suffer. I think 2-4 weeks is good enough to get a read on most experiments without needing a holdout.

There’s also the interaction effect from having a two-sided marketplace.

Yes, in marketplaces, a change in one side could impact the other side. That’s why guardrail metrics are so important.

At my previous company Cameo, we rarely ran experiments for the supply side (creators) due to small sample size. Most of our experiments were on the demand side (users), like optimizing the onboarding experience.

A common criticism of experimentation is that you end up optimizing for a local maxima vs. make bold new bets. Any thoughts on that?

You could test bigger product changes to avoid the local maxima.

These big bets are unlikely to show positive results at first because users need to get adjusted. But if you have confidence that the change is good for users long-term, then you can just ship it and then optimize through experiments.

Do you have an example of a small product change that had an outsized impact?

Yes. One thing that always seems to work is to add trust and safety icons to a product’s checkout process.

Icons like locks, credit card logos, and refund guarantees all seem to have a huge impact on checkout conversion rates. 

How to introduce experimentation to your company

Let’s say I’m at a fast growing startup. How do I convince leadership that experimentation is important?

I introduced experimentation to Cameo in small steps:

1. I shared examples of companies that grew through experimentation (e.g., Airbnb).

2. I emphasized how experiments could help us find the truth faster.

3. I found a few 3rd party tools to start running experiments in a scrappy way.

You shouldn’t introduce it if you are pre-product market fit or lack scale. A good signal that your company needs experiments is when people start asking:

1. “How should we optimize this page?”

2. “How do we move more people down this funnel?”

3. “How are we going to measure impact?”

Let’s wrap up by talking about Eppo. What makes Eppo better than in-house or other 3rd party experimentation tools?

Two reasons:

1. We build on top of your data warehouse. This makes it easy for you to track server metrics. Many other platforms only let you client metrics (e.g., mobile).

2. We provide real time results. You can check the results of your experiments anytime. You don't have to wait for an analyst to understand what’s going on.

I think the data warehouse element is key.

Yes. With other tools, you have to ask engineering to write client events, which are then logged separately. This is inconvenient, expensive, and risks privacy. We let you run analysis on data that is already stored in-house.

Thank you Eric! If you liked this interview, give Eppo a try with this free trail link.