In the increasingly fast paced environment, good decisions require more than collecting data. It demands strategic analysis and creating actionable insights. Without a structured approach, teams risk falling into analysis paralysis. It is easy to be overwhelmed by noise through excess data. This guide outlines key principles and practical steps. Often the issue is not accessing the data, it is an overabundance of data and not knowing how to use or interpret it.

Source: DallE

Using data and analysis correctly

We must stay one step ahead in anticipating customer needs. The best way to do this is maintaining a continuous state of discovery. That includes analyzing data and engaging in ongoing analysis to refine our product. Also, remember that bad data in means poor insights out, garbage in, garbage out. 

Consider these factors when doing analysis to derive insights to improve the product. 

1. Always start with an hypothesis 

Without starting with a clear hypothesis there are risks of running in circles of knowing what to look for and where to start. There is a risk of falling into analysis paralysis. The chance of drowning in data is high, most organizations do not have a shortage of data. What they do have a shortage of is clear, concrete hypotheses and good data.

Hypothesis building - what should a strong hypothesis have

A strong hypothesis should include the following - 

• Perspective of a customer

• What problem are we solving 

• Outcome of solving that problem or opportunity

• Which actions will we take to drive that outcome or solve the problem (optional)

For example, if we are analyzing the reasons behind a feature's low adoption, the hypotheses could include:

• The feature does not align with user needs or expectations.

• Users require additional information to understand and use the feature effectively.

• The feature fails to address the core problem it was intended to solve. Leading to its rejection or disuse.

• This approach allows us to explore potential causes systematically. Then focus the analysis on validating or disproving these assumptions.

2. Exclude the users who have opted out due to data privacy 

These users cannot be used for experimentation or analysis. This will help ensure your organization is compliant with GDPR. These data laws are strictest in the European Union (EU) with EU based organizations, organizations with data centers in EU, or customers in the EU.

Understand how to exclude these customers. Data hygiene is very important when it comes to regulatory adherence. They could be easily excluded with certain flags or opt out. You may need to work with an engineering or data person to find out what the flag is named.

3. Analyze the behavior for all accounts using the feature or get to statistically significant data

One data point is anecdotal. Seeing a large enough data set gives you the ability to see something to pay attention to. 

An example how to assess if this is a one off or actually a pattern

There are 400 users using a feature. If you look into those 400 users more carefully you will see lots of different segmentation. Most of them belong to different industries, have different workflows and deal with different issues. By the time you look at the different ways that this group can be dissected the numbers can dwindle. Running the analysis for less than 10 accounts may not be an accurate representation of our customer base, and may have bias. It gets closer to being anecdotal.

4. If using random sampling of users, think about major user segments of the market  

Take a systematic approach to identifying the random sample of users. To do this you need to have a good sense of your different types of user segments.

How would you do this if you are a Product Manager for Notion

Notion is a productivity and note taking app. They have different customer segments including enterprise, students and individuals. For this example, let's assume those working in technology companies are their biggest customer segment. The company’s product strategy is to increase revenue from this segment. If you are looking for insights, use adoption and engagement data from paid subscribers working in technology to inform your roadmap. 

5. Assess if this is an insight or customer feedback from a single accounts or consistent from a few accounts 

Consistent feedback or data that can be triangulated creates an insight. If you think you have an insight, reverse it and see if the data matches up to the insight to draw the same conclusion. Like science experiments, the goal is to be replicable. 

We should understand whether other users have similar workflows or have the same needs. This can be done by identifying and running the test on other accounts over a period of time. 

Testing the hypothesis/insight

One insight to test is up to 20% of the users never see the description of a product. At this point, we should look at other users to see if this is a common problem that needs to be solved.

 6. Use a mix of qualitative and quantitative analysis 

Qualitative research is non-numerical data like interview data, customer reviews, customer support ticket information. Quantitative research can include usage data, Google analytics. Using a blend will give you a more holistic perspective on both the what and the why.

Start with a hypothesis or pain point, and then use data to validate or invalidate it. 

If we realize we do not have sufficient data to be confident or data could be interpreted in different ways, work with qualitative data. 

7. Identify the noise and outliers 

There will be noise and outliers in the data which skews the average / median. There is a need to understand the reason behind the noise and outliers. However, exclude the noise in order to produce insightful data. 

8. Write a document to capture the analysis and key insights 

There are many reasons to document the analysis well. It could be for people new to the team or trying to understand decision making and context. Other teams may be trying to replicate the same experiment and will use the document as a starting point. Or if you wanted to keep a record so you could replicate your analysis later.

Include the following in your documentation to make it the most useful: 

• Hypothesis / problem statement / goal 

• How were the accounts chosen and why 

• Time period and other filtering mechanisms used 

• Approach for analysis

• Insights and recommendations

Possible tools you can use

There are a wide range of tools you can use to help to both get and analyze the data. Some include - Google Big Query, Amazon QuickSight, Tableau, Pendo, Power BI or even something as simple as Google Sheets and Excel.

If you learn the principles above, the learning of the tool can come second. The best way to learn the tool is pairing up with an internal expert, working with example data to learn how to interpret it and complement it with learning some basics through an online course such as Udemy. Here’s one for Udemy and learning Tableau.

Effective data analysis is not about crunching numbers. It is about extracting actionable insights that drive innovation and solve customer problems. You can turn analysis into a powerful tool for refining your product and enhancing customer experiences.

Thanks for reading our article! Join our community:

✉️ Subscribe now to stay in the loop. We have more helpful articles coming out soon.

😊 Share directly with your peers and anyone who can benefit. Teach others what you've learned for better retention.

Share

🎤 Engage on Social Media (Twitter (we’re @readaskwhy), LinkedIn)

📥 Drop us your thoughts, questions, and ideas. Simply hit reply!

Let's make learning and sharing a collaborative journey! 🚀