As a product manager, you're no stranger to tough decisions. Should you prioritize that new feature your biggest customer is clamoring for or focus on improving the onboarding experience to boost activation rates? Is it time to overhaul your pricing strategy or double down on customer retention initiatives?

​

The answer is simple in today's data-rich world: follow the data. After all, isn't that what being "data-driven" is all about? But here's a thought that might give you pause: what if relying too heavily on data could lead you astray?

​

Welcome to the world of data-informed decision–making, a nuanced approach that's revolutionizing how savvy product managers navigate the complex landscape of product development and strategy.

​

Imagine you're at the helm of a ship. Data is your compass, providing crucial direction. But you're not blindly following its every twitch and turn. Instead, you combine its guidance with your knowledge of the seas, the weather, and your crew's capabilities. That's the essence of being data-informed.

​

This essay will explore why this approach is gaining traction among successful product teams. We'll dive into the art and science of collecting quantitative and qualitative data and how to strike the right balance between the two. You'll discover strategies for turning the firehose of incoming data into actionable insights and learn how to avoid common pitfalls that can lead even the most well-intentioned product managers astray.

​

But here's where it gets really interesting: we'll challenge some commonly held beliefs about data in product management. Can having too much data hinder decision-making? How do you reconcile conflicting data points? And in a world obsessed with metrics, is there still room for intuition and experience?

​

Whether leading product at a scrappy startup or a Fortune 500 company, you'll find practical advice for building a data-informed culture that empowers your team to make better decisions faster.

​

Let's dive in and discover how being data-informed can transform the way you work – and the products you build.

Unlock AI Secrets in Product Management at INDUSTRY: The Product Conference!

​

AI is the hottest topic in tech right now. To say it will be impactful is an understatement, considering how transformative this era-defining marvel is destined to be. So, what are we going to do about it?

​

An excellent first step is to assess how we can leverage this technology today. We are doing that at this year's INDUSTRY: The Product Conference, which will be held September 23-25 in Cleveland, Ohio.

​

There, you'll be treated to several sessions delving deep into the topic:

• 4hr Workshop: Mastering the use of Generative AI
• Keynote: Product in the Age of AI
• Keynote: Selling AI Transformation
• Keynote: A.I. & the Future of Product Management
• 1 hr Working Session: AI for Product Managers

​

And Group Roundtable Discussions to discuss the findings with your peers.

​

You are running out of time to save on your INDUSTRY ticket! Earlybird pricing ends on August 1st -- so register before then and save $100.

Understanding Data-Informed Decision Making

In the world of product management, the mantra "data-driven decision-making" has been repeated so often it's practically become gospel. However, as Richard White, the founder of businesses like Fathom and UserVoice, points out, a subtle yet crucial shift is happening: the move from being data-driven to data-informed.

​

At first glance, this might seem like a trivial distinction. After all, isn't using data to make decisions the whole point? But dig a little deeper, and you'll find that being data-informed is a more nuanced and ultimately more effective approach for product managers.

​

Let's start with a definition: Data-informed decision-making is the practice of using data as a key input in the decision-making process while also considering other factors such as domain expertise, intuition, and qualitative insights. It's about striking a balance between the cold, hard numbers and the human elements that make product management as much an art as it is a science.

​

The fundamental principles of data-informed decision-making include:

1. Using data as a guide, not a dictator
2. Combining quantitative and qualitative insights
3. Considering context and nuance
4. Embracing uncertainty and experimentation
5. Continuously learning and adapting based on outcomes

​

So, how does this differ from a purely data-driven approach? Milene Davis, a Product Manager now at You Need a Budget, offers a valuable perspective. She cautions against blindly following data, pointing out that this can lead to overlooking essential nuances or misinterpreting results. Instead, she advocates for challenging the data and using it as a tool in the product manager's toolkit rather than the sole determinant of decisions.

​

This approach acknowledges a fundamental truth about data: it's not infallible. Data can be incomplete, biased, or misinterpreted. By being data-informed rather than data-driven, product managers can avoid falling into the trap of false certainty that often comes with an overreliance on metrics.

​

The benefits of adopting a data-informed approach are numerous:

1. More holistic decision making: By considering quantitative data and qualitative insights, product managers can make more well-rounded decisions considering the full product and market complexity.
2. Increased agility: Data-informed decision-making allows faster pivots when necessary, as it doesn't require waiting for statistically significant results for every decision.
3. Better risk management: By combining data with experience and intuition, product managers can often spot potential issues that pure data analysis might miss.
4. Improved stakeholder communication: A data-informed approach provides a framework for explaining decisions to stakeholders, balancing hard numbers with strategic reasoning.
5. Enhanced creativity: Product teams have more room for innovative thinking and bold experiments by not being constrained solely by what the data dictates.

​

Ruben Ugarte, founder of Practico Analytics, emphasizes the importance of this approach, particularly for smaller companies or those in the early stages. He notes that relying too heavily on data can be misleading when it is limited. Instead, he suggests using data to inform hypotheses and guide experimentation rather than as the final arbiter of decisions.

​

It's also worth noting that the data-informed approach aligns well with the concept of "product sense" – that almost intuitive understanding of what will work for users that many successful product managers possess. By combining this product sense with data insights, managers can make analytically sound and intuitively right decisions.

​

Being data-informed is about elevating the role of data by integrating it more thoughtfully into the complex, often messy reality of product management. It's about using data not as a crutch but as a powerful tool in the product manager's arsenal – one that, when wielded skillfully, can lead to better products, happier users, and more successful businesses.

​

Types of Data for Product Managers

Not all data is created equal, and knowing which types to focus on can make or break your decision-making process. Let's dive into the two main categories of data you'll be working with: quantitative and qualitative.

​

Quantitative data is all about the numbers. It's measurable, countable, and often comes from statistics and metrics. Here are some key types of quantitative data you'll encounter:

1. Behavioral Analytics: This is the bread and butter of product metrics. It tells you what users are doing with your product. How many people signed up today? Which features are they using most? Where are they dropping off in the onboarding flow? Tools like Mixpanel or Amplitude are two platforms that can give you this granular insight into user behavior.
2. Usage Metrics: These metrics give you a broader view of how your product is being used. Think daily active users (DAU), monthly active users (MAU), session length, or time spent in-app. These numbers can give you a sense of overall engagement and product health.
3. Performance Indicators: These metrics directly tie to your business goals. Conversion rates, customer lifetime value (LTV), churn rate, and revenue per user are all examples of performance indicators that can help you gauge the success of your product.
4. ​

But here's the thing about quantitative data: while it's great at telling you what's happening, it often needs to be revised to explain why. That's where qualitative data comes in.

​

Qualitative data is all about the why. The rich, descriptive information gives context to your quantitative data. Here are some key types of qualitative data:

1. Customer Feedback: This includes everything from app store reviews to survey responses. It's the unfiltered voice of your customers telling you what they love, hate, or wish your product could do.
2. User Interviews: These are in-depth conversations with users that can uncover insights you'd never get from quantitative data alone. They can help you understand user motivations, pain points, and desires in a way that numbers simply can't.
3. Support Tickets: Users' issues with your product can be a goldmine of qualitative data. They can highlight usability issues, feature gaps, or confusion that might not appear in your usage metrics.

​

​Richard White emphasizes the importance of qualitative data, especially in B2B environments where usage doesn't always equal satisfaction. He points out that while quantitative data can tell you what's happening, qualitative data helps you understand why and what to do about it.

​

The real magic happens when you combine quantitative and qualitative data. Let's look at an example:

​

Imagine you're seeing a spike in user churn (quantitative data). Your first instinct might be to panic, but hold on. You dig into your customer feedback (qualitative data) and discover that users leave because they find your new feature confusing. Now you have the what (increased churn) and the why (confusing new feature), giving you a clear direction for your next move.

​

​Chris Abad, Director of UX at Google, advocates for this balanced approach in this keynote talk he gave at INDUSTRY: The Product Conference. He suggests using quantitative data to identify trends and patterns and then diving into qualitative data to understand the reasons behind those trends. This combination allows you to make decisions that are both data-informed and user-centric.

​

But how do you decide which data to prioritize? Ruben Ugarte suggests thinking about your current goals. If you're focused on winning new business, prioritize data from prospective customers. If retention is your main concern, focus on feedback from existing users.

​

The key is to avoid getting stuck in analysis paralysis. It's about finding that sweet spot where you have enough data to make an informed decision but not so much that you're overwhelmed. Remember, the goal isn't to collect every possible data point. It's gathering the correct data to help you make better decisions for your product and users.

​

Strategies for Collecting Data

Alright, you're sold on the importance of both quantitative and qualitative data. But how do you get your hands on this goldmine of information? Let's dive into some effective strategies for data collection that will turn you into a veritable product detective.

​

Direct Customer Feedback: Straight from the Horse's Mouth

1. In-app feedback tools: These are your secret weapons for capturing user sentiment in real time. Tools like UserVoice or Canny allow users to submit ideas, report issues, or give feedback without leaving your product.
2. Surveys and questionnaires: Don't write these off as old school. When used strategically, surveys can provide valuable insights. The key is to keep them short, focused, and timely. For instance, try sending a quick survey right after a user completes an essential action in your product. Their experience is fresh, and they're more likely to give meaningful feedback.
3. User testing sessions: Nothing beats watching real users interact with your product. Tools like UserTesting or Maze allow you to set up user scenarios and tasks and observe how they navigate your product. It's like being a fly on the wall in your users' homes (but, you know, less creepy).

​

Internal Feedback Channels: Tapping into Your Team's Collective Wisdom

Your colleagues are a treasure trove of user insights. Here's how to tap into that knowledge:

​

1. Sales team insights: Your sales team is on the front lines, hearing directly from potential customers about their needs and pain points. Set up a regular cadence for collecting their insights. But don't just ask for their top feature requests. Instead, have them log the feedback they're hearing, word for word. This helps you avoid the "telephone game" effect, where information gets distorted as it's passed along.
2. Customer support data: Support tickets are like the canary in the coal mine for your product. They can alert you to usability issues or feature gaps before they become significant problems. Work with your support team to categorize and quantify the issues they see. But don't stop at the categories – dig into the actual conversations to understand the context behind the numbers.
3. Success team observations: Your customer success team has a unique perspective on how customers use your product in the wild. Regular check-ins with this team can reveal patterns in how successful customers are using your product and common stumbling blocks.

​

While feedback from customers and internal teams is crucial, a wealth of data can also be gathered by simply observing how users interact with your product.

1. Analytics tools: The various platforms can give you a detailed look at user behavior. But here's the catch: you need to set them up thoughtfully. Work with your engineering team to ensure you're tracking the right events and properties. As Ross Walker from Figma points out, it's not just about collecting data – it's about collecting the correct data to inform your decision-making.
2. Event tracking and user journeys: Don't just track individual actions – look at the entire user journey. Tools like FullStory or Hotjar allow you to view session recordings and heat maps, visually representing how users navigate your product. This can reveal unexpected user behaviors, or pain points you might not have anticipated.

​

Creating a Comprehensive Data Collection Process

Now, here's where the magic happens. The key to effective data collection is creating a process that combines all these sources into a unified view. Work to create a system where all feedback flows into a central repository regardless of source. This allows you to interleave feedback from different sources and get a holistic view of user needs and behaviors.

​

But it's easy to fall into the trap of collecting data for data's sake. Always tie your data collection efforts back to your product goals and key questions you're trying to answer. Just be sure not to wait for perfect data before using it. Chris Abad emphasizes the importance of starting with whatever data you have, whether imperfect or incomplete. You can always refine your process over time.

​

Analyzing and Prioritizing Data

You've set up your data collection systems and are now swimming in a sea of information. But data without analysis is just noise. The real challenge lies in turning that raw data into actionable insights. Let's explore how to understand it all and prioritize what truly matters.

​

One of the most powerful tools in a product manager's arsenal is a solid framework for analyzing and prioritizing data. These frameworks provide a structured approach to decision-making, helping you cut through the noise and focus on what's most important.

​

​The RICE framework, popularized by Intercom, is a favorite among product managers. RICE stands for Reach, Impact, Confidence, and Effort. This method scores potential initiatives based on these four factors, giving you a quantitative way to compare different options.

​

Here's how it works: You estimate the Reach (how many users will this affect?), Impact (how much will it affect them?), Confidence (how sure are you about your estimates?), and Effort (how much time and resources will this require?). By multiplying the first three factors and dividing them by Effort, you get a RICE score to help you prioritize your product roadmap.

​

But RICE isn't the only game in town. Some teams prefer the more straightforward ICE framework (Impact, Confidence, Ease), while others swear by the classic Value vs. Effort quadrant. The key is to find a framework that resonates with your team and aligns with your product goals.

​

Whichever framework you choose, use these methods to "de-risk" decisions by comparing the potential impact of different initiatives against actual customer demand.

​

Segmentation and Cohort Analysis

Raw numbers can be misleading. That's where segmentation and cohort analysis come in. By breaking down your data into specific user groups or time-based cohorts, you can uncover insights that might be hidden in the aggregate data.

​

For instance, you might analyze churn by user segment instead of looking at the overall churn rate. You might discover that while your overall churn looks stable, you're losing high-value customers and replacing them with lower-value ones. This kind of insight can dramatically shift your priorities.

​

Cohort analysis, on the other hand, allows you to track how user behavior changes over time. This is particularly useful for understanding product changes' long-term impact or identifying patterns in the user lifecycle.

​

​Milene Davis stresses the importance of segmentation, mainly when dealing with conflicting data points. By drilling down into specific user groups, you can often resolve apparent contradictions and gain a more nuanced understanding of user behavior.

Identifying Patterns and Trends

Data analysis isn't just about crunching numbers—it's about spotting patterns and trends that can inform your product strategy. Look for correlations between different metrics. For example, you might notice that users who engage with a particular feature have significantly higher retention rates. This could suggest that promoting or improving that feature could boost overall retention.

​

Pay attention to changes over time, too. Sudden spikes or drops in key metrics often signal important shifts in user behavior or product performance. These can be opportunities for quick wins or early warning signs of potential issues.

​

Dealing with Conflicting Data Points

It's not uncommon to encounter conflicting data points in your analysis. Maybe your usage metrics show high engagement with a feature, but your customer feedback suggests users find it frustrating. How do you reconcile these contradictions?

​

The key is to resist the urge to discard data that doesn't fit your expectations. Instead, treat conflicting data as an opportunity to understand your product and users better.

​

Start by questioning your assumptions. Are you interpreting the data correctly? Are there alternative explanations for what you're seeing? Often, apparent contradictions can lead to valuable insights when you dig deeper.

​

Consider the context of different data points. Quantitative data might tell you what's happening, but qualitative data can often explain why it's happening. Combining these perspectives can usually resolve conflicts and gain a more complete picture.

​

Remember, the goal of data analysis isn't to find a single "right" answer but to develop a nuanced understanding that can inform better decision-making.

​
Using Data to Inform Product Decisions

Now that you've collected and analyzed your data, it's time to implement those insights. One of the most critical applications of data in product management is shaping your product roadmap. Your data analysis should determine which features to build next and how to allocate your resources.

​

Start by aligning your data insights with your overall product strategy and business goals. For example, if your data shows that a particular feature is highly correlated with user retention and improving retention is a key business objective, that feature should likely be prioritized in your roadmap.

​

Data can also be a powerful tool for identifying and addressing user experience issues. By combining behavioral data with user feedback, you can pinpoint areas of friction in your product and prioritize improvements. For example, if your funnel analysis shows a significant drop-off at a particular step in your onboarding process, that's a clear signal that something needs attention. Pair this with qualitative feedback from user interviews or support tickets to understand users' specific issues, and you have a solid foundation for making informed UX improvements.

​

​Chris Abad emphasizes the importance of continuous iteration in UX improvements. For instance, you can use A/B testing to validate your changes and monitor your metrics to ensure your improvements have the desired effect.

​

Pricing is often one of the most challenging decisions for product managers, but data can provide valuable guidance. Analyze usage patterns to understand which features offer the most value to user segments. This can inform tiered pricing structures or help you identify opportunities for premium features.

​

You can also examine conversion rates and churn data across different price points and packages. If you see a significant drop-off in conversions above a certain price point, it might indicate that you've hit a psychological barrier for your target market.

​

Data can also play a crucial role in shaping your go-to-market strategies. Use your user segmentation analysis to identify your most valuable customer profiles. This can help you target your marketing efforts more effectively and tailor your messaging to resonate with these high-value segments. You can also analyze your acquisition data to understand which channels are most effective at bringing in quality users. This can help you allocate your marketing budget more efficiently.

​

​Ross Walker from Figma suggests using data to create a feedback loop in your go-to-market process. Monitor critical metrics as you roll out new campaigns or enter new markets, and be prepared to pivot quickly based on the results you're seeing.

​

It’s essential to be aware of potential biases in your data collection or analysis and not be afraid to challenge results that don't align with your understanding of the market or user needs. Remember that not all decisions need the same level of data backing. You'll want robust data support for major strategic shifts or high-risk initiatives. For more minor tweaks or experiments, moving forward with limited data and learning as you go might be okay.

​

Coming up:

• Common Pitfalls and How to Avoid Them
• Adapting Data Strategies to Company Stage and Size
• Building a Data-Informed Culture
• Future Trends in Data-Informed Product Management

​

​

•

Again, we hope you enjoyed this article and got value from it. Please consider sharing this Product Collective email with your colleagues, or head directly to our blog to view this post on the web.