Metrics serve as the foundational elements that guide strategy, inform decisions, and drive success. After all, you can’t manage what you can’t measure. Yet, as the discipline of product management has evolved, so too have the frameworks for measuring and interpreting these vital signs of a product's health and market fit. This exploration into product metrics is meant to dive into the sophisticated yet practical approaches shaped by some of the most thoughtful product leaders today. By understanding the evolution of these metric frameworks and their pivotal role in steering product development and strategy, we equip ourselves with the tools necessary to make informed decisions that align with our overarching goals of delivering value to users and achieving sustainable growth.
So where do we start? From the beginning, naturally.
A brief history of metrics
Today, when we say “metrics” – it can mean many things to many different types of people. Some have defined Key Performance Indicators (KPIs). The inception of KPIs is deeply rooted in the early 20th-century industrial optimization practices pioneered by Frederick Taylor, who introduced scientific management to enhance productivity and efficiency. This concept laid the groundwork for KPIs by emphasizing the importance of data and measurement in performance assessment. Over time, the evolution of KPIs was significantly shaped by Peter Drucker's Management by Objectives (MBO), which in the mid-20th century, advocated for performance measurement against predefined objectives. This period marked a shift towards more strategic management practices, with KPIs serving as crucial tools for evaluating progress towards organizational goals.
As management theories advanced, so did the application and understanding of KPIs. Total Quality Management (TQM) and the Balanced Scorecard, emerging in the latter half of the 20th century, expanded the scope of KPIs beyond financial metrics to include qualitative measures, emphasizing continuous improvement and a holistic view of organizational performance. TQM, influenced by thinkers like W. Edwards Deming (who was once a mentor to our good friend and Jobs to be Done early pioneer, Bob Moesta!) and Joseph Juran, stressed statistical process control and the use of performance metrics to uphold quality and consistency. Concurrently, the Balanced Scorecard, developed by Robert Kaplan and David Norton, introduced a multi-faceted approach, incorporating various perspectives such as customer, internal processes, and learning and growth into the evaluation framework.
Another metrics-driven framework – Objectives and Key Results (OKRs) were conceptualized by Andrew Grove at Intel in the 1970s and later detailed in his 1983 book "High Output Management," was initially introduced to John Doerr during his time at Intel. Doerr, who later joined Kleiner Perkins, brought OKRs to Google, where they became integral to its management culture. Christina Wodtke and John Doerr have since popularized OKRs further through their writings, emphasizing their role in aligning organizational efforts.
Today, KPIs are indispensable across industries, tailored to the unique goals and priorities of each organization. They encompass a broad spectrum of metrics, from financial performance and customer satisfaction to employee productivity and operational efficiency. The modern era, characterized by data abundance and technological advancements, enables real-time tracking and analysis of KPIs, facilitating informed decision-making and strategic planning. KPIs, when effectively selected and monitored, provide invaluable insights into organizational performance, guiding continuous improvement and strategic alignment.
Understanding the Landscape of Product Metrics
How does the history and evolution of metrics apply to today’s Product Managers and Product Leaders? It’s true that the landscape of metrics is vast and varied, spanning across different stages of product development and the user journey. Whatever metrics you track, they should serve as a beacon, guiding product teams through the murky waters of market demands and user needs. To navigate this landscape, it's essential to understand the diversity of metrics available and how they can be effectively applied to different contexts. So where to start?
One great framework to start with – especially for early products and product-led companies – is Dave McClure's Pirate Metrics. Introduced in 2007, this framework offers a straightforward yet powerful approach to measuring and optimizing customer interactions across their lifecycle. The AARRR acronym—standing for Acquisition, Activation, Retention, Referral, and Revenue—provides a structured way to dissect the user journey into manageable, measurable segments. (And go ahead – say it outloud and you’ll quickly realize why these are called “Pirate Metrics.”) This framework has become a staple in the industry, guiding countless businesses, especially in their early finding-product-market-fit phases, in their quest for growth by focusing their attention on key aspects of the customer experience.
However, as companies evolve and the broader macro landscape evolves, so too do the frameworks we rely on. The rise of Software as a Service (SaaS) and product-led growth models has necessitated adaptations to McClure's original metrics. SaaS businesses, in particular, face unique challenges and opportunities that require a slightly different approach to measurement, as pointed out by the team at UserPilot.
In the SaaS world, the stages of Acquisition, Activation, Adoption, Retention, Revenue (specifically, expansion), and Referral take on new meanings. Acquisition is no longer just about users landing on a website; it's about users beginning a trial or demo of the product. Activation shifts focus from a first positive experience to users realizing and experiencing the product's value—often defined as the "AHA moment." Adoption then delves deeper into the user's engagement with the product's key features, beyond the initial activation.
Retention in SaaS is critical, as it's not just about users returning to the product, but also about ensuring they continue to find value and remain paying customers. Revenue expansion becomes a pivotal stage, where the goal is to increase the average revenue per account through upselling, cross-selling, and add-ons. Finally, Referral in the SaaS context is about turning satisfied users into advocates who actively promote the product to others.
This adapted framework highlights the importance of not only attracting and activating users but also deeply engaging them with the product, ensuring they find ongoing value, and encouraging them to become promoters. It's a more nuanced approach that reflects the complexities of SaaS and product-led growth models, where the product itself is at the forefront of acquisition and retention strategies.
The adaptation of metric frameworks for SaaS and product-led growth underscores a broader trend in product management: the need for flexibility and context-specific strategies. While foundational frameworks like Pirate Metrics provide a valuable starting point, it's the thoughtful adaptation and application of these metrics to one's unique circumstances that drive true product success. By understanding the diversity of metrics and how they can be tailored to different stages of product development and user journeys, product managers can equip themselves with the tools needed to steer their products toward success in an ever-changing market.
Signup conversion is essential—but real growth lies in user onboarding
Gain a competitive edge in 2024 with Chameleon's User Onboarding Benchmark Report. View industry benchmarks and get insights from SaaS companies of all sizes—from startup to enterprise. Compare your performance, stay ahead of trends, and refine your growth strategy.
North Star Metrics
When starting out, aside from an adapted version of Pirate Metrics, where else can we look to help us keep things on track? What other metrics really matter? Maybe it’s not “metrics” but more a “metric” that can guide us – a North Star Metric.
By definition, a North Star Metric is simply a guiding metric that’s meant to measure success for a company and/or product. It’s meant to be a singular focus that aligns and motivates entire teams and organizations. This concept, central to product strategy, represents the core value that your product delivers to customers. It's the metric that, above all others, embodies the essence of your product's impact and success.
The significance of the North Star Metric cannot be overstated. It transcends mere numbers to encapsulate the mission and vision of a company, guiding product decisions, and strategic initiatives. It's about coherence and alignment, ensuring that every team member, from engineers to marketers, is working towards a common goal that directly contributes to the overarching success of the product.
Lenny Rachitsky's exploration of North Star Metrics provides a profound insight into the process of selecting this pivotal metric. He emphasizes the necessity of choosing a metric that not only accelerates your business's growth but also reflects the core value you deliver to your users. This selection process is far from arbitrary; it requires a deep understanding of your business model, how your product grows, and, crucially, how it's used by your customers.
Through his research, Rachitsky offers a framework for thinking about North Star Metrics across different types of companies, from marketplaces and platforms to SaaS businesses driven by product-led growth. For instance, marketplaces like eBay and Uber might focus on consumption growth—such as auctions listed or rides taken—as their North Star, highlighting the direct usage of their platforms. In contrast, SaaS companies, especially those employing a freemium model, might prioritize engagement or customer growth metrics, underscoring the importance of not just acquiring users but deeply engaging them with the product.
So, what have some of today’s most successful companies actually selected as their North Star Metrics? Airbnb, for example, initially centered on "nights booked" as a clear indicator of their platform's utilization and value to users. This focus on a single, impactful metric helped align the company's efforts towards enhancing user experiences and expanding the service's reach. Similarly, Spotify has navigated through various North Star Metrics, from subscriber growth to engagement metrics like monthly active users, reflecting its evolving business strategies and the expanding scope of its service offerings.
Brian Crofts, CPO at BambooHR, has shared that one strategy in helping come up with reliable metrics – including North Star Metrics – involves Jobs to be Done (JTBD), a business theory, framework, and perspective on why customers buy products. In this fireside chat held with Crofts during his time at Pendo, Crofts discusses how going deeper and understanding customer intent can be critical when developing core metrics – and JTBD can be a tool to help you do that by putting yourself in the shoes of your customers. By understanding your customers better, you can continually check your metrics to make sure they’re in line with the value you’re actually delivering to customers. If they’re not, then JTBD may help you identify potential “Vanity Metrics” that you’ve set in place – metrics that might look nice, but actually aren’t meaningful.
Ultimately, choosing the right North Star Metric is an exercise in precision and strategic foresight. It involves not just understanding where your product stands today but also envisioning where you want it to go. By selecting a metric that truly encapsulates the value your product offers and the impact it seeks to make, you set a clear, cohesive direction for your entire team—a direction that leads to meaningful growth and enduring success.
The Science of Product Metrics
The approach we take towards metrics can significantly influence our journey's outcome. Holly Hester-Reilly, with her unique background in chemical engineering and research science, offers a compelling perspective on this subject. In this discussion with John Cutler, Hester-Reilly advocates for treating product management as a science, where hypotheses are tested, data is meticulously analyzed, and conclusions are drawn with precision—a perspective that underlines the importance of a data-driven approach in product development.
This scientific method, as applied to product management, emphasizes the need for continuous experimentation. Just as a scientist would not rely on assumptions without testing them, Product Managers and Product Leaders are encouraged to adopt a mindset of constant inquiry and validation. Each feature, user flow, or marketing campaign is an opportunity to learn more about what truly resonates with users and drives value. This process of continuous experimentation is not about reckless trial and error but about informed, structured exploration aimed at uncovering deeper insights into user behavior and product performance.
However, embracing this data-driven, experimental approach does not mean losing sight of the real-world context in which our products exist. The balance between rigor and practical application is crucial. Metrics and data analysis should not become so esoteric or convoluted that they lose their connection to the actual experiences of our users. Hester-Reilly's own journey from the structured world of chemical engineering to the dynamic field of product management illustrates this balance beautifully. She draws parallels between understanding complex molecular systems and deciphering the intricate web of user interactions within a product. In both realms, the goal is to understand and predict behavior within a system, yet the ability to apply these insights in a practical, impactful manner is what truly makes a difference.
The rigor in this scientific approach comes from the methodologies used to gather and analyze data. It's about setting clear hypotheses, defining measurable outcomes, and systematically testing our assumptions. Yet, this rigor must be coupled with a flexibility that allows for the nuances of human behavior and market dynamics. It's about being thorough and methodical but also agile enough to adapt when the data points us in a new direction.
In the real-world application of metrics analysis, the focus should always remain on actionable insights. Data for data's sake holds little value. The metrics we obsess over should directly inform our decisions and strategies. For instance, a high churn rate might prompt a deeper investigation into user onboarding experiences, while a drop in activation rates could lead us to reevaluate our product's value proposition. Each metric tells a part of the story, but it's the action we take in response to these metrics that ultimately shapes our product's trajectory.
Hester-Reilly's emphasis on the scientific method in product management serves as a reminder of the power of a data-driven approach. It encourages us to question our assumptions, to seek out evidence, and to base our decisions on a solid foundation of empirical data. Yet, it also acknowledges the artistry involved in translating these insights into tangible product improvements that resonate with users.
In adopting this balanced, scientific approach to product metrics, we equip ourselves with the tools to navigate the uncertainties of product development. We learn to value experimentation not just as a means to an end but as an integral part of the product development process. And in doing so, we open ourselves up to a world of insights that can propel our products to new heights, ensuring that our efforts are not just guided by data but are also deeply connected to the real-world experiences of our users.
The importance of experimentation with metrics
In the dynamic world of product management, experimentation is not just a tool; it's a mindset. It's about embracing the unknown with a structured approach to learning and adapting. Holly Hester-Reilly, with her rich background in scientific research, underscores the pivotal role of experimentation in product development. Similarly, insights from the UserPilot blog post shed light on the practical applications of this approach, particularly in the SaaS domain, where the rapid iteration and user feedback loops are integral to growth and success.
Experimentation serves as the crucible within which our assumptions about metrics can be validated or refuted. It's about moving beyond gut feelings and into the realm of evidence-based decision-making. For instance, consider the process of defining activation points within a product—the moments that truly matter to the user's journey. These are not merely milestones but pivotal experiences that can make or break the user's engagement with the product. Experimentation allows us to pinpoint these activation points with precision, ensuring that we're not just guessing what matters to our users but truly understanding it.
Once these activation points are identified, the real work begins. A/B testing becomes a powerful ally in this process, allowing us to tweak, adjust, and refine our approach to these critical junctures. It's not about making wholesale changes based on a hunch but about nuanced adjustments that can lead to significant improvements in user engagement and, ultimately, product success. For example, a slight change in the onboarding flow or a new feature introduction could dramatically affect the activation rate, and only through rigorous A/B testing can these effects be accurately measured and understood.
The importance of experimentation extends beyond the activation points. It permeates every aspect of the product development process, from the initial acquisition strategies to the retention and referral mechanisms. Each stage of the user journey presents an opportunity for hypothesis testing and learning as a continuous cycle of testing and optimization, especially in SaaS, where the product itself is the primary driver of growth. Here, experimentation is not a one-time effort but an ongoing commitment to improving and evolving the product in response to real user needs and behaviors.
Moreover, experimentation fosters a culture of curiosity and resilience within product teams. It encourages a mindset where failure is not a setback but an opportunity to learn and grow. By embedding experimentation into the very fabric of the product development process – including the metrics we’re using to measure success, teams can become more agile, responsive, and, ultimately, more effective in delivering value to their users. This includes sharing your metrics internally – and being open to feedback. Crofts points out that, too often, product teams don’t necessarily share metrics so broadly within the company. But a part of buying into a culture of experimentation is demonstrating transparency. Being open to sharing metrics broadly internally exemplifies this.
Evolving your Metrics Strategy
As products mature and markets evolve, the metrics that once served as reliable indicators of success may no longer provide the same level of insight or guidance. Adopting that experimentation mindset will allow you to keep your mind open to this evolution.
As Rachitsky pointed out, even the most successful companies have found themselves re-evaluating their North Star Metrics in response to new challenges and opportunities. This evolution is not a sign of previous misdirection but an acknowledgment of the product and market's natural growth and transformation.
Take, for example, the transition of a product from a startup phase into a scale-up phase. The metrics that mattered when the product was seeking market fit—perhaps user acquisition or activation rates—might shift as the product gains traction and looks towards scaling. Here, retention or revenue expansion metrics might come to the fore, reflecting the new priorities of deepening user engagement and maximizing the value derived from an established user base.
It’s an example how the metrics we’re using to measure success – including our North Star Metrics – should evolve over time. As companies grow and their market dynamics change, so too must their guiding metrics. Dropbox's shift from focusing on user engagement to emphasizing paid customer growth is one such example. This transition marked a strategic pivot from a broad user base to a more sustainable, revenue-generating customer base, reflecting the company's maturation and changing priorities.
The evolution of our metrics underscores a critical point: they are not static. According to Rachitsky, about a quarter of the companies he surveyed had recently changed their North Star Metrics or were in the process of doing so. This fluidity is a testament to the dynamic nature of products and markets. It highlights the need for Product Managers to remain adaptable, continuously reassessing their metrics to ensure it remains aligned with their current objectives and the value they aim to deliver to users.
This flexibility requires a willingness to question and reassess the assumptions that underpin our chosen metrics continually. Are they still reflective of the value we aim to deliver to our users? Do they align with our current business objectives and market realities? These questions are crucial for ensuring that our metrics strategy remains relevant and effective.
The evolving nature of metrics strategies also highlights the need for a culture of continuous learning within product teams. This culture encourages experimentation and data-driven inquiry, fostering an environment where insights are actively sought and valued. It's about creating a feedback loop where metrics inform decisions, and the outcomes of those decisions, in turn, inform the refinement of those metrics.
This importance of this culture was reinforced by Adam Thomas in his talk at the Virtual edition of INDUSTRY: The Product Conference. In the talk, Thomas encouraged Product Managers to set Survival Metrics in place – as metrics that help product teams decide whether an initiative is worth investing in more, whether it’s time to pivot, or whether it’s time to stop altogether – and are set on a foundation of three pillars: speed, being data-informed, and having political safety. These metrics can be very useful in evaluating products and projects continuously, but once again – it requires a culture of experimentation and the political safety that comes with that culture.
Summing it all up
In crafting effective product metrics, it's crucial to navigate the evolving landscape with agility and insight. It’s true that every company and product is different and will have its own approach to measuring success metrics. But understanding the different frameworks and models that are at our disposal – including Pirate Metrics, North Star Metrics, and Survival Metrics – can help you develop what’s right for you, your company, and the products you manage.
Remember that it’s not just about tracking progress but about embodying the core value your product delivers. This demands a culture of curiosity, collaboration, and resilience, urging product teams to question assumptions and pivot with precision.
A big thanks to everybody I referenced in this post – including Holly Hester-Reilly, Brian Crofts, Adam Thomas, Lenny Rachitsky, Dave McClure, and others as I’ve learned a lot from all of you and hope that this essay can help others learn from your work as well!
•
Again, we hope you enjoyed this article and got value from it. Please consider sharing this Product Collective email with your colleagues, or head directly to our blog to view this post on the web.