Heap How to Measure Feature Success Instructions
- June 10, 2024
- Heap
Table of Contents
How to Measure Feature Success
Instruction Manual
The best first step to building a data-driven culture
How to Measure Feature Success
We get it. You’re interested in using data to make decisions around your product. Maybe you’ve been reading about how data helps teams produce better outcomes. Maybe you’re comfortable with data, but are joining a team where most of your colleagues aren’t. Maybe your CEO just sent down a message: “Hey! I want to see the data!”
For PMs and digital teams, these are common scenarios.
But … so is the feeling that using data would be great, but you’re not sure
where to start. There’s a lot of high-level marketing out there around leading
with data. And there are a ton of metrics you could start looking at. And a
ton of ways to start using them. There are worries enough with being a PM. Why
add more?
We say: worry no more!
Whether you’re new to being data-driven, or trying to nudge a team or even a
larger org into using data, what follows is an easy, nearly-foolproof way to
start.
In our work with hundreds of customers, we’ve found that the following
framework is the least-stressful approach, whether you’re starting as a single
PM or as a team leader.
The key is to keep things low-pressure. For now you’ll only be figuring out
what to measure, and how. Evaluating yourself according to these metrics can
come later. The goal here is simply to adopt some low-lift techniques that
can start introducing data and metrics into your process.
Then, once you’re comfortable with these metrics, you can move on to using
them to set goals, to tweak existing features, to plan your product roadmap,
and document your impact and push for a better title, and many other antastic
things!
So: we recommend that you read the following and discuss it with your team.
What follows will give you the foundation you need to start becoming a data-
driven machine.
Follow along with our Worksheet on Measuring Feature Success.
Start by measuring something you’ve already launched
Ideally you’ll get to the point where you can use data to make decisions for
the future. That’s a ways away. If you’re really looking to start adding
metrics into your team’s workflow, a
great beginning point is to gather data around a feature you’ve already
launched.
Doing this gets you far. Already, it’s a great way to start getting your team out of the “output” treadmill and start thinking more about outcomes. Shipping features is important, but it’s not the be-all-end-all. As PMs we need to start thinking about moving from output to outcomes.
Of course, we also all know that just shipping a feature doesn’t necessarily do anything for the business. While you deserve a pat on the back for shipping (good job!), what’s way more important is whether what you’ve shipped actually helped the business.
The trick: start small. (The journey of a thousand miles…)
The goal here is simple: to get a handle on what metrics matter for you and
how you can go about collecting them.
And the only way to know that is to start tracking metrics.
Even the most basic numbers can be deeply informative: how many people use
your feature?
Who are they? How easy is it to use? Do users like it?
Then, once you start collecting data, you can start setting baselines you can
use to measure the effectiveness of your next feature.
Assess your feature across the five dimensions below
To start, we recommend using the following framework to start measuring
success. In general, it’s important to keep five dimensions in mind. Depending
on your product or feature, all may not apply. But often teams intuitively
track two or three of these, while failing to think about the others.
It may help to have a quick discussion about which one or two of these most
matters to you, or which one or two matters most right now. It’s difficult to
improve all five at once, so it can be useful to think about the ones you
think will most help the business.
For each dimension, you want to think about which questions you want to
answer, and how to gather the data you need to answer them. (We have some
ideas about how to gather the data. Visit us at heap.io.)
We’ve also attached a companion worksheet for this guide.
You can use this to define the metric(s) you’ll be using to track each
dimension, and to measure baseline numbers.
Start sharing and iterating!
Now, once you have some data, you can start thinking about how to improve. Do
you have discoverability issues? Are different cohorts using the product
differently? Are people using the product as frequently as you’d like? How
many users complete the journeys you set up for them?
Why or why not?
With a little data at hand, you can start hypothesizing endlessly, and start
creating experiments to test those hypotheses out. You can start prioritizing
efforts based on metrics you’d like to move. You can report up on the success
of your work.
Above all, you can start thinking about outcomes over output.
Becoming data-driven takes time. We commend you on taking the first step!
The Five Dimensions of Product Success
Breadth
“How many users have adopted this feature?”
Breadth looks at the total number of customers using a feature, or the number
of users per account. It’s typically tracked as “adoption,” but as adoption
may involve some of the below dimensions, we recommend thinking of it as
breadth.
Measuring breadth is important for multiple reasons. Of course it tells you
how widely adopted your feature is, and if the users you built the feature for
are actually using it. It can also give you signal about discoverability (low
breadth numbers may indicate a discoverability issue), communication (do
customers know about the feature?), or even usefulness.
From a business perspective, breadth is often a measure of “stickiness,” or
potential churn risk.
Some potential breadth questions:
- How many users overall have seen or tried the feature?
- How many users per organization have seen or tried it?
- What overall percentage of your users is that?
- If there are specific users you built this feature for (in specific organizations or verticals or with specific job titles), how many of them have seen or tried it?
Frequency
“Within a given time period, how often do users engage with this feature?”
Frequency tells you if your feature is providing repeatable value for your
users.
See how often users take advantage of the product or feature. Is the feature
one they can’t live without? Is it a regular part of their workflow? Looking
at the number of daily, weekly, or monthly active users (DAU, WAU,
Depth
“How much does this feature matter to users?”
Depth measures the level of engagement users have with a particular feature.
Depending on the feature or product, level of engagement may be reflected in
frequency of use or average time spent.
In general, depth looks at the number of product features or areas that are
used by the average person or account. How much of your product are customers
using? What is the level of engagement that users have with all of the
features that you offer? Take Spotify’s product features, for example. When
you use Spotify, are you just playing songs? Or are you catering playlists or
sharing music?
Some potential frequency questions:
- For each user group that is using the feature, how often are they using it?
- Are they using the feature in the way you’d imagined?
- How much time do users spend within the product?
- Where in the user flow are they using the feature?
Some potential depth questions:
- What percentage of daily, weekly, or monthly active users use this particular feature?
- What’s the ratio of Daily Active Users (DAU) or Weekly Active Users (WAU) to Monthly Average Users (MAU)? What percentage of monthly active users are weekly active users?
- What percentage of users actively engage with 3 or more features?
- What is the frequency of engagement within specific user groups?
Usability
“How hard is it to use this feature?”
Usability tells you how much effort it takes users to accomplish their goals
with your feature. Are they able to successfully perform a task from start to
finish? Can they perform tasks quickly and successfully?
Measuring usability can sometimes be as simple as tracking completion rate,
but we highly recommend using Effort Analysis to assess the amount of friction
that shows up between funnel steps in a given user flow.
Sentiment
“How do users feel about this feature?”
Sentiment looks at how customers feel when engaging with a product. It’s a
more qualitative metric that is harder to score, but we highly recommend
thinking about ways to track this kind of response. As we all know, products
and features can elicit strong emotional reactions from users, and those that
establish positive emotional connections tend to exhibit superior retention
rates.
To track sentiment, you can use customer interviews, external reviews, support
tickets, and/or NPS ratings, among other measures.
Getting a good signal on sentiment often requires partnering with customer
success, or reaching out to customers and/or partners.
Some potential usability questions:
- What percentage of users who start a given flow end up finishing it? (Completion rate)
- What is the total number of interactions it takes users to complete the task?
- How much time does it take to move from one interaction to the next?
Some potential sentiment approaches:
- Develop a follow-up plan with your customer success team to address NPS survey responses.
- Ask high-scoring promoters the reasons behind their good score.
- Meet with detractors to learn how you might improve the feature experience.
That’s it! Start by using our worksheet to track the metrics that matter to
your feature or product. Likely some will surprise you.
They might even turn out differently than you’d imagined. If so, great. You’re
taking your first step towards actually using data to make decisions.
When you learn which metrics are most critical to improving the success of
your product releases, you can set measurable targets before developing a
feature, and then use the results to further refine your goals. You can learn
more about this in our next guide.
Your journey to becoming a more insights-driven organization is just beginning
—congrats on taking the first step!
About Heap
Heap is the premier system of insight for digital experience builders. Our
mission is to illuminate hidden opportunities for fast-moving digital teams to
delight their customers and move the needle on key metrics. Over 8,000
businesses use Heap to increase revenue, improve conversion, accelerate
decision-making, and drive business impact at scale.
Visit heap.io to learn more.
© 2022 Heap Inc. | heap.io
Measuring Feature Success
Use the following framework to create a dashboard for measuring product or
feature performance and identifying opportunities for improvement.
Core Indicators of Success
Breadth, depth, usability, frequency, and sentiment: together, these five core
indicators of feature success will help you establish baseline performance.
For each, identify the metric(s) that deliver the most information about your
specific feature or product.
Breadth
Breadth measures the number of users who have adopted your feature.
In how many accounts was your feature activated?
Ways to Measure Breadth:
- Adoption Rate = Number of people using this feature / number of total users
- Number of users per account = number of users / number of accounts
- Adoption rate in specific accounts
The metric(s) you’ll use to track Breadth:
Your initial measurements for them:
Frequency
Frequency measures how often users engage with your feature within a given
time period.
Is your feature one that users can’t live without?
Ways to Measure Frequency:
- Number of engagements per day/week/month
- Number of engagements per day/week/month for a specific user group
The metric(s) you’ll use to track Frequency:
Your initial measurements for them:
Depth
Depth measures the level of engagement that users have with your feature.
Are users maximizing value from your feature?
Ways to Measure Depth:
- % of daily, weekly, or monthly active users who use this feature
- Time spent by users on this feature, or by feature users in the product as a whole
- Correlation of use of this feature to use of other key features
The metric(s) you’ll use to track Depth:
Your initial measurements for them:
Usability
Usability measures the amount of effort it takes users to accomplish their
goals.
Are users able to successfully perform a task from start to finish?
Ways to Measure Usability:
- Completion rate
- Average time users spend completing a task
- Total number of interactions for the specific user flow The metric(s) you’ll use to track Usability:
The metric(s) you’ll use to track Usability:
Your initial measurements for them:
Sentiment
Sentiment captures how customers are feeling when engaging with your feature.
A more qualitative measurement.
Ways to Measure Depth:
- Positive sentiment in customer interviews
- External reviews (Reddit, G2, etc.)
- Number of support tickets
- In-app NPS
The metric(s) you’ll use to track Sentiment:
Your initial measurements for them:
© 2022 Heap Inc. | heap.io