Heap How to Measure Feature Success Instructions

How to Measure Feature Success
Instruction Manual

The best first step to building a data-driven culture

How to Measure Feature Success

We get it. You’re interested in using data to make decisions around your product. Maybe you’ve been reading about how data helps teams produce better outcomes. Maybe you’re comfortable with data, but are joining a team where most of your colleagues aren’t. Maybe your CEO just sent down a message: “Hey! I want to see the data!”

For PMs and digital teams, these are common scenarios.
But … so is the feeling that using data would be great, but you’re not sure where to start. There’s a lot of high-level marketing out there around leading with data. And there are a ton of metrics you could start looking at. And a ton of ways to start using them. There are worries enough with being a PM. Why add more?
We say: worry no more!
Whether you’re new to being data-driven, or trying to nudge a team or even a larger org into using data, what follows is an easy, nearly-foolproof way to start.
In our work with hundreds of customers, we’ve found that the following framework is the least-stressful approach, whether you’re starting as a single PM or as a team leader.

The key is to keep things low-pressure. For now you’ll only be figuring out what to measure, and how. Evaluating yourself according to these metrics can come later. The goal here is simply to  adopt some low-lift techniques that can start introducing data and metrics into your process.
Then, once you’re comfortable with these metrics, you can move on to using them to set goals, to tweak existing features, to plan your product roadmap, and document your impact and push  for a better title, and many other antastic things!
So: we recommend that you read the following and discuss it with your team. What follows will give you the foundation you need to start becoming a data- driven machine.
Follow along with our Worksheet on Measuring Feature Success.

Start by measuring something you’ve already launched

Ideally you’ll get to the point where you can use data to make decisions for the future. That’s a ways away. If you’re really looking to start adding metrics into your team’s workflow, a
great beginning point is to gather data around a feature you’ve already launched.

Doing this gets you far. Already, it’s a great way to start getting your team out of the “output” treadmill and start thinking more about outcomes. Shipping features is important, but it’s not the  be-all-end-all. As PMs we need to start thinking about moving from output to outcomes.

Of course, we also all know that just shipping a feature doesn’t necessarily do anything for the business. While you deserve a pat on the back for shipping (good job!), what’s way more important is whether what you’ve shipped actually helped the business.

The trick: start small. (The journey of a thousand miles…)
The goal here is simple: to get a handle on what metrics matter for you and how you can go about collecting them.

And the only way to know that is to start tracking metrics.
Even the most basic numbers can be deeply informative: how many people use your feature?
Who are they? How easy is it to use? Do users like it?
Then, once you start collecting data, you can start setting baselines you can use to measure the effectiveness of your next feature.

Assess your feature across the five dimensions below

To start, we recommend using the following framework to start measuring success. In general, it’s important to keep five dimensions in mind. Depending on your product or feature, all may  not apply. But often teams intuitively track two or three of these, while failing to think about the others.
It may help to have a quick discussion about which one or two of these most matters to you, or which one or two matters most right now. It’s difficult to improve all five at once, so it can be  useful to think about the ones you think will most help the business.
For each dimension, you want to think about which questions you want to answer, and how to gather the data you need to answer them. (We have some ideas about how to gather the data. Visit us at heap.io.)

We’ve also attached a companion worksheet for this guide.
You can use this to define the metric(s) you’ll be using to track each dimension, and to measure baseline numbers.

Start sharing and iterating!

Now, once you have some data, you can start thinking about how to improve. Do you have discoverability issues? Are different cohorts using the product differently? Are people using the  product as frequently as you’d like? How many users complete the journeys you set up for them?
Why or why not?
With a little data at hand, you can start hypothesizing endlessly, and start creating experiments to test those hypotheses out. You can start prioritizing efforts based on metrics you’d like to  move. You can report up on the success of your work.
Above all, you can start thinking about outcomes over output.
Becoming data-driven takes time. We commend you on taking the first step!

The Five Dimensions of Product Success

Breadth

“How many users have adopted this feature?”
Breadth looks at the total number of customers using a feature, or the number of users per account. It’s typically tracked as “adoption,” but as adoption may involve some of the below dimensions, we recommend thinking of it as breadth.
Measuring breadth is important for multiple reasons. Of course it tells you how widely adopted your feature is, and if the users you built the feature for are actually using it. It can also give  you signal about discoverability (low breadth numbers may indicate a discoverability issue), communication (do customers know about the feature?), or even usefulness.
From a business perspective, breadth is often a measure of “stickiness,” or potential churn risk.

Some potential breadth questions:

• How many users overall have seen or tried the feature?
• How many users per organization have seen or tried it?
• What overall percentage of your users is that?
• If there are specific users you built this feature for (in specific organizations or verticals or with specific job titles), how many of them have seen or tried it?

Frequency
“Within a given time period, how often do users engage with this feature?”
Frequency tells you if your feature is providing repeatable value for your users.
See how often users take advantage of the product or feature. Is the feature one they can’t live without? Is it a regular part of their workflow? Looking at the number of daily, weekly, or  monthly active users (DAU, WAU,

Depth
“How much does this feature matter  to users?”
Depth measures the level of engagement users have with a particular feature.
Depending on the feature or product, level of engagement may be reflected in frequency of use or average time spent.
In general, depth looks at the number of product features or areas that are used by the average person or account. How much of your product are customers using? What is the level of  engagement that users have with all of the features that you offer? Take Spotify’s product features, for example. When you use Spotify, are you just playing songs? Or are you catering playlists  or sharing music?

Some potential frequency questions:

• For each user group that is using the feature, how often are they using it?
• Are they using the feature in the way you’d imagined?
• How much time do users spend within the product?
• Where in the user flow are they using the feature?

Some potential depth questions:

• What percentage of daily, weekly, or monthly active users use this particular feature?
• What’s the ratio of Daily Active Users (DAU) or Weekly Active Users (WAU) to Monthly Average Users (MAU)? What percentage of monthly active users are weekly active users?
• What percentage of users actively engage with 3 or more features?
• What is the frequency of engagement within specific user groups?

Usability
“How hard is it to use this feature?”
Usability tells you how much effort it takes users to accomplish their goals with your feature. Are they able to successfully perform a task from start to finish? Can they perform tasks quickly  and successfully?
Measuring usability can sometimes be as simple as tracking completion rate, but we highly recommend using Effort Analysis to assess the amount of friction that shows up between funnel  steps in a given user flow.

Sentiment
“How do users feel about this feature?”
Sentiment looks at how customers feel when engaging with a product. It’s a more qualitative metric that is harder to score, but we highly recommend thinking about ways to track this kind of  response. As we all know, products and features can elicit strong emotional reactions from users, and those that establish positive emotional connections tend to exhibit superior retention rates.
To track sentiment, you can use customer interviews, external reviews, support tickets, and/or NPS ratings, among other measures.
Getting a good signal on sentiment often requires partnering with customer success, or reaching out to customers and/or partners.

Some potential usability questions:

• What percentage of users who start a given flow end up finishing it? (Completion rate)
• What is the total number of interactions it takes users to complete the task?
• How much time does it take to move from one interaction to the next?

Some potential sentiment approaches:

• Develop a follow-up plan with your customer success team to address NPS survey responses.
• Ask high-scoring promoters the reasons behind their good score.
• Meet with detractors to learn how you might improve the feature experience.

That’s it! Start by using our worksheet to track the metrics that matter to your feature or product. Likely some will surprise you.
They might even turn out differently than you’d imagined. If so, great. You’re taking your first step towards actually using data to make decisions.
When you learn which metrics are most critical to improving the success of your product releases, you can set measurable targets before developing a feature, and then use the results to further  refine your goals. You can learn more about this in our next guide.
Your journey to becoming a more insights-driven organization is just beginning —congrats on taking the first step!

About Heap
Heap is the premier system of insight for digital experience builders. Our mission is to illuminate hidden opportunities for fast-moving digital teams to delight their customers and move the  needle on key metrics. Over 8,000 businesses use Heap to increase revenue, improve conversion, accelerate decision-making, and drive business impact at scale.
Visit heap.io to learn more.
© 2022 Heap Inc. | heap.io

Measuring Feature Success
Use the following framework to create a dashboard for measuring product or feature performance and identifying opportunities for improvement.

Core Indicators of Success
Breadth, depth, usability, frequency, and sentiment: together, these five core indicators of feature success will help you establish baseline performance. For each, identify the metric(s) that  deliver the most information about your specific feature or product.

Breadth
Breadth measures the number of users who have adopted your feature.
In how many accounts was your feature activated?
Ways to Measure Breadth:

• Adoption Rate = Number of people using this feature / number of total users
• Number of users per account = number of users / number of accounts
• Adoption rate in specific accounts

The metric(s) you’ll use to track Breadth:
Your initial measurements for them:

Frequency
Frequency measures how often users engage with your feature within a given time period.
Is your feature one that users can’t live without?
Ways to Measure Frequency:

• Number of engagements per day/week/month
• Number of engagements per day/week/month for a specific user group

The metric(s) you’ll use to track Frequency:
Your initial measurements for them:

Depth
Depth measures the level of engagement that users have with your feature.
Are users maximizing value from your feature?
Ways to Measure Depth:

• % of daily, weekly, or monthly active users who use this feature
• Time spent by users on this feature, or by feature users in the product as a whole
• Correlation of use of this feature to use of other key features

The metric(s) you’ll use to track Depth:
Your initial measurements for them:

Usability
Usability measures the amount of effort it takes users to accomplish their goals.
Are users able to successfully perform a task from start to finish?
Ways to Measure Usability:

• Completion rate
• Average time users spend completing a task
• Total number of interactions for the specific user flow The metric(s) you’ll use to track Usability:

The metric(s) you’ll use to track Usability:
Your initial measurements for them:

Sentiment
Sentiment captures how customers are feeling when engaging with your feature.
A more qualitative measurement.
Ways to Measure Depth:

• Positive sentiment in customer interviews
• External reviews (Reddit, G2, etc.)
• Number of support tickets
• In-app NPS

The metric(s) you’ll use to track Sentiment:
Your initial measurements for them:

© 2022 Heap Inc. | heap.io

References

• Heap - Better Insights. Faster. | Heap

Read User Manual Online (PDF format)

Read User Manual Online (PDF format)  >>

Download This Manual (PDF format)

Download this manual  >>