Heap Digital Experience Insights Report User Guide
- June 10, 2024
- Heap
Table of Contents
Heap Digital Experience Insights Report
Product Information
The Digital Experience Insights Report is a research report that provides insights into user behavior on websites and digital products. The report is compiled by Heap’s data science experts who analyze thousands of anonymized user funnels and millions of user events to locate key patterns and trends. The report includes qualitative interviews with hundreds of customers to detail their concerns about understanding their customers’ journey. The report focuses on identifying the problems with traditional funnels that track user behavior step-by-step, capturing the drop-off rates at each step. It highlights how these funnels were built for an earlier era when websites offered few choices and users had a single path to follow. Today’s digital experience provides more freedom, choices, alternate paths, and multiple routes, and enables all sorts of complicated behavior. The report identifies how funnels are an outmoded tool for understanding this kind of behavior.
Product Usage Instructions
The Digital Experience Insights Report is designed to provide insights into user behavior on websites and digital products. To use the report, follow the steps below:
- Download the Digital Experience Insights Report. Read the report to understand the problems with traditional funnels that track user behavior step-by-step.
- Understand the key takeaways from the report, including the unseen detractors, miscalculations, and invisible paths that traditional funnels miss.
- Consider how your team could have better, more reliable insights on user behavior if they move beyond traditional funnels.
- Reflect on how much faster your team could work, how confident they could be about their decisions, how impactful their prioritization could be, and what would be the effect on your business if you had better insights into user behavior.
Key Takeaways
Methodology: From August to Oct 2021, Heap’s data science experts analyzed thousands of anonymized user funnels and millions of user events to locate key patterns and trends. In addition, we performed qualitative interviews with hundreds of customers to detail their concerns about understanding their customers’ journey. Here’s what we found.
The Problem With Funnels
- of all funnels delivers incorrect or misguided information, leading to misguided goals and incorrect assumptions.
Why does this happen?
-
of funnels miss an interaction that’s associated with lower conversion, leading teams to prioritize the wrong fixes
-
of funnels miscount converting users as drop-offs, because those users skip steps before converting.
-
of funnels contain an alternative path to conversion that isn’t tracked, leading teams to misunderstand how users really navigate their site.
INTRODUCTION
What Funnels Get Wrong
- Welcome to the second Digital Experience Insights Report!
- Our first report analyzed thousands of funnels and literally millions of user events to figure out what events funnels tend to overlook. For this report, we had some bigger questions:
Can funnels capture user behavior as it actually happens?
How well do funnels capture the twists and turns of real user behavior? The
multiple paths to the same goal? The optional activities users take or don’t
take on the way to conversion?
In Short
Can funnels tell us how users really behave? If not, how misleading are they?
The answer
After digging into millions of user events, we found some alarming
information: 84% of funnels provide incorrect or misleading information!
- How is this possible? The answer is that traditional funnels — whichtrack user behavior step by step, capturing the dropoff rates ateach step — were built for an earlier era, one when websites offeredfew choices: users had a single path to follow, which they either followed or didn’t.
- For users, today’s digital experience provide more freedom. Theyoffer choices, alternate paths, multiple routes, and enable all sortsof complicated behavior. That’s a good thing! The problem is thatfunnels are an outmoded tool for understanding this kind of behavior.
For example: if one is trying to answer any of the following, funnels will always return incorrect or misleading data. (Yes, always! It’s how they work.)
- Does performing an optional step (reading product reviews before adding an item to the cart, say) increase or decrease the likelihood of conversion?
- How many users perform that optional step and convert? How does that compare to the number of users who convert without performing that step?
- When there are two possible paths to a single goal, which one has a higher conversion rate? How many users take each path?
- In any given user flow, do users skip any steps on the way to conversion? If so, how many users skip steps, and which steps do they skip?
- Do in-app guides help users get through a specific user flow? If so, how many users interact with the guide, how many don’t, and what are the relative conversion rates between these groups?
For this Insights Report, we dug through millions of events to see how often funnels mislead teams into thinking they understand user behavior, when in fact they’re missing what’s really happening on their product or site! As you read, we encourage you to ask: If teams could have better, more reliable insights on user behavior, what could they do? How much faster could they work? How confident could they be about their decisions? How impactful could their prioritization be? What would be the effect on their business?
Why can’t funnels answer these questions?
The short answer is that funnels are built to track user behavior that is both
linear and sequential. They can only capture users who follow a prescribed set
of events, and that perform those steps in a prescribed order.
The problem is that users rarely act this way!
As we all know, user behavior is rarely linear and rarely sequential. But
today’s digital experiences are filled with options, choices, alternate paths,
and different ways to reach the same goal. For this kind of modern experience,
funnels have too restricted a tool to deliver accurate information.
Unseen Detractors
- of funnels miss an interaction that’s associated with lower conversion.
What this means
59% of the time, teams remain unaware of a key area.
Why do funnels miss so much?
- As we described in the first Digital Insights Report, most teams put funnels together by assembling the steps they think users take on the way to conversion. What’s the problem with this? It leaves a large number of interactions untracked.
- According to our data, for every step tracked in a funnel, almost 20 steps stay untracked.
- This is a problem of what we might call “selection bias”: we tend to look at the behaviors we think will be important, whether or not those are the behaviors that actually are important.
- On the way from landing on the homepage to clicking “buy,” for example, there is any number of steps users take. The key dropoff event might not be the one teams are tracking — it might be an event that’s in between those steps!
If you don’t have a tool that can surface these untracked intermediate steps, you might assume that the 30% dropoff is a problem with Step 3. Maybe you even put your team on fixing that problem! But what happens if the actual difficult step is one of the many intermediate steps? What if that’s the key dropoff event?The crux: because traditional funnels rely on teams to choose the events they think are important, they lead teams to miss the events that are important for improvement.
Miscalculation
- of funnels miscount converting users as drop-offs, because those users skip steps.
What this means
23% of funnels deliver misleading conversion numbers. These skewed numbers can
easily push teams to prioritize the wrong things. Even worse, with traditional
funnels, there’s no way to know how many users are skipping steps, which steps
they’re skipping, and what the relative conversion rates are. (Really!)
Why do funnels miscount conversion?
As we described above, funnels require behavior to be linear and sequential. What exactly does this mean? One way to think of it is that funnels work according to an AND logic. Given a four-step user flow, for example, a funnel analysis first captures the number of people who do Step 1, then the number of people who have done Step 1 AND Step 2, and so on. The 150 users at the end represent the number of people who did Step 1, ANDStep 2, Step 3, AND THEN Step 4. But let’s say that it’s possible for users to go directly from Step 2 to Step 4: to convert without first doing Step 3. How will these users show up in your funnel?
- Let’s see. First, the user completes Step 1. Great! Then the user completes Step 2. Also good. But then … the funnel is waiting for the user to complete Step 3. And the user never does Step 3! Because funnels work on AND logic. They’ll count users who get to Step 4 only if those users have also done Steps 1, 2, and 3 — in that order. If a user doesn’t do any of those steps first, the funnel will count them as having dropped off — having completed Step 2 but not moved forward in the funnel.
- What this means is that users who DO convert but who skip steps along the way will be counted as NON-CONVERTING. How many are there? We don’t know! But let’s imagine some numbers, just to see.
- Let’s imagine you see the numbers on the left above. Looks like your overall conversion rate is 5%, with the biggest drop between Step 2 and Step 3. Wow — if you could increase the conversion between Step 2 and Step 3, you could probably make a big impact on the business. But, what if users aren’t actually dropping off, but are just skipping Step 3? What if there are actually 300 users who do just that? Now you get a very different picture of user behavior. What you thought was a terribly performing funnel (5% conversion rate) is actually a SUPER-performing funnel — 35% conversion!
- AND … what you saw as the greatest moment of friction — moving from Step 2 to Step 3 — isn’t actually friction at all. It’s just that Step 3 is optional, so lots of users skip it. What’s the result? Instead of focusing on reducing friction at Step 3, you might prioritize something else. After all, this funnel is working great! But note, in 23% of funnels, you can’t see that users skip steps. What do funnels conceal? Turns out: A LOT.
Invisible Paths
- of funnels contain an alternative path conversion — that isn’t tracked
What this means
63% of funnels miss the path users actually take to conversion. This leads to
profound misunderstandings about what users actually do, and what improvements
should be prioritized.
Why can’t funnels compare alternate paths?
Above we talked about selection bias. Now let’s examine another limit of funnels: their ability to compare alternate paths. Let’s say you’re running an e-commerce site. Your site has two basic ways to get from your homepage to “add an item” to your cart: you can search, or you can browse by category.
As the site owner, you’re interested in two questions:
- How many people use each option?
- Which option converts better?
As a digital experience owner, how would you answer this? You might be tempted to create two funnels and compare them. Something like the example below: identical funnels, with one tracking people who search before adding to the cart, and the other tracking people who browse categories before adding to the cart.
Looks great! Except … what about people who do both? Maybe someone starts by browsing through the categories in the nav, but then remembers exactly what they wanted and put it in the search bar. What happens with those people — the ones who first do categories, then search, then add to the cart? Let’s call them Person Y. How does Person Y show up in your analysis? Take a look: Person Y would be counted in both funnels.
- On the top, Person Y satisfies the criteria for converting Funnel A: they land on the site, use the search function, then add something to cart. Does Funnel A care if they also viewed the categories? Not at all! Same thing on the bottom: Person Y lands on the site, views categories, then adds to cart — all the criteria for converting in Funnel B.
- So any Person Y — anyone who does both search and category before adding to cart — will count as having converted in both Funnel A and Funnel B. Which…
- Makes it hard to know how many people did Funnel A as opposed to Funnel B, and.
- Makes it impossible to figure out the thing you really care about, which is the relative} conversion rates between Funnels A and B!
- Plus … we’re ignoring all the people who may have done either search or category after having added to the cart! Let’s say there’s a Person Z who gets to the site, uses the category nav to find their item, buys the item, gets sent back to the homepage, and then uses search to find something else. Where do they end up in the funnels?
- Funnel B will count them as it probably should: as a converting user who viewed the category nav first. But let’s also notice that they’d show up in Funnel A’s count as well!
- As happens in the previous example, Funnel A would count this person as a non-converting user who dropped off at Step 2! From the perspective of Funnel A, the fact that Person Z used the category nav and then bought something isn’t relevant: after a user lands on the site, Funnel A will just wait until they use the search bar, and increase the count when they do.
- Could you find a way to eliminate all the Person Z’s from the count you get in Funnel A? It might be mathematically possible, but it would take so much work that it would simply not be worth it.
Let’s review.
You start off with two simple questions:
- How many people take each path to conversion?
- What’s the relative conversion rate between the two paths?
And it turns out that funnels let you answer neither! As in the same example, there are users who will be counted in both funnels, users who count as converted in one funnel and unconverted in the other, and users who convert but are registered as unconverted. And, most importantly, there’s no way to know how many there are. That is, in addition to not being able to tell you the answers to your questions, funnels actually conceal information about users who perform actions in both paths.
Conclusion
As was shown, funnels — at least as traditionally constructed — are a tool that’s both misleading and inaccurate. Because they’re sequential and linear, they’re not able to capture moments when users face a choice. The truth is that modern digital experiences offer nothing but choices!
- Can you afford to be missing a significant percentage of what your customers are doing on your site?
- How much better would you prioritize if you could see all the paths users took on your site?
- What would the dollar impact to the business be on those more informed decisions, and how much more confident would your decisions be?
- What if you could have a tool that delivered accurate numbers and captured what actually happens when users have a choice?
- To learn more about how you can get a real insight into your customer’s digital journey go to heap.io
About Heap
Heap is the premier system of insight for digital experienced builders. Our
mission is to illuminate hidden opportunities for fast-moving digital teams to
delight their customers and move the needle on key metrics. Over 8,000
businesses use Heap to increase revenue, improve conversion, accelerate
decision-making, and drive business impact at scale. Visit heap.io to learn
more.