The problem with looking at only big data

My co-workers know when they see me pull this image up it means somebody’s violated some basic principle of research. It’s not necessarily that the data is bad per se - it could be that people are misreading the data or looking at the wrong data and drawing the wrong conclusions as a result - but hey, the cat is cute and it’s a good way to bring humor into the conversation.

With any research, we’re trying to answer a pertinent business question. In order to do that, the first thing we need to figure out what’s the best research method to answer the question at hand. And we have infinite options: in-person or remote usability studies, surveys, AB tests, clickstream analysis, interviews, diary studies … you get the idea. We need all of these approaches because there’s no one method that’s always best.

Unfortunately, people have become so enamored with big data that they’re starting to forget about the value small data can bring. (Just a quick note: I’m defining big data as anything that comes from analytics (clickstream, AB test results) and small data as anything that comes from qualitative approaches (usability studies, interviews, diary studies, etc.). With tactical questions one approach is often sufficient but for large strategic questions, to get a full, 360 view of whatever it is you’re studying it’s often best to triangulate quantitative and qualitative insights, and to pull quantitative insights from not just analytics but also survey results. Otherwise at best you won’t have the full picture, or worse, will draw incorrect conclusions with minimal data.  

A lot is being written on blending qualitative and quantitative methods (e.g., by researchers at Microsoft and Google), which is great and we need more researchers to write on this, but I still hear people in the wild draw conclusions that are inappropriate by looking at data that can’t possibly answer their question.

Here are two of the most common ways I hear people confounding insights by looking at the wrong data.

Trying to infer happiness or other preferences from engagement metrics

Recently I heard a colleague report a much lower bounce rate on a new version of one of our main pages. Yay! Cause for celebration! A lower bounce rate is always great news. But he said: “the bounce rate is 50% lower than the last version so that means our customers like it.” What’s wrong with this conclusion? The bounce rate is lower simply because customers are staying on the page and not bailing immediately. But from just this metric alone we don’t know WHY the bounce rate is down. Is it truly because customers like the experience? Because they’re staying on the page and finding what they need but they don’t really like the experience? Because they’re not finding what they need and are spending more time looking for it than they did before? We have no idea by simply seeing that the bounce rate is lower.

The only way to find out if customers like the experience is to either ask them directly (likely through a survey), or watch them use the site and see for yourself if they express joy. Qualitative research is needed to understand WHY the bounce rate is lower.  

Trying to predict engagement by asking customers what they’d do

People always want to predict how likely customers are to engage with a site or better yet to convert. No one wants to allot resources to build something that customers don’t want, so it’s totally reasonable that researchers would be asked to help identify the likelihood of conversion.

We were recently asked to help answer just this question. The requester wanted to know how likely customers would be to fill in personal info on a form by showing a few variations of the design and asking users directly “how likely would you be to fill in this form”. This approach is challenging for several reasons, the primary one being there are factors beyond just the design (how much trust the user has with the site, the perceived value provided) when people decide whether to give personal information on a website. But the main reason we balked at the approach is because people are lousy at predicting their own behavior. How many times have you said you were giving up sugar or not going to buy the expensive shoes but then caved? I know I have. People are emotional and we often go with the emotional decision instead of the logical one.

We can’t know how likely people are to pick one option or another unless we run an AB or vaporware test. We can (and frequently do) run usability tests to narrow the options before an AB test by determining which versions are more understandable, but even then we can’t predict which one people will more likely complete.

So what can we do?

As I said earlier, it’s awesome that we have so many research methods available and can use whichever method we need to get the desired answer. If you’re a researcher, start by sitting down with the requester to fully understand what is the objective of the research - what is the person trying to understand? Then carefully pick the method(s) that will get that answer. If you’re a product manager, marketer, designer, then start to think through the outcome you’re trying to get instead of thinking about the research methods you already know. So instead of going to your research team and saying “we need to run a survey” simply tell them “we need to find out why people who gave us their contact info haven’t yet used our service.” Let your research partner choose the appropriate method.