Burning Questions

You know exactly what to do with the answers!

Last week, we discussed the perils and consequences of collecting data without context.

Today, we’ll discuss all the good things that happen in the presence of proper context.

Ready, are we? Alright then.

When context exists, people start asking questions. And asking a lot of questions without expecting immediate answers to every question is the only way to get better at asking the right questions – it’s a muscle that one needs to build. 

When I was leading Growth at Integromat (now Make.com – Zapier’s toughest competitor), and we were setting up our data infrastructure, I had a ton of questions.

In fact, we were acquiring hundreds of new users every day – sometimes even crossing a thousand in a day – and if we wanted to, we could have collected massive amounts of data to track everything every user did on the website and inside the app. However, as a bootstrapped company, Integromat operated lean, and thankfully so – it pushed me to think things through and question popular narratives like why it’s a good idea to collect as much data as you can.

I had been a power user of Integromat even before I joined the team in early 2018. And I’d already helped a ton of people with their questions about the product through a couple of communities I was active in. As a result, I intricately understood the product and the various personas it catered to, and by the time we started implementing our data stack in late 2019, I had enough context to start asking questions – lots of questions.

However, as I started listing them down – questions I thought I badly wanted answers to – a pattern emerged and it became crystal clear that every question of mine fell into one of two categories:

• Type 1 Questions: Whose answers I didn’t know what to do with – nice-to-have answers that would serve my curiosity but not much else. It wasn’t clear how to use the insights from those answers to drive growth.

• Type 2 Questions: Whose answers I knew exactly what to do with – answers that would give me, for instance, the insights to figure out what’s preventing users from hitting the activation milestone. I had clear use cases and the tools in place to use the insights to drive growth.

The Type 2 questions are essentially what I refer to as burning questions.

Context always precedes burning questions

Here are two distinct characteristics of a burning question: 

1. It is very specific

2. It has context baked in

Let’s look at an example:

“We’re acquiring a ton of users every day but very few end up hitting the activation milestone; what’s preventing users from performing the actions leading to activation?”

It’s specific with context baked in.

The first part of the question, “We’re acquiring a ton of users every day but very few end up hitting the activation milestone” is the context to ask something specific like, “What’s preventing users from performing the actions leading to activation?”

On the contrary, a Type 1 question is super broad and lacks context. The answer tells what’s going on but it doesn’t help understand the cause and the fix.

Here are a few examples:

• “How many new users did we acquire in the last 30 days?”

• “How many users did we convert?

• “How many users did we lose?”

Answers to Type 1 questions like these provide a bird’s eye view of the health of the business – perfect for an executive dashboard powered by a BI tool.

But a bird’s eye view is not what you’re looking for as the person responsible for getting more users to use your product more often, is it? You’re not looking for numbers without any context for you to do something to increase or decrease those numbers.

You’re looking for specific, contextual information that you can act upon immediately and continuously.

For instance, you can identify the causes responsible for a low activation rate if you have context about the actions performed by users as they move through the onboarding process

But that’s not it, is it?

When you have context, you also have the exact data points you need to run campaigns and experiments to fix the problem.

As you start thinking through the solution, you are likely to have more questions and might need additional data points in the destinations where you intend to consume and act upon the data. At the very least, you’ll need to run through a series of steps to figure out whether the issue lies with how your product works, how and when a feature is presented, or something else altogether.

You might end up talking to inactivated users only to figure out that they had very different expectations from the product, indicating a misalignment between Marketing and Growth¹. By relaying the data to your marketing team, you might further learn that one of the ad campaigns being run by an external agency is driving visitors to an outdated landing page that talks about features no longer available on the free plan.

It’s good to keep in mind that context leads to burning questions, which lead to better collaboration and ultimately, better quality data. Let’s explore how.

Not just about insights  

It’s important to highlight that the goal of a burning question is not only to derive an insight but also to figure out how to run an experiment, how to do so effectively, and how to measure the impact of an experiment. And experiments lead to additional insights which lead to more context and thereby, more burning questions. 

1. Marketing ≠ Growth