Answering Burning Questions: Data from external sources
Needed to answer burning questions
Hey there,
If you’re new: Thank you for subscribing to the databeats newsletter where I publish guides for growth practitioners to drive data-powered growth.
If you’re not, well, thank you for sticking around!
Last week’s post covered data from internal sources needed to answer a burning question.
Today’s post will cover data from external sources.
As a reminder, external or third-party tools and APIs used for messaging, support, feedback, authentication, payments, etc. that users interact with as part of the product experience are external data sources.
Let’s get into it.
Data from external sources
Growth-obsessed individuals thrive on experiments because they understand that driving growth is not just about big bold moves; it’s about testing, iterating, failing, learning, and making incremental improvements that lead to predictable, measurable, and sustainable growth. And to run experiments successfully, all they need is for the requisite data to be made available in the right shape in the right tools.
At Integromat, thankfully, we got there sooner rather than later – with data from our app being synced in near real-time to our activation tools, Userflow and Customer.io. The insights from the Mixpanel reports made it evident that to increase the activation rate, we need to walk new users through the process of creating their first scenario on Integromat (via in-app tours) and at the same time, bring users of inactivated accounts back into the app (via lifecycle emails).
As I’ve often mentioned, Integromat had a relatively steep learning curve and the process of creating a scenario involved many steps, leading to a couple of very detailed in-app product tours (created using Userflow) that were embedded in the core product experience. At the start of the onboarding process, users were prompted to choose their preferred email provider and based on their choice, they entered one of the product tours. Moreover, since there were external dependencies (a user had to connect external services that they wanted to use in their Integromat scenarios) while building the tours, we had to account for potential errors that a user might encounter; therefore, we spent a lot of time building and testing the guides (and offering error-handling tips) – all of which ultimately led to the following burning question:
“We have improved the onboarding experience with detailed product tours to get more users to hit the activation milestone; how do we measure the impact of the product tours on our activation rate?”
In other words, how do we establish causality between the newly embedded product tours and a higher activation rate?
A higher activation rate couldn’t automatically be attributed to the new and improved onboarding; it would be foolish to assume that a product tour was effective without measuring how it impacted activation – without figuring out whether the users who hit the activation milestone after completing the tour outnumbered those who dismissed the tour but created (and tested) a scenario nonetheless; or was it the other way around (more users dismissed the tour and became activated than those who completed the tour), indicating that the product tour had a negative impact on the activation rate.
Moreover, an increase in the activation rate could very well be caused by some other factor, such as the marketing team’s efforts to acquire more relevant users. Therefore, we had to go beyond measuring the performance of our experiments – to measure how the experiments impacted the metrics that matter the most. If we only looked at the completion rate of a product tour and kept iterating it to increase the completion rate, we’d be stuck measuring the performance of that product tour rather than measuring its impact on the activation rate – the metric that mattered the most for this experiment.
Therefore, it was a priority to figure out the most efficient way to collect events pertaining to the product tours (that originated in Userflow) and send them to Mixpanel to see whether or not the tours were driving more users to the activation milestone. When I brought this up to Sebastian, the Userflow founder, within a matter of days, he shipped an integration that sent Userflow-generated events to Mixpanel, and with a few clicks, I got what I needed to answer my burning question!
I built a series of funnel reports comprising events from Userflow and events from our app, leading to much-needed insights from our in-app experiment. For instance, I could easily calculate and compare the following metrics for any given period:
Accounts where a scenario was created as a result of the product tour being completed
Accounts where a scenario was created even though the product tour was left incomplete
Accounts where a scenario was created even though the product tour was not started
Userflow’s readymade integration with Mixpanel made this quick and easy. If we were to conduct the same analysis outside of Mixpanel, we’d have to use an ETL tool to ingest the data into a data warehouse, write SQL queries to build data models for each of the above metrics (such as a model that would check if a user completed the product tour before saving their first scenario on Integromat), and then use the models to create reports in a BI tool – a more expensive process that would take longer to implement ( we’ll be paying for the ETL jobs, storage, compute, and most importantly, for people’s time spent building the models).
This use case, I believe, nicely illustrates the importance of looking for the fastest and easiest way to implement a use case, especially one that involves data from external sources. In fact, in this particular example, using the readymade integration was quite economical as it only increased our Mixpanel consumption (and we had a good deal from them). At the end of the day, a growth-obsessed individual’s primary concern is to get fast answers to their burning questions so that they can drive incremental improvements to the product experience; therefore, for one to know the most efficient way to collect the data and make it usable in the requisite destination(s) goes a long way.
Now, let’s look at yet another burning question: