👋 Hey there, hope you’re well.
In Part 1 of the series titled Modeling Meaningful Metrics, I covered what the prerequisites are to create metrics as well as the dependencies between them.
Welcome to Part 2 where I cover how to avoid confusing similar-sounding metrics like Active Users and Activated Users.
Permanent link to this post to bookmark and read later →
I was confused and overwhelmed when I was getting started with metric definitions at Integromat (Make). I didn’t know most of the stuff I know today and even though I had listed down a bunch of metrics complete with definitions, I wasn’t familiar with the process of creating metrics.
Why is that a problem?
Well, you wouldn’t know what data to collect in the first place if you don’t understand what it takes to create certain metrics. This will sound familiar if you’ve gone through the process of answering burning questions. That said, the data needed to answer a burning question may or may not be enough to create a metric.
Let me invoke one of my favorite burning questions:
“We’re acquiring a ton of users every day but very few end up hitting the activation milestone; what’s preventing the rest from performing the actions leading to activation?”
Here, all I needed was to explore a set of events on Mixpanel and create funnels to see where in the onboarding journey were users getting stuck and dropping off. The activation milestone (activation criteria), in Integromat’s case, was also straightforward: a user had to create and turn on their first scenario (workflow) to be considered activated.
However, looking at user actions in isolation wasn’t enough and we needed account-level data to calculate the activation rate. Therefore, the first step was to come up with the definition of an activated account which brought up the following questions:
If any one user in an account hits the activation milestone, do we consider the account activated? Or should we consider an account activated only when every user in the account hits the activation milestone?
What happens if a user turns off their scenario soon after turning it on? What if the scenario is turned off automatically after it encounters repeated errors?
And how do we treat active but incomplete scenarios that only comprise a trigger module and no action module?
I decided to keep things simple in the early stages and concluded that an account would be considered activated as soon as the first scenario in it was turned on. At the time, it was a priority to increase the activation rate, and therefore, it was desirable to keep the definition of an activated account simple.
However, if I could go back in time, I’d make this definition a bit more thorough by including that at least 5 scenario runs took place in an account (scenario_run_count > 4) for it to be considered activated.
Doing so would ensure that a user can test the scenario a few times and experience the magic of a successful scenario execution rather than turning it on without even testing that it works as expected. But all said and done, I don’t regret my decision because in the early stages, simple is always better.
Now, it quickly dawned on me that is_activated_account and is_active_account were completely different metrics and that I should come up with the definition of an active account sooner rather than later. After all, even though the priority was to increase the activation rate, the goal was to drive growth by increasing the number of active accounts or workspaces, which in turn would lead to an increase in the number of paid accounts.
Active vs. Activated
In simple terms, an account is considered activated if it meets the activation criteria and hits the activation milestone – activation, in essence, is a one-time event.
On the other hand, for an account to be considered active, it needs to meet certain criteria on a continuous basis.
Moreover, “active” is a state and at any given time, an active account can become “inactive” (and vice versa).
Therefore, from a metrics perspective, the active_account_count metric must always contain a time dimension such as daily, weekly, or monthly to enable one to answer a basic question like “how many monthly active accounts do we have?”.
Answering questions like the one above is often challenging for SaaS businesses due to the nature of their products. Allow me to illustrate this by discussing what an active account looks like for two very different products: Mixpanel and Integromat.
To derive value from a product like Mixpanel continuously, you have to keep going back to the product to create or view reports. However, you can continue to derive value from a product like Integromat as long as your workflows continue to run smoothly – you only need to log into the product to modify or fix a workflow or to create a new one.
Let’s dig a little deeper.