The world of metrics

Job Ladder, Rubrics, Grades or Metrics whichever way we call it, in the corporate ecosystem it all means the same. These are gradings allocated to an employee every performance cycle (quarter or half-year or in some cases annually) in reward of their performance at the job compared to their peers. There are obvious rewards for some who do good and motivation for others to perform like their successful friend - Sharma Ji ka beta. For most people, if a company is doing well, it will shower bonuses and basic salary increases per performance cycle and for some, it can lead to promotions over their current job ladder. I used to be euphoric about these performance cycles after joining MNCs and after years of going through these cycles repetitively, I feel I can share something which has helped me evade the monotonicity.

I think about measuring the impact of my projects at Google every time I do my planning. I feel it’s a very important prerequisite even to begin working on any task. One can not improve if one can’t measure. And so, measuring becomes a super important part of the daily/weekly work in everything that I do so that longer-term objectives can stay on track. I get happy when I read other people’s past performance (self-written) summary which is fortunately viewable internally at Google. I get to put in extra thought while deciding on my projects or subdividing my tasks so that I get the metrics to prove or disprove my or my project’s value. With this blog, I will cover the types of metrics and how do people usually go about measuring them.

Types Of Metrics

Metrics usually do 3 types of things: (might be others as well depending on the context)

1. Understand the purpose or goal of the project or work: a. Is the user problems being solved b. Are customers satisfied with the solution c. Are there any failures or scale issues
2. Identify what steps you need to take to make the goal happen a. Do users opt-in and try the product b. Do users want improvements c. Do users feel underserved
3. Track progress and growth of each critical success factor a. Do users want to keep using the product b. Are users finding the product usable

Each of the above points can be measured by a metric or a set of metrics. Some of them are often quantized into grades from high to low. For example, a project could have subjective feedback from users in the form of a UI component where users can vote on the number of stars where more stars mean higher satisfaction. There could be qualitative metrics that need not be seeded by users and can provide a more robust measurement, such as the percentage of monthly active users who were also active last month.

There can be many other types of metrics that I have spared out of this blog but they all often lie in one of these categories!

Measuring vs Visualization

There’s one thing to create metrics and collect them, but there’s another big task to visualize them. Humans are bad at hypothesising numbers. Its often said that once you have a hypothesis, you can always find some metrics to reinforce the argument as well as disprove it. So, often when we show the metrics on some chart, we pay extra focus to understanding the context in which the numbers will be used. Else it might lose its core value, its deviation and errors in representation. Metrics are inherently cluttered and overuse of statistical tools to aggregate them is not at all preferred. Its often dependent on the project, and often the ideal metrics such as MAU and DAUs are not relevant in the weekly planning of a product.

The use of visualization becomes super impactful to highlight the project in its entirety vs my role in it. For example, a chart that can show the total lines of code written by the team vs me. There might be historical dashboards in every company which automatically get these metrics like this.

Measuring tasks completed vs impact

There’s one set of efforts to show the project’s impact so that overall something of value is built. But in a corporate setting, a higher reward is often given to those employees who contribute more impact to their project rather than the volume of work done. So, certain metrics also help project members to create a niche problem area in their overall bigger project so that they have their impact apart from the one of their project. Examples of such metrics include creating metrics to measure something, fixing an adoption blocker, writing content to ensure a wider adoption, etc.

Measuring an individual performance vs measuring a team’s performance

This is difficult and I don’t want to be a judge in such a situation where a person claims all the work done by their team is their own. I understand that most of the work a person claims as their performance would not have been possible without their team, but just that the team doesn’t have something in their plan can mean it’s an individual performance.

Northstar metrics

I keep a set of metrics handy to understand any project. For example, if I am planning a new chatbot at the start of the performance cycle, I will track:

1. Adoption: # of Bot installs / adds to rooms - target - 100
2. Engagement: % of Bot messages that users interacted with - target >10%
3. Retention: % of users that uninstalled the Bot - target <33%
4. Satisfaction: % CSAT - target >70%

HEART metrics for project success:

Usually, at Google, I observed that most projects follow a pattern called HEART metrics as described below:

1. Happiness CSAT surveys run by the UX team show changes in developer sentiment Reduction in the number of unhappy customer-filed bugs, cloud.google.com feedback, or GitHub issues Increase in SLO adherence as a result Peer feedback from sales/marketing/product/engineering

2. Engagement Documentation pageviews (developers viewing/engaging with your content) GitHub stars or views Unique projects using our libraries Requests generated by our libraries X developers have run this piece of code Percentage requests from our libraries vs customer-written libraries

3. Adoption New developers or projects using our libraries

4. Retention Projects that sustain usage of our libraries for X months Projects that switch from using our libraries to calling a raw API directly (optimize for 0)

5. Task Success Friction logs that lead to fixed usability issues UX benchmark studies showing improvements over time based on artefacts we produced Sample and library health (high test and feature coverage, builds pass, etc)

To conclude, I feel one needs to pay some extra attention to the metrics they want to rely on, how to collect them, how to visualize and in what context these numbers will be used. Often spending some time on petty things like this can make or break a project.