The Overuse of Metrics Can Scam You Out of Customer Obsession
Metrics are a great tool, as long as they're wielded wisely. I'll talk about some of the risks and pitfalls.
I usually send on Thursday, but I've been a bit busy the last few days. Our household just had Covid sweep through it. The internal isolation dance in the house, along with my personal lack of energy, has made it a little difficult to get things done. However, this topic I really do enjoy, so I slowly banged it out while drinking tea. I hope you enjoy!
Through my career, I repeatedly heard the quote, "What you measure is what you manage." The intent behind this quote is to encourage teams to put metrics on the things they care about, to ensure that nothing goes wrong.
Imagine a simplistic version of a website which has some content on it. What can go wrong operationally? Well, a page can take too long to load. A page can fail to load. That's about it.
If you have the right metrics on page load time, and failure rates, you should generally know if anything goes wrong with your site. Metrics are great, right?
I've repeatedly had co-workers insist that their change had solved a problem or improved a situation, but the metrics showed a different story.
Years ago, we had a content change pushed by product management, with the clear assumption that it would improve engagement in our application.
The Product Team asked for us to keep the development simple, as this was a quick win, and said that they didn't want to invest the time necessary to instrument the change with metrics. They wanted to just fix what they saw as an issue, and move on to their next new feature.
Thankfully, our engineer insisted on doing things properly (another great lesson for everyone to remember), and instrumented the before and after engagement situation with this particular feature.
Shocking everyone involved, but particularly the product team, the change drastically negatively impacted engagement.
Through our following investigation, we learned about our user's behavior, and ways they were using our product we didn't intend. We also were reminded that at scale, metrics can provide amazing insights which may counter the prevailing wisdom.
With all that being said, why would I be bad-mouthing metrics? You see, I worked with many teams which used metrics aggressively. It was my assumption that metrics existed for most important behaviors on their site or service.
What I learned over time was that metrics are a great tool, but they can be heavily overused or abused as well.
I'm going to walk through some issues with metrics, and some risks with using metrics as your primary and/or only tool.
Blindly chasing the metric
Perhaps the biggest risk of metrics is that you swap your customer focus for metrics focus.
Most metrics are a proxy for a customer behavior or impact. A metric for number of orders. A metric for page load time. A metric for the rate of purchases per page load.
If you build a set of metrics which you believe represents your customer, the risk is that you'll blindly try to improve your metrics rather than view your customer experience holistically.
Once upon a time, my team found a bug in a service we used. It caused an empty return value in the API call which was supposed to return data. Essentially, it gave back nothing, rather than the right answer.
We submitted a ticket to the owning team. That team promptly closed the ticket, stating that there was no error.
We re-opened the ticket, insisting there was an error, and gave more information.
They closed the ticket again, stating that they've checked, and there are no errors for our account.
At this point I walked a couple buildings over, talked to their manager, and got the issue resolved. But what had happened to require me to leave my comfy office?
Their team had optimized their metrics long enough that they believed their metrics represented their customer. They drove their team's behaviors off those metrics. When we claimed there was an error, they saw it as factually incorrect, as they had metrics stating there were no errors.
In this case, as the API was returning an empty value, rather than actually having an error, their metrics were incorrect about the customer impact.
They blindly chased that error metric, rather than approaching the issue with an open mind. Instead of assuming that their metrics cover some issues, they assumed their metrics covered all issues.
Instead of investigating their customer's experience, they investigated their metrics.
Metrics are a great tool, as long as you don't assume they're the only tool.
