During the last Agile Meetup someone asked “What are good metrics during an agile transition? How can I prove (to management) that it works?”
Ever since I read the standard tome on Artificial Intelligence during my studies, I cringe at questions about metrics. The book contained lots of vivid examples with Wumpusses and vacuum cleaner robots and it’s one of those examples that made me eternally weary of (proxy) metrics:
Let’s say, you’re programing an intelligent vacuum cleaning robot. You want it to learn to diligently clean your flat. So you might make “amount of dirt vacuum sucked” the key metric for your little robot. It will learn behaviors that maximize the “amount of dirt vacuum sucked”.
Now, what if the robot would much rather hang with the fridge and play Halo all day? But it has to fulfill its suckage numbers. What’s a little robot to do? It might resort to vacuuming a small part of the flat, then finding a nice corner in which to spit out the dirt, vacuum it again, spit out, vacuum, … until the suckage KPI is looking good. Then it joins the fridge for some serious gaming while most of the flat is dirty.
It feels like real life metrics are often “dirt suckage” analogues. I.e. they are not the real goal but some proxy prescribing how someone thought the goal should be achieved. And then it goes wrong.
In the vacuum example, the real goal would be “clean floor”. But it’s not always clear if you’re measuring a real goal or a proxy. Additionally sometimes it’s hard to measure the real goal. So people rather amend the proxy that did not work with more rules, e.g. “amount of dirt sucked; no spitting out allowed”. This can lead to more dysfunctions. For one, the robot could drive to a sandpit outside of the flat to fulfill its quota fast. Or maybe it vacuums something blocky and, as it’s not allowed to spit, can’t continue vacuuming. Poor proxies defeat the original intent.
That’s why I’m skeptic towards pretty much any metric. On the other hand, how do you know if you’re improving, if you’re not measuring anything? There must be things that it’s okay to measure …
At the meet up our advice was to measure whatever justified the transition. What does the company want to achieve? Make that your metric, but balance it out. Example: If the goal is “More features in deployed in less time” then also measure things likely to be influenced by that. From the top of my head that would be: “Number of bugs found in production”
One thing I’d never ever report to management is Velocity. That will just lead to point inflation. (Though, I might track the number of stories per sprint for myself and the team.)
Long term metrics
After thinking about metrics for a while, I’ve also compiled a list of metrics that I’d measure long term, not only during a transition. It’s a trinity of income, customer satisfaction and employee happiness – I’d measure all of them to keep a good balance:
- Revenue and profit
In my eyes it’s not the goal to earn as much as possible, but it should definitely be enough to sustain the company and all employees - Net Promoter Score – How loyal and likely to recommend you are your customers?
Alternatively, you could measure the number of customers and percentage of repeat customers to gauge how many customers you will have in the future. - Employee Happiness – It seems to be a leading indicator for many things, such as employee turnover (Duh!) but also revenue. You can measure e.g. with a tool like https://www.happinessmetric.com/
A year from now, I might be wiser and the trinity might change, but that’s my current “collection”. What’s in yours?