Why metrics are broken

Large organisations are great at creating metrics for things. They always have to have dashboards and scorecards and other little devices that let them monitor how they're doing on so many different fronts.

That's especially true for IT organisations, at least, any that are big enough that you can't just ask everyone involved how things are going.

Governance people, especially, love metrics and scorecards. 

The problem with metrics is they aggregate information to such a degree that you often know there is a problem, but not what it is. You'll see, for example, some scorecard with a red traffic light against a project, so you know something is going wrong, but you have no idea at all how to fix it.

Fixing it requires deep diving into human conversations with the people involved. Metrics are pale reflections of the people-reasons things are breaking. And, of course, it is always people reasons why things go wrong, when you boil everything down.

My problem with this whole process is that the aggregate data you get on scorecards always lags reality by a significant degree because the metrics collection process is incredibly flawed.

Metrics collection is based on the premise that people with the most to lose when something goes wrong will admit they are in trouble soon enough for management to help them. By the time the scorecard reflects reality, it is invariably because the people involved can no longer hide what's broken - they may, for example, have missed a delivery date or had to request more money.

This, actually, is a problem with all systems where the reporting process is based on top down self-reporting.

I think it is time for a new approach to this. Why can't we have reporting systems that go directly to the human front line and aggregate up? Why can't everyone in a project team be polled on what they think the status is and use that as the reporting metric? At least it would be more honest and be much closer to the human conversations which provide the solution.

Certainly, I've never worked in an organisation that manages its scorecards in this way. Has anyone got examples they'd care to share where this kind of thing has worked?

Comments

[JBM]

In a previous company I worked for, all IT staff had to log their time every month to a list of large projects, continuous improvement/small projects, break/fix initiatives, vacation, training etc. This forced everyone to do a very sober review of where they spent their time for the month and see was it aligned with the priorities for the quarter. This allowed a constructive discussion at the direct report level so we got to see rolled up results that confirmed the real level of capacity we had for new projects.

While we did not poll everyone on what they thought the status of a project was, this was a topic that came up at review meetings if a date or deliverable looked to be at risk. Management style plays a large factor in building the environment and atmosphere to enable meaningful data to be collected and rolled up through the organisation. While a red light is something to be worried about, it would be far more worrying if everything was green or orange. Building a culture where a red light does not mean someone is going to get fired is a huge success. Yes it would be great to have better drill down capability so you can understand the issues and drivers that are making a project go red but in most rudimentary scorecard type displays there should be some backup data available to help explain the exceptional events.

In the world of on-shore/off-shore, in-sourced/out-sourced, virtual orgs, teleworking, distributed organisations, mergers and acquisitions etc we need this capability and culture now more than ever. Management style and culture must be mature enough to facilitate good data capture mechanisms as well as progressive means of drilling down to the issues so the real problems can be uncovered and managed without people running for cover.

[Andrew Rouse]

James
I think you have missed the importance of Spin in all this. How can we have failure without someone senior having egg on their face?
I work in central government IT like you but further down the chain, I see supplier performance stats, our stats and also speak to people in suppliers I used to work along side and those poor end users we have and for some reason what the working end know never seems to be reflected.
Oh we get the big consultancies in to tell us the time with our watch and then clear off with the watch but if it is bad news (which is rare- spin again) it gets lost.
The age of quality, caring about what we deliver and focusing on a good service has gone - cheap is good but don't look at TCOO. Or better still lets look at a new innovation to fix it all, don't look at the mistakes - that is painful.

[James Gardner]

I think, in your comment Andy, you have come to exactly the point I was trying to make. Those who are affected by bad news are motivated not to give bad news to those who can do something about it for many reasons. I presume, therefore, that you agree with my assertion that bottom up reporting - at least on a project basis - would actually be a good thing if it could be made to work?

[Andrew Rouse]

James
Yes I agree we need to get that end user, ground level view and then take action to do something about it - however don't expect it to happen. "Don't bring me problems - bring me solutions" - I assume you've heard that said before!
Plus fixing broken things is hard, far easier to say everything is great and aren't I great.
Look forward to you convincing the CIT exec that stuff is broken, lets know about it and fix it.
I'll see you in the queue at the Jobcentre testing JSA front end (as a claimant).

[InformationAge]

Hi James,

We recently wrote about CrowdCast, whose software allows managers to use prediction markets to find out if employees think projects are going to fail. May be of interest.

http://tinyurl.com/22ndj2d

Pete Swabey
Information Age

[James Gardner]

That's a great tip - thank you very much, Pete.

[Robin Gower]

Doesn't the 'red' in 'red, amber & green' indicate: 'it's now too late to do anything about this'!?

As Pete has mentioned, predict markets might be what you're looking for. They allow staff to reveal bad news anonymously (through bets rather than tense meetings). I wrote this on the topic a couple of years ago: http://clients.infonomics.ltd.uk/?q=info_markets

I noticed, reading Information Age, that Gartner are recommending that pattern recognition be used for performance management (http://www.gartner.com/it/page.jsp?id=1212913). In the context of the Work Programme, for example, this might involve modelling the determinants of the transition from benefits to work to provide a set of leading, rather than lagging, indicators (logistic regression http://associates.infonomics.ltd.uk/?q=node/35 or maybe neural nets).

Clearly if there's insufficient data in the first place one might wish to consider a soft-systems method whereby a logic model is used to link bottom-up observations (outputs) with top-down objectives (impacts). This provides an intuitive framework for management that can gradually be supported by evidence as the data becomes available. More on impact evaluation: http://associates.infonomics.ltd.uk/?q=node/45.

More generally, I believe this problem arises because we find ways of analysing the data that we have rather than finding data for the analysis we need. If performance management focused on informing decisions rather than documenting "results" then we would probably have a lot more confidence in our metrics.

[William]

James - this is of course a good point. But can we take it further? Frontline measures are good (is the hospital running efficiently) but customer side measures are better (how long does it take the patient to get into the system in the first place?) Let's improve public services using metrics for people, not metrics for public services. Alan Mitchell has done a lot of excellent work on this.

[Stephen]

Are we measuring the right things in the right way?

http://ow.ly/2mRjt

[Alan Mitchell]

Hi James,
You raise some really interesting points. Here are some thoughts.

First, there is a fundamental difference between reporting metrics and operational metrics. Reporting metrics are aggregated, rear view mirror data - what Deming called 'accident statistics'. They tell you what's happened, but now why, or how to fix it. In most cases reporting metrics are worth than useless. But many organisations, because they are organised around hierarchies with people reporting upwards to their bosses, treat reporting metrics as if they were the only metrics that mattered.

Second, I'm not sure you're right to say that all metrics issues are ultimately 'people' issues. Often, the real issue is 'process' or, if you like, how well people are working and coordinating together. That's important, because you can never measure processes using the metric that most organisations focus on, which is money. You have to forget money, and look at the work and how it gets done. Here, time (especially time wasted) is often the key metric.

Third, most organisations' metrics are literally meaningless. They are measuring the wrong thing - an illusory artefact. Take 'cost'. You can't have something more real than 'cost' can you? Well, some costs are incurred providing value to customers. Those costs are good. And some costs are incurred but provide no value to customers. This is just waste. If you 'cut costs' without being able to distinguish the two, you end up cutting both value and waste at the same time.

The metric that really matters here is the customer's metric - what value looks like to the customer. But most organisations are far too narcissistic in their approach to take this into account.

Likewise (by the way) with income or revenues. Most organisations earn some of their money by adding value to their customers, and some by extracting value from their customers. By ripping them off or underserving them in some way. If you look only at your revenue or income figures, you cannot between distinguish the two. This is the worst possible starting point for sensible decision-making.

So I think you are right. Basically we need to start from scratch with metrics - and the best place to start (I think) is to understand which metrics matter to the customer/citizen. Everything follows from that.

[Pragmatist]

I've seen the Six Sigma methodology deployed to produce timely, accurate metrics that are very helpful in determining root causes of businness process problems and whether a business process is 'out of control'.

[Chris Hugo]

Hi James

Techstat in the US uses the opinions of the management board, powered by the IT dashboard (currently being worked on unuecessarily by Cabinet Office as we created it four years ago but lack one key data piece, i.e the traffic light openness of the CIOs).

The process allows the public to provide their own viewpoints on whether the service is working correctly and is value for money, with comment boxes to leave further details. That picks up on Alan's point, which is very timely during cost cutting at DWP. is it added value spend, or is it wasted money?Once evidenced, others can comment on whether they agree or disagree to whether that is a personalised view, rather than a collective view. The outcome then, would be a balanced scorecard, rather than a scorecard simply based on volume of opinion.

However, there is a much easier way to open this up and get a genuine, consensus poll of whether the status really is red, amber or green.

You mention in a later blog the importance of the human factor in assessing opportunities for improvement. Anonymised forums to allow users to flag problems, along with a 'rate this' button for others to understand , assess and vote on the comment made, can then be aggregated to create a genuine, rather than prismed view of where the service truly stands.

An example, a friend has recently been forced to use DWP. She applied for 2 benefits, having been rendered disabled. .

She needed two separate doctors assessments, which could not be combined, because of 'data protection issues'. At no point was she asked to relinquish her rights to data sharing to make the process, or DWP more efficient.

She insisted they use the other doctors report, which was identical, only to be told this could not be done. Estimated additional cost to DWP? The doctor was paid a fee of approx £30. Plus admin overheads, bill processing, postage, etc? Say another £125 (I did a time and motion study plus a data access request). So roughly £150. Multiplied by the number of times that those 2 benefits are jointly applied for each year and then require medical certification? Don't know yet as I am waiting for the data, but if the figure doesn't come to at least 7 figures, I would be amazed. Perhaps the worst comment was from the ESA telephone helpline, when asking for the data to be shared. "It comes out of our budget, for our Dr, they (the DLA)have their own budget, Drs etc.We wouldn't let them use something we had paid for, and neither would they pay for something that helped us".

Had they asked for permission to share that data, they would have found a willingness to speed the process, share the data, and accelerate the service. One extra question on top of the already voluminous forms perhaps? "In order to speed up our service and our claims management, may DWP share this data internally with our partners and agents?". You used to working in banking James, I am sure that phrase sounds familiar and made you life as an innovator much easier!

Those delays caused a human factor of a relapse due to the stress of having been a working, taxpayer forced into benefits that so far have taken almost 5 months to apply for, due to 'process issues'.

There are set targets for fulfilling claims. But if the service provider sends a letter, however meaningless and in no way moving the process along, then the timeline gets pushed back a further two weeks. Thsi of course creates revenue for the service provider, who appear to be on target. The DWP though are seen in a very bad light by the customer, who cannot understand how an SI makes money for nothing and damages the reputation of the service.

It's a crucial observation as to why the users should have the ability to comment on service, anonymously share user experience to inform the environment for service improvement and allow change to be driven not just from the inside, but from the outside facing in also. Only then might management know not only what the traffic light is for, but also why that opinion exists and how easily those areas could be fixed.