BankerVision

Why is the failure rate so high in innovation?

Why is it there's such a high failure rate in innovation efforts? 

Why is it I run around talking about our (the departments) 10% success rate on new ideas and think that's a good performance?

And why is it that other organisations are thrilled when they get even one new thing out the door?

I think the answer is we've all programmed ourselves to think this innovation thing is hard, and since it is hard, you should only be successful some of the time.

I also think all this programming we're giving ourselves has become something of a self-fulfilling prophesy.

I mean, the reality of large organisations is that most of the preconditions for success are already present. There is never a lack of great ideas, for example. Ideas are bubbling around everywhere, and all you have to do is collect them. Neither do large organisations lack people who want to get things done: getting magic to happen is really a question of linking them up and giving them permission to do things.

Most importantly though - large organisations are paying more than lip service to their desire to be more innovative. In the last week, for example, I learned that two more banks have started new innovation programmes complete with massive budgets.

So why, why, why does this stuff keep failing?

My experience at Lloyds with Innovation Market, and more latterly at the DWP where we've built largely the same thing (we call it IdeaStreet) suggests part of the answer, to me at least.

Innovation is a social activity.  People want to talk and share. They want to co-create. If you want people - of their own free will - to give you innovation, you must create a system that lets them link up with each other independently of any central effort and concentrate on the things they care about.

Why? I suspect potential innovators don't want to be little islands of greatness, trying to push uphill against the combined weight of everyone else trying to make things stay the same. That's very hard work; probably too hard most of the time, when you also have to do a day job.

When you have this situation, you only get innovation in the presence of individual heroics by very special, very rare people. Most organisations have too few of them to ever get reliability out of their innovation efforts. And, at the very best, you're only going to get one or two really meaningful things before your heroes burn out.

This, then, is the cause of all the failure: all those little islands having to push against the big corporate machine. You hear so many stories where even those at the top of organisations can't get anything done because all the machinery at lower levels frustrates their innovation efforts.

I say, if even those at the top of an organsiation can't mandate innovation, then you have little choice but to start innovation from the bottom up.

That's why building social innovation systems are important.  People down the pecking order are very powerful agents for change when they band together to do something.  And its the reason why central innovation efforts that only invest in big game changing things usually fail.

Is it OK for IT managers to let skills lapse?

Here is an interesting question: why is it in IT we allow our managers to let their specialist technology skills lapse?

That doesn't happen in other disciplines as much. In marketing, you can still talk marketing even if you're running a whole division. Otherwise, you can't run marketing.

If you're an economist, even if you are the Chief Economist in a bank, you're still an economist. Otherwise, you don't know enough to approve the forecasts on which the whole business is based.

And if you're a medical doctor, you keep up to date with the latest advances, particularly if you have to teach. Otherwise you lose your right to practice altogether.

Not so in IT, where we do something quite different.

You learn your skills, probably getting more and more technical as your skill set specialises. To get ahead, you have to be the best person in some particular discipline. World best technologists are treated almost as if they're rock stars, commanding big audiences whenever they can be convinced to appear in public. Getting ahead in IT is a race to specialism, as it is in any other industry.

But then we get to that inflection point where we ask people in IT to become managers. That's when things change. What happens then is you have a race to be as generalised as possible. The more generalised you get, and the larger the number of non-technical functions you get hold of, the more successful you are. Scope of control is the measure of success for an aspiring IT Manager.

Suddenly, its OK to lose your knowledge of the current state of the art. It's OK to have to rely on less experienced people to make critical decisions because you're no longer current enough to make them yourself. It is almost a badge of honour, in fact: managers who are too technical aren't seen to be effective, because they haven't risen above their roots.

Can you imagine all that being OK if you were a senior medico?

So here's the question I'm posing today: should we allow managers to lack currency in a discipline that touches almost all others in some way?

I know the standard argument: the pace of change is such that its impossible to keep up to date in various technologies and still do the day job. 

There's another standard argument as well: you don't need technical knowledge to effectively manage IT staff.

But I'd like to make an observation: many of the critical decisions which relate to major IT estates in large organisations both private and public are made behind closed doors by managers, who probably don't have access to their staff at the precise moment they are called on to decide something. Nothing unusual about that: quite often critical decisions can't be shared widely for lots of sensible reasons.

But this means managers either have enough residual knowledge to guess at the correct answer, or they guess at the answer regardless of correctness.

So, is it OK for this to happen? I'd be interested in your thoughts.

Why metrics are broken

Large organisations are great at creating metrics for things. They always have to have dashboards and scorecards and other little devices that let them monitor how they're doing on so many different fronts.

That's especially true for IT organisations, at least, any that are big enough that you can't just ask everyone involved how things are going.

Governance people, especially, love metrics and scorecards. 

The problem with metrics is they aggregate information to such a degree that you often know there is a problem, but not what it is. You'll see, for example, some scorecard with a red traffic light against a project, so you know something is going wrong, but you have no idea at all how to fix it.

Fixing it requires deep diving into human conversations with the people involved. Metrics are pale reflections of the people-reasons things are breaking. And, of course, it is always people reasons why things go wrong, when you boil everything down.

My problem with this whole process is that the aggregate data you get on scorecards always lags reality by a significant degree because the metrics collection process is incredibly flawed.

Metrics collection is based on the premise that people with the most to lose when something goes wrong will admit they are in trouble soon enough for management to help them. By the time the scorecard reflects reality, it is invariably because the people involved can no longer hide what's broken - they may, for example, have missed a delivery date or had to request more money.

This, actually, is a problem with all systems where the reporting process is based on top down self-reporting.

I think it is time for a new approach to this. Why can't we have reporting systems that go directly to the human front line and aggregate up? Why can't everyone in a project team be polled on what they think the status is and use that as the reporting metric? At least it would be more honest and be much closer to the human conversations which provide the solution.

Certainly, I've never worked in an organisation that manages its scorecards in this way. Has anyone got examples they'd care to share where this kind of thing has worked?