Big data: the perils of Goodhart’s Law


Big data. It’s suddenly being discussed everywhere.

Including York at the Big Tent event there a couple of weeks ago – when the celebrated NHS blogger Roy Lilley chaired a panel discussing it.

And he followed up with a post which we published here as a blog – and which we used as the basis for a Friday forum on LinkedIn.

It was then that I remembered the huge significance of Goodhart’s Law for big data – and any other form of tickbox.

This is how it began.

By the early 1970s, most countries had begun to adopt monetary targets to regulate how much money was going into the economy, which was not an easy task in a market where banks were creating money in the form of loans whenever they felt like it.

They would track ‘broad money’ or ‘narrow money’ – each one a different interpretation of how much money there was and in different combinations of cash or bank deposits.

As the man who became Sir Charles Goodhart – then a monetary economist at the Bank of England – was going through the figures that came across his desk, as they did from most countries every month, he began to notice something peculiar.

It didn’t matter which country it was or which target they chose – the act of making it a target seemed to change it. Whatever monetary target was chosen, because of its relationship to GDP, would lose that relationship pretty quickly. It was certainly not what was supposed to happen.

This provided the bones of Goodhart’s Law. It was clear that the very act of making one figure a target and paying attention to it changed the behaviour of bankers. They would put in extra effort, or interpret their work in slightly different ways, to please their superiors and to meet the target.

It was this quirk of monetary theory that Goodhart pointed out in Sydney in 1975.

By doing so, he guaranteed a place for himself in the encyclopaedias, and he also upstaged the American economist Robert Lucas – though he did not know it at the time – because Lucas was in the process of setting out in the traditional academic format what became the Lucas Critique, which appeared in 1976.

Lucas said something similar: if there is a relationship between two variables, and this relationship depends on policy or authority, then changing the policy is likely to change the relationship.

So Lucas took the credit in economics, but by then Goodhart’s Law was being studied in social policy instead.

Perhaps that is right and proper because it is really a statement about how people behave – so that, if you use a piece of data as a target or as a box that must be ticked, then the data will become inaccurate.

Just as exam results tend to improve over time, because teachers teach to the test, so target figures tend to get better too, because frontline staff learn – not necessarily how to fake it – but how to make the numbers look better, how to burnish their reputations. How to massage the figures in the desired direction.

“You never know which bit of your work will be influential or widely read,” says Goodhart now, when I interviewed him for my book Tickbox (now only £1.99 on kindle!). “The fact is that I’m better known for a throwaway line in a speech than I am for maybe hundreds of more serious academic articles.”  

That may be so, but Goodhart’s Law is a profound thought and it has huge implications for tickbox and therefore for data, big and small. Because it is not just right that the act of setting out the tickbox that needs to be ticked skews the data, it also skews everything else.

What Goodhart’s Law is unable to do is to work out how much the data has been skewed, or how much time and public money has been wasted skewing it. It has been assumed over the years that it points out a small peculiarity, but actually the implications may be huge.

Just imagine for a moment that every frontline public servant, and their bosses, have had their judgement skewed and have changed the way they work to meet the target, rather than what their professionalism judged they should do.

Have they wasted five or ten percent of their time, or five or ten per cent of their professionalism has been suborned to turn the service into more of a machine – taken across the whole public sector, that is a great deal of money.

It may be much more. Why after all, should we employ professionals at professional charities to deliver cookie-cutter services by numbers? It is also fascinatingly difficult to avoid.

Therein lies the power of Goodhart’s Law. There appears to be no escape from its effects. And if you try to escape from it, it simply turns more of your system into the kind of judgement-free machine that tickbox seems to prefer – and all in the name of transparency.

Rate this post!

Average rating 0 / 5. Vote count: 0

No votes so far! Be the first to rate this post.

Radix is the radical centre think tank. We welcome all contributions which promote system change, challenge established notions and re-imagine our societies. The views expressed here are those of the individual contributor and not necessarily shared by Radix.

Leave a Reply

The Author
Latest Related Work
Follow Us