NHS data people know all about Goodhart’s law. First stated by Goodhart as the not very catchy
Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes
It was then popularised by Strathern as
When a measure becomes a target, it ceases to be a good measure
So far, so ordinary. What else do we all know? League tables are a bad idea too. Somewhat related to Goodhart’s law, the problem with league tables is that they often ignore nuance. "Bad" schools aren’t bad, they just have an intake that starts way behind. "Bad" surgeons aren’t bad, they just take on the more complex cases or work in a geographical location where there are more complex cases. There are ways of trying to balance out this bias, "value added" in the case of schools (being married to a teacher, I hear a lot about this one), and case mix adjustment in the case of surgeons. Although these methods go some way to ironing out problems there is always the chance that an unmeasured variable is affecting the results and distorting the picture.
In my own area of work, patient experience data, league tables are a terrible idea because the services are all so different and they serve so many different people. Older people are, on average, more satisfied with healthcare. People who are detained under the mental health act for what can be 10 years are, unsurprisingly, often less positive about their healthcare.
I would like to add another law of data to the canon.
The more data is used as a punishment the less engaged those it punishes will be with all data
I’ve read so many beautifully written and persuasive arguments about the power of data, all, naturally enough, written by True Believers. People who can see how data can be used to inform service delivery and planning. If I’m honest I think I live in an echo chamber, filled with analysts and technical types, all passionate about the insights data can generate. But sadly the reality outside my little bubble is that mostly people have data done "to" them. Every quarter a manager or public body drops out of the sky and asks them for all these numbers.
They’re punished if the numbers are late. They’re punished if the numbers are wrong
And they’re punished if the numbers don’t show them in a good light even if everybody knows there’s a perfectly good explanation as to why the data looks like that (thinking back to value added and case mix adjustment above).
They get it all done, feeling harried and that the way the data is collected and scrutinised is unfair and makes their team/ department look bad, they cross their fingers and pray that it doesn’t come back with red pen on it three days later, and then they forget about it for another quarter.
And the really sad thing is that these people have questions that could be answered with data. Everybody makes suppositions and hypotheses all the time, it’s impossible not to, and with help and understanding they could refine their ideas and be more effective at their job.
But data has crashed through their window like a rabid dog and made a mess of the place and it never occurs to them to take that same dog to the park to see if it can help them find the jacket they left there last week
They’re just glad to close the door and lock it and forget about the whole ordeal until the next quarter.
I think we do need to keep promoting the value of data and analytics, and I obviously spend a decent chunk of my time either selling people that dream or (more often) fighting Python dependencies on a server to make that dream a reality.
But I think it’s just as important to stop beating people with data, to try to work with them to understand why their numbers look the way they do, and to try to make all processes that involve data more to do with learning and less to do with punishing people with an unfair analysis of what really goes on in their department.