In defence of the ordinary

I think this might be a general cultural problem, but I notice it a lot in my field of healthcare analytics. There are lots of “case studies” and “pathfinders”, that kind of thing, groups of people who are doing amazing stuff.

I don’t think that it has the desired effect though. People look at these groups doing incredibly complicated things with new tools and they think it just doesn’t apply to them. Let the pathfinders pathfind and we’ll just churn out some rubbish in Excel, same as last week.

I wonder if we’d be better off setting the bottom higher, rather than raising the bar at the top again. Instead of showing off amazing teams, showcase a solid team getting the basics right, week in, week out. And try to pose the question “If you’re not doing this stuff, why not?”

In defence of looking at jobs

Nobody should ever have to apologise for looking at jobs, however settled they are. Looking at jobs just means one of:

  1. there might be a better job out there and I want to find it if there is
  2. I want to understand what skills people are recruiting so I can learn and do the right stuff
  3. I’m interested in the jobs and skills that are popular now because I want to understand the playing field of human resources in analytics as an actual or potential leader
  4. I want to understand my worth under agenda for change to I can advocate for the development of my professional role (and my salary :wink: ) in my current role

Any manager that doesn’t like that is a fool, a charlatan, a psychopath, or all three, and then you can add:

5. My line manager is a fool, charlatan, or a psychopath (or all three), and I am trying to leave my current job as fast as possible now I have realised this

In defence of holiday working

This might be a little bit controversial, this post. If you hate it please don’t be offended, it’s just my opinion. Totally open to being completely wrong always, especially here.

I’ve heard a lot of people talk about the importance of taking your leave, and encouraging your staff to take your leave, and I do think it’s super important to take leave, and I certainly think if you’re managing someone you should definitely encourage them to take their leave. There is more to life than work, and having your leave can even make you better at your job when you get back, so it’s win-win, you benefit and your job benefits.

However, I’ve had people tell me that it’s wrong that I sometimes work in my leave, because I’m setting a bad example to my staff. This I don’t agree with. Particularly now during COVID but really for the last decade my life has been limited by health problems (primary sclerosing cholangitis, chronic ulcerative colitis, and now hepatic arterial stenosis, but that’s another blog post) and my job has become something I can do that I love that I’m good at, replacing some of the other stuff I used to do (aikido and running, mainly). I’ve been isolating from my whole house on and off since March 2020 and my job really keeps me sane.

So although I may manage people I’m also a human being with my own emotional needs, and I choose to work sometimes in holidays and weekends because it makes me feel good to work and be productive. My whole team completely recognise my position and don’t at all feel any pressure to do likewise.

So I don’t agree with that. The other thing that I don’t agree with that I see sometimes is the idea that we should view others in our team or whom we manage working in the holidays as something “bad” that we need to “save” them from. The best boss I ever had was absolutely brilliant at taking leave. He would work 50+ hours a week and then in the holidays he would just disappear completely. No phone calls, no email, nothing. It was like he didn’t exist. And that was great, and I think it’s a really good example of doing something that works for you (ignoring the 50+ hours, not really advocating that, but he did work hard, put it that way).

But just like me I think some people blend their work in a bit more. They like to tickle their brain a bit in their holidays, maybe write some code that’s been bugging them, do a bit of reading, or write a blog post (writing blog posts absolutely should be part of your paid hours). It avoids the cliff edge stress of the last day before your holiday, because you can think “never mind, I’ll just knock that out in the middle of next week and email it over”. And if you then take a bit of time back for yourself in your core hours, that’s all good too. Because that’s one thing I am good at. I may work funny hours and at the weekend and in my leave but if I feel like having a three hour lunch break with a friend and nobody will miss me I do it without hesitation or guilt and that works really well for me.

So that’s it. I’m not saying “everyone should work in the holidays”, or even “you should try working in your leave, it’s great”. But I am saying that everyone is different, and some people like to blend their work and home lives together for lots of good reasons. As always, what’s important is understanding the needs of the people around us and respecting their autonomy and not coming up with social media friendly soundbites about what you should and shouldn’t do with your self or your team.

In defence of eating

One weird thing in remote/ pandemic times is that people have started turning their cameras off when they eat. I’m all for the right to turn your camera off, I think you should be able to do that at any time without giving a reason, but I think it’s a shame if people think that they can’t eat on camera. I have therefore been glad to eat several large, difficult to eat things in meetings on camera recently, to perhaps give others the idea that it’s okay if they want to do it too.

So far I’ve eaten a footlong Subway which was kind of falling apart really and today it was half of a pretty massive pizza which was also a bit lacking in the structural integrity stakes. So please join me if you wish, we all used to eat in front of each other in The Before Times, it’s totally natural and normal- but also if you don’t want your camera on because you’re eating or for any other reason that’s okay too.

In defence of dashboards

I’ve had lively debates about dashboards with various people, including someone in my own team, and somebody on Twitter just mentioned that dashboards are often not used (this blog post will be my response to this tweet, not the first time I’ve answered a tweet with a blog and very on brand for me 😉).

I should acknowledge at the top that I’m The Dashboard Guy. I’ve written books about making dashboards in R. It’s my role in the data science team in which I sit. Shiny in production. It’s my thing. So take all this with as much salt as you wish, I promise I won’t be offended.

People say that dashboards proliferate, and that nobody uses them. That is quite true in a lot of cases. I’d like to suggest why that is so, and talk about when people do use them.

The first thing to say is that in the NHS (which is where I have always and will always work) many staff are not engaged with data in general. They’re not engaged with data, they’re not engaged with analysis, they’re not engaged with reports, they’re just not engaged. They see data as a punishment, a “test” they cannot win. They can’t see the point of it and it’s just a distraction at best and over-critical and unfair at worst.

So my first question would be what do you replace the dashboards with? What will they engage with? I became The Dashboard Guy because we used to make a 300 page report by hand every quarter. It took literal person weeks and every recipient was only interested in their four page chunk. It was ridiculously inefficient. But everybody did want their four page chunk. Dashboards are useful for data that people want to see.

The team and I have recently built a classification algorithm for free text patient experience data. There is no way on earth anybody could use that without a dashboard because there are thousands of points of data and you are typically only interested in a few hundred at a time. So we built one. If they are using the algorithm at all, they’re using the dashboard (or someone else’s, it’s open source so you can DIY the dashboard if you want). If they’re not, that means they’re not looking at what their patient experience data is about, or they’re reading a 4000 row spreadsheet (and I do know people who have done/ are doing that).

So I think dashboards are very useful when you have a large highly structured dataset where everyone wants their own bit, and I’ve deployed perhaps three or four that have certainly been used by the individuals who wanted that data.

But something else that I think they can be used for is putting data science tools in the hands of your users. I just built a dashboard that allows you to pick how many LDA topics you want to use, and then shows you:

  • Term frequency for each topic in a graph
  • Five example comments from each topic

(I should say, it’s very rough and unfinished, but you get the idea). The idea of this is to allow the person who is interested in the data to make the topic model work themselves without writing R code. In this particular project all the data hasn’t come in yet anyway so the dashboard will contain more data in the future. It may be that the first tranche of data contains four topics, and the final dataset six. Building a dashboard makes the enduser part of that decision making process.

In fact, I actually build them for myself. A few years ago I was fitting a lot of structured topic models and it was such a pain in the neck fiddling around with the code that I just made a dashboard for myself so I could just sit and play around with it. The analysis itself took days, tens of hours, and the dashboard took an afternoon (to be fair, I’m probably a little faster than the average person with Shiny at least, because I have a lot of experience with it).

If your users are not using your dashboards, ask yourself why that is so. If they aren’t engaged with data or don’t care about the metrics then you have a bigger problem than dashboards. Would they engage with a report? An email? A bullet point in a team meeting? Build what they want, not what you think they want or should want.

Dashboards give you an opportunity to devolve analysis to your users. I work with a lot of text data. The users are the experts, not me. I want them to be able to do all the stuff I do in code but in a browser- sort, filter, summarise, aggregate, and even tune models (do they want five topics or fifteen? Do they want an overinclusive multilabel classification algorithm that captures all of a theme, or a conservative one that only shows them the highlights?).

In my opinion the dashboard is the icing on the cake. The team and I are building a very large, complex dashboard summarising many types of data, in the hope of driving customers between the types (patient experience data users looking at staff sickness levels, clinical effectiveness data users looking at all-cause mortality). But the work is in uniting the data, in producing analytics that are meaningful and useful. I don’t think it’s terribly important whether it’s a dashboard or a report or just played over the PA every Monday morning (there’s an idea 😉). Engage people in data and analytics first, and then serve that need using all the tools at your disposal.

Make statistics sexy

I’ve been enjoined to make statistics sexy. It’s really taking root in my brain.There are no easy answers in statistics; it’s a long, hard road. It’s rigorous, and honest, and that’s why I love it. Frank Harrell’s talk at NHS-R typified this. The price of truth can be very high- like binning all the data from your £1m study instead of doing what many do which is slice it into so many pieces that something looks shiny and publishing that.

At the same time there’s a lot of hype in ML and AI. There are serious minded people doing ML properly and making a real difference but there are also a lot of private companies selling the NHS snake oil in a new, fancy bottle and giving them overfit, janky, ungeneralisable models or even just selling them a pile of promises and a dashboard and leaving them with nothing.

How can the “I think you’ll find it’s more complicated than that” brigade compete with the “Our data science team showed that this model increased patient flow by 17%” (but please don’t ask us about pre-existing trend, or the Hawthorne effect, or the other 8 models they deployed that didn’t do much at all).

I truly have no idea, but I’m going to have a good go at finding out. Rest assured if I discover the answer I shall write it here first 😃

Data saves lives- and so does open code

I’m reading data saves lives. And look, it’s that thing that I say all the time, that I have pinned on my Twitter profile, the thing I’ve heard time and time again for a decade that everybody just stands around nodding at and nothing actually changes. I quote:

“Working in the open

Public services are built with public money, and so the code they are based on should be made available for digital pioneers across the health and care system, and those working with it, to reuse and build on.

Analysts should be encouraged to think from the outset of a project about how work can be shared, or consider ‘coding in the open’ for example, through use of open notebook science. This will include sharing of technical skills and domain knowledge through sites like Cross Validated and StackOverFlow, and sharing code and methodology through platforms like GitHub, will build high quality analytics throughout the system

Our commitments:

  • we will begin to make all new source code that we produce or commission open and reusable and publish it under appropriate licences to encourage further innovation (such as MIT and OGLv3, alongside suitable open datasets or dummy data) (end of 2021)”

My team does this, proudly. Does yours?

Pay and reward

I’m reading No Rules Rules, the Netflix management book. It’s really good, lots of interesting stuff in there about doing things differently and although clearly there’s a lot I can’t do in the NHS, there are lessons in there for sure.

I’ve just read the bit about paying people top of market rate and I’m so happy because it describes loads of people who are on $250,000, crazy money, moping around and leaving because they can get a 40% pay rise somewhere else.

How does 40% on top of $0.25m make you any happier? It’s a nonsense. But the people I work with and recruit don’t chop and change for money. They do what they do because it’s meaningful, and because they believe in the founding principles of the NHS.

So although I can’t pay top of market I don’t need to, either. I just need to ask people to do useful stuff, in the open, for everyone, and to do it their way with whatever support they need. And pay them properly, of course.

That I can do ❤️

That is, in fact, my jam 🎸🎸🎸

Graphs > tables

One phenomenon that I find very strange in the NHS (and elsewhere, probably, I’ve never worked anywhere else) is the obsession people have with having tables of numbers instead of graphs. I have encountered this absolutely everywhere. People really want to know whether something is 21.4% of the whole or 19.2% of the whole and they can’t tell by looking at the beautiful graph that you’ve drawn.

I saw an analysis today which had nine separate tables of proportions. I’m going to go out on a limb and say no human being can understand a thing of such complexity. Nine tables, each with three categories, 27 proportions given. You could fit the whole thing on one graph and it would be readily apparent how they compare with each other.

But no, people want to know is it 13% or 15%, even though in almost all cases the amount of precision far exceeds the confidence levels of the sample.

Your report needs to say “category A is found twice as often as C, whereas A and B are similar”. Not “category A is found 17.6% of the time, whereas C is found 9.2% of the time- on the other hand category B is found 19.5% of the time”. Just writing it is exhausting me, never mind trying to understand it from cold in a meeting.

There are of course rare exceptions to this rule, sometimes you really need to know that something is 13.5% of the whole. But you should be asking yourself more questions- how reliable is the measure? What is the sampling error associated with this estimate? Otherwise your 13.5% is 14.6% is 12.3%. And who is usually saying this, if anyone-me!

Stop punishing people with data

NHS data people know all about Goodhart’s law. First stated by Goodhart as the not very catchy

Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes

It was then popularised by Strathern as

When a measure becomes a target, it ceases to be a good measure

So far, so ordinary. What else do we all know? League tables are a bad idea too. Somewhat related to Goodhart’s law, the problem with league tables is that they often ignore nuance. "Bad" schools aren’t bad, they just have an intake that starts way behind. "Bad" surgeons aren’t bad, they just take on the more complex cases or work in a geographical location where there are more complex cases. There are ways of trying to balance out this bias, "value added" in the case of schools (being married to a teacher, I hear a lot about this one), and case mix adjustment in the case of surgeons. Although these methods go some way to ironing out problems there is always the chance that an unmeasured variable is affecting the results and distorting the picture.

In my own area of work, patient experience data, league tables are a terrible idea because the services are all so different and they serve so many different people. Older people are, on average, more satisfied with healthcare. People who are detained under the mental health act for what can be 10 years are, unsurprisingly, often less positive about their healthcare.

I would like to add another law of data to the canon.

The more data is used as a punishment the less engaged those it punishes will be with all data

I’ve read so many beautifully written and persuasive arguments about the power of data, all, naturally enough, written by True Believers. People who can see how data can be used to inform service delivery and planning. If I’m honest I think I live in an echo chamber, filled with analysts and technical types, all passionate about the insights data can generate. But sadly the reality outside my little bubble is that mostly people have data done "to" them. Every quarter a manager or public body drops out of the sky and asks them for all these numbers.

They’re punished if the numbers are late. They’re punished if the numbers are wrong

And they’re punished if the numbers don’t show them in a good light even if everybody knows there’s a perfectly good explanation as to why the data looks like that (thinking back to value added and case mix adjustment above).

They get it all done, feeling harried and that the way the data is collected and scrutinised is unfair and makes their team/ department look bad, they cross their fingers and pray that it doesn’t come back with red pen on it three days later, and then they forget about it for another quarter.

And the really sad thing is that these people have questions that could be answered with data. Everybody makes suppositions and hypotheses all the time, it’s impossible not to, and with help and understanding they could refine their ideas and be more effective at their job.

But data has crashed through their window like a rabid dog and made a mess of the place and it never occurs to them to take that same dog to the park to see if it can help them find the jacket they left there last week

They’re just glad to close the door and lock it and forget about the whole ordeal until the next quarter.

I think we do need to keep promoting the value of data and analytics, and I obviously spend a decent chunk of my time either selling people that dream or (more often) fighting Python dependencies on a server to make that dream a reality.

But I think it’s just as important to stop beating people with data, to try to work with them to understand why their numbers look the way they do, and to try to make all processes that involve data more to do with learning and less to do with punishing people with an unfair analysis of what really goes on in their department.