Medical Director Cole Stanley continues his journey though the Circles of Healthcare Data Hell with Circle One. You can find the other circles here

Seeing a rhino in the clouds, a looming set of eyes on the floor tiling, or Jesus Christ himself on a grilled cheese sandwich? Now down to our next circle, where much like the first, our brains are up to no good. This time though, starved for causal explanations for what we are seeing, we identify patterns amongst randomness. And again the real problem happens when we use these to move to judgement and decision. How common is it for us to overcall variation we see in the data? Was that 2 pounds weight loss really because of the new diet you started, or is that change just part of the normal day-to-day fluctuations you experience? Or a throwback to 2020-2021: the news is telling me COVID-related deaths are up today vs. yesterday, and they are implying things must be getting worse. Well not necessarily, because day-to-day counts occur within the complex adaptive system of healthcare, where we expect variation. To respond to this, the Institute for Healthcare Improvement (IHI) started using control chart methodology to detect meaningful change in the numbers, but this remains mostly beyond that of the mainstream approach to data [1].

In medicine, we are well-versed in research methodologies in which we compare our data before and after an intervention to detect statistical significance. But this before-and-after approach can lead us astray. I may attribute my 10% decrease in No Show rate this week to the appointment reminder emails we sent, while missing the fact that the No Show rate normally varies week to week by much more than this. I’ve hallucinated the effectiveness of what I did.

Again, the CDSC helps us out. With the guidance of our QI-trained physicians, we are careful to not overcall the changes we see in the data, and we value looking at data over time. We use the same question when looking at changes in the data: “what could be the possible explanations for what I am seeing, and how can I test these hypotheses?”. Was this just normal “common cause” variation over time, or maybe it’s because the part of my system I am measuring is not yet stable enough to detect meaningful changes?

CDSC exaltations

  • Enlist support of people trained in QI methodology and understanding of variation (common vs. special cause)
  • Look at data over time (e.g. on a run chart) and avoid limiting to before-and-after

Return to the Healthcare Data Hell Table of Contents

 

1. https://www.ihi.org/Topics/COVID-19/Pages/COVID-19-Data-Dashboard.aspx