Medical Director Cole Stanley begins his journey though the Circles of Healthcare Data Hell with Circle One. You can find the other circles here

If a picture contains a thousand words, then you could say a graph of healthcare data might contain a thousand explanations.
In this first circle, when it comes to the data, judgements swirl, stories are spun, and the fundamental attribution error is in full effect (where we discount contextual factors and blame the individual for the results we see). This is a fitting first circle, right below the surface, since it’s so easy for our human brains to slip to this level. Remember, when that truck on the freeway cuts you off, making you miss your exit, there are many potential explanations for the data you are seeing, and only a subset includes the fact that can feel so certain of in your mind– “that guy is an @$!%#!!” (from firsthand experience, I’d suggest a cooldown period before reminding your road-raging partner of this). Similarly with healthcare data, we can quickly jump to an explanation for what we are seeing, then double down on it with confirmation bias and fail to consider context and other possible explanations. I asked GPT-4 to help me demonstrate, with eerily plausible output:

COLE: A graph of data from a primary care clinic shows that the proportion of the patient panel with diabetes and an A1c test result within target range has been decreasing over the few years it has been measured. From the perspective of a different individuals in the list below, give a few examples of possible explanations they might have for seeing this. Use direct quotes from these individuals: program funders, diabetic patients, patient family members, physicians, front desk staff, nurses, and an online troll looking at the graph

GPT-4:

Program Funders

  1. “It appears there’s a decline in the quality of care provided, which is concerning considering the resources we’ve allocated. Perhaps the clinic isn’t utilizing funds effectively.”
  2. “This graph shows that our investments in diabetes management aren’t yielding the expected results. Maybe it’s time to reevaluate our funding strategies and potentially invest in alternative approaches.”

Diabetic Patients

  1. “I’ve been coming here for years, and it feels like the doctors are always rushed now. Maybe that’s why fewer people are reaching their A1c targets.”
  2. “They started charging for parking recently. It might not sound like much, but it’s another hurdle to coming in for regular checks.”

Patient Family Members

  1. “My mom has to wait forever to get an appointment. By the time she’s seen, her blood sugar’s all over the place.”
  2. “They hardly ever follow up with us about test results. We have to chase them down, and by that time, it’s too late to adjust medications properly.”

Physicians

  1. “We’ve had an increase in new patients with already advanced diabetes; it skews the overall control numbers down.”
  2. “With the increasing administrative demands, we’ve had less time to focus on individualized patient care, affecting our performance metrics.”

Front Desk Staff

  1. “People have been missing their appointments more frequently. Maybe they’re going somewhere else, or they’re just not as committed to their health as before.”
  2. “We’ve had issues with the appointment scheduling software, causing double bookings and delays. That probably affects patient care.”

Nurses

  1. “Our staffing levels haven’t kept up with patient numbers, so we’re spread pretty thin. That could be affecting patient outcomes.”
  2. “We switched to a new brand of A1c test kits recently; maybe they’re not as accurate.”

Online Troll Looking at Graph

  1. “Ha, looks like this place is going down the drain. Good luck getting any decent care here!”
  2. “These guys can’t even keep their diabetics healthy. Must be a money-making scheme rather than healthcare.”

The scary part happens when these explanations turn into judgements that prompt decisions. Funding could be cut, patients could switch clinics, and real improvements in care could be cast aside, masked simply because they are unseen with the data as presented. To avoid this, we must look for context, and think about how to test alternative hypotheses.

This is where the CDSC comes in. As a group including frontline experienced physicians, we have seen how lack of context can have disastrous effects, and so a core value of the group has been that we promote “context dependence”. In other words, we take steps to protect the data from being inappropriately judged. This includes a repeated message that our measures are best used as part of a suite, and that clinicians who know the clinical context agree that this suite of measures can be a useful tool for measuring improvement. We debate how widely available the measures should be and worry about the risks of certain audiences reviewing the results without proper context. We promote an exploratory approach, where the measures are initially reviewed with the question “what could be the possible explanations for what I am seeing, and how can I test these hypotheses?”

This team is a unique arena for physicians to develop nuanced understandings of the challenges of metrics selection & data stewardship in healthcare. ~Dr. Lawrence Yang, CDSC member

CDSC exaltations

  • Remember that context matters – involve people close to the problem
  • Test alternative hypotheses for what you are seeing
  • Don’t overfocus on a single measure, use a suite
  • Be careful about presenting data to audiences less familiar with the context

Return to Healthcare Data Hell Table of Contents