Medical Director Cole Stanley continues his journey though the Circles of Healthcare Data Hell with Circle Three. You can find the other circles here.

Our next ring down is rife with strange behaviours. Elementary school teachers are furiously erasing wrong answers from their pupils’ standardized tests, inflating scores for their school. Combat athletes are making use of the hellish temperature, sweating buckets in their garbage bag outfits, ensuring they hit their weight class. Wannabe social media influencers are being over-the-top, outrageous, and even self-harming, all in the name of getting more likes and follows. Social media companies hack human cognition to keep their users mindlessly scrolling, in the name of more “engagement” (read ad revenue). Local residents in an area where cobras are real pests have actually started breeding them, so they can collect more of the bounty money offered by the colonial government. What’s going on here? Its measurement gone wrong again, as we’re assigning some value judgement to our numbers, which can unleash human creativity (less charitably, some truly perverse behaviour). This has sometimes been called The Cobra Effect, named after the example above that actually happened when the British government did this during India’s colonial rule [1].

Before we start measuring something, we usually have an underlying purpose in mind. The measure is a means to an end, but too often it can become an end in itself, where we forget about the underlying purpose. We are reduced to a puppet controlled by the all-powerful measure. Healthcare is rife with examples of where measures have led to perverse behaviour. Measuring and rewarding primary care with fee-for-service leads to quicker lower quality visits, as we assign value to volume. In acute care, incentivizing reduced length of stay can lead to increased readmissions and excess pressure to discharge before it’s safe. Paying for performance on quality metrics like A1c control in diabetes can overfocus generalists’ efforts on this problem, leaving too little time left to manage problems that are less easily measured but more important to the patient in front of them. We focus on the problems we can easily see, and if we measure something it usually brings the problem into the spotlight. The low priority easily measurable problem supplants the more important yet more difficult to measure problem (similar to the idea in Monkeys and Pedestals below).

CDSC guidance on our use of measures can again help us out, preventing some of this perversion. We view our measures as tools to help those closest to the problem (including patient voices) do QI to improve, rather than as something primarily used to assign value to our work and get us extrinsic rewards (e.g. praise, funding, etc). In QI, we stay focused on the underlying problem and purpose of the work and use balancing measures for early detection of perverse behaviours. Discussion of outcomes needs to go beyond whether we hit our numeric goals (see Human Targets below) or got some extrinsic rewards from our QI work. Instead we focus on team learnings and impacts (quantitative AND qualitative) on quality of care for our patients.

CDSC exaltations

  • Use QI methodology and a suite of measures
  • Prioritize QI efforts with high potential impacts, instead of those easiest to measure
  • Our measures are primarily tools for improvement work, and not meant to be the goals themselves
  • Consider how you could “game the system” to get your outcome measures to improve without actually fixing the problem you want to, then add balancing measures to detect inklings of this
  • Beware of extrinsic motivating factors and separate out performance evaluation from QI (see The Wrong Toolkit below)

Return to Circle of Healthcare Data Hell Table of Contents

[1] Bad Data: Why We Measure the Wrong Things and Often Miss the Metrics that Matter by Peter Schryvers (Prometheus 2020)