Medical Director Cole Stanley continues his journey though the Circles of Healthcare Data Hell with Circle Four. You can find the other circles here.

Next to this circle that’s somewhat of a subset of Perversion above, where we set numeric goals or “targets”. The classic QI example of weight loss fits here, so let’s call up some imagery of weight loss class from hell, a group of 10 individuals of various genders, ages, shapes, and sizes with elevated BMI. This group of motivated people believes in the power of social accountability, and so have come together to collectively lose weight. In discussion, most agree the underlying purpose of their efforts is to benefit their long-term health, but this is difficult to measure, so they choose the group’s total pounds lost as their surrogate outcome and want to lose 100lbs by 6 months from now. Not only that, but the group’s coach will also get a monetary bonus if they reach their goal. Cue the human ingenuity. The coach decides to focus on the most overweight individuals to get the most efficient losses, and others are left with no attention paid. The group chooses extreme calorie restriction diets, and avoids strength training, since they worry that putting on muscle mass will get in the way of their goal. A participant is shunned after coming back a few pounds heavier from an Alaska cruise, and decides to stop coming back to the group. When that happens, it’s only 4 months in, but it’s celebration time. They’ve already hit their goal! The coach gets his bonus, and the group feels the pressure is off. Soon after though, people stop coming back. In general, their diets and losses aren’t sustained. Some individuals lost muscle mass and one fell and broke their hip soon after. In sum, the numeric goal was reached, but we got almost the exact opposite of what we aimed to improve.

This rather extreme example is used to illustrate that reaching the numeric goal should not be the only thing we consider, and that setting these goals at all can be fraught. Though we may see real improvements, we can also perversely incentive behaviour we don’t want. This can include distortions to processes that aren’t real improvements (e.g. losing weight by starving), distortions to the measurement (e.g. changing time of day of weight check), and some nasty human tendencies to shame and blame. In addition, we can leave some improvements unrealized if the goal is hit early, or we can lose motivation and start to feel hopeless if it looks like we might not reach our goal. Instead of continuously improving, we can also see “sandbagging”, where we intentionally lower expectations and coast, allowing us to set lower targets and achieve “better than expected” results.

It’s interesting to me that one of the founders of QI thinking, W. Edwards Deming, was not a fan of numeric goals.

…if management sets quantitative targets and makes people’s job depend on meeting them, “they will likely meet the targets – even if they have to destroy the enterprise to do it.”

W. Edwards Deming, quoted in Profits Beyond Measure by H. Thomas Johnson (in the forward to the book)

We see a phenomenon is sometimes known as Goodhart’s Law, from when it was originally noted in an Economics context that when a measure becomes a target, it ceases to be a good measure. It’s difficult to square this in my mind given how central our SMART (specific, measurable, achievable, realistic, time-bound) aims are to the Model for Improvement at the centre of QI methodology. I’ve now come to a point where I still believe SMART aims are useful, but we need to be more mindful of their pitfalls and not forget the underlying purpose of our improvement efforts. And we need to remember that a QI project that does not meet its’ numeric goal should not be called a failure, as we neglect learnings and improvements that may have been made.

CDSC exaltations

  • Remember that hitting a numeric target is not the primary purpose of your efforts
  • Beware of targets tied to extrinsic rewards, as they can especially incentivize perverse behaviour
  • Value continuous learning and improvement over hitting numeric targets

Return to Circle of Healthcare Data Hell Table of Contents