Medical Forecasting

(Cross-posted to the AAN’s stroke listserv)

For your new year’s reading pleasure, I have a very interesting book recommendation: Superforecasting: The Art and Science of Prediction, by Philip Tetlock. Some of Prof. Tetlock’s academic research made news when he showed that political pundits have the forecasting accuracy of “dart-throwing chimps”. On the other hand, the top forecasters in his online forecasting study outperformed CIA agents in predicting geopolitical events (you can join the fun by clicking on the preceding link and weighing in on matters such as, “Will Russia conduct a naval exercise in the Western Hemisphere with a Central or South American country before 1 July 2016?”). Superforecasting provides a broad survey of what makes a well-crafted prediction versus one that is so loosely worded that the person making it can always argue that he was right, how we weight evidence as it comes in and make and update our predictions, how to generate objective data (called Brier scores) on our predictive abilities, and how a rigorous approach to forecasting can be applied in various fields.

It occurred to me that it would be desirable to have a forecasting app that would allow docs to make clinical predictions in real time and then see, over time, how well we fare as forecasters and whether we can use the feedback to improve our performance. Example: A young patient comes in with a left brain TIA due to carotid dissection. There’s a tight stenosis and MR perfusion is strikingly abnormal. The patient has had only one clinical event, so the Stroke Association guidelines would support antiplatelet or anticoagulant therapy, but some members of the team suggest that in this particular case, the imaging data are so compelling as to warrant carotid stenting. The final decision is to stick with antiplatelet therapy, but some remain nervous about the patient’s short-term prospects.

A poorly-crafted forecast would be, “I really think that this man might have a stroke in the near future unless we revascularize him”. This might sound like a strong prediction, but if the patient doesn’t have a stroke, the person can always say, “Well, I just said that he might have a stroke . . . ” A better-crafted question would be: “How likely is it that Mr. Smith has another TIA or a stroke referable to the left carotid territory within the next week?” Everyone on the team could assign a probability to that outcome. Those who are certain that the patient is destined to have a stroke might assign a probability of 90%, but if it doesn’t happen, that person’s Brier score would take a big hit, and vice versa.

Of course, there would be some challenges with outcome ascertainment, maintaining confidentiality, etc., but for the sake of discussion, we can assume that these are surmountable problems (and that we wouldn’t bother with predictions that would be very hard to follow up on). Importantly, the goal wouldn’t be to answer PICO questions such as whether medical or interventional therapy is better for cases like that above–that’s what clinical trials are for, when feasible. Rather, the idea is to have a mechanism for individual doctors to obtain ongoing quantitative feedback about their predictive abilities in every day clinical practice, with all the nuances and tough judgment calls that that entails. This might be especially valuable for residents, and the data could even feed into their Practice-Based Learning and Improvement milestones.

  1. The idea of medical forecasting is really challenging and interesting and getting more focus now in neurocritical care but under the name ” providing prognosis ” !!! and we all know that we are horrible at it . but its kinda the same concept, although studies support that we can prognosticate who does bad better than our ability to say who can do well . But looking back and provide feedback to improve our ” medical forecasting ” some times is challenging .
    Interesting article Dr. Sattin .

    • The case of Stephen Hawking is a great example of what you said in your essay–what’s true for a population doesn’t necessarily hold for the individual. Stephen Hawking might be a medical outlier, which actually wouldn’t damage the idea of population-based predictions too much. The harder problem is that he may have otherwise typical ALS, but because of the essentially unlimited resources he can bring to bear and his own desire to bring them to bear (rather than transition to comfort-only measures), he’s seen much longer survival than average. Stephen Jay Gould famously articulated this:

      The choices we make as patients, doctors, and society can have a large influence on the prognosis of a disease as its underlying biology. We see this all the time in neurocritical care, where the self-fulfilling prophesy in cases of intracranial hemorrhage is now well-known. This point is also relevant to the current enthusiasm for outcome-based quality reporting and reimbursement. For example, the risk of hospital re-admission is inversely correlated with socioeconomic status. Another example: It looks better for a stroke program if it can send a patient to acute rehab rather than to a SNF. But often, the determination of which disposition is most appropriate depends not on characteristics of the disease or our treatment of it, but rather on the patient’s social support. If the patient will have ample family supervision at home upon d/c from rehab, then to rehab he goes. But if not, then he’ll need to go to a SNF.