(Cross-posted to the AAN’s stroke listserv)
For your new year’s reading pleasure, I have a very interesting book recommendation: Superforecasting: The Art and Science of Prediction, by Philip Tetlock. Some of Prof. Tetlock’s academic research made news when he showed that political pundits have the forecasting accuracy of “dart-throwing chimps”. On the other hand, the top forecasters in his online forecasting study outperformed CIA agents in predicting geopolitical events (you can join the fun by clicking on the preceding link and weighing in on matters such as, “Will Russia conduct a naval exercise in the Western Hemisphere with a Central or South American country before 1 July 2016?”). Superforecasting provides a broad survey of what makes a well-crafted prediction versus one that is so loosely worded that the person making it can always argue that he was right, how we weight evidence as it comes in and make and update our predictions, how to generate objective data (called Brier scores) on our predictive abilities, and how a rigorous approach to forecasting can be applied in various fields.
It occurred to me that it would be desirable to have a forecasting app that would allow docs to make clinical predictions in real time and then see, over time, how well we fare as forecasters and whether we can use the feedback to improve our performance. Example: A young patient comes in with a left brain TIA due to carotid dissection. There’s a tight stenosis and MR perfusion is strikingly abnormal. The patient has had only one clinical event, so the Stroke Association guidelines would support antiplatelet or anticoagulant therapy, but some members of the team suggest that in this particular case, the imaging data are so compelling as to warrant carotid stenting. The final decision is to stick with antiplatelet therapy, but some remain nervous about the patient’s short-term prospects.
A poorly-crafted forecast would be, “I really think that this man might have a stroke in the near future unless we revascularize him”. This might sound like a strong prediction, but if the patient doesn’t have a stroke, the person can always say, “Well, I just said that he might have a stroke . . . ” A better-crafted question would be: “How likely is it that Mr. Smith has another TIA or a stroke referable to the left carotid territory within the next week?” Everyone on the team could assign a probability to that outcome. Those who are certain that the patient is destined to have a stroke might assign a probability of 90%, but if it doesn’t happen, that person’s Brier score would take a big hit, and vice versa.
Of course, there would be some challenges with outcome ascertainment, maintaining confidentiality, etc., but for the sake of discussion, we can assume that these are surmountable problems (and that we wouldn’t bother with predictions that would be very hard to follow up on). Importantly, the goal wouldn’t be to answer PICO questions such as whether medical or interventional therapy is better for cases like that above–that’s what clinical trials are for, when feasible. Rather, the idea is to have a mechanism for individual doctors to obtain ongoing quantitative feedback about their predictive abilities in every day clinical practice, with all the nuances and tough judgment calls that that entails. This might be especially valuable for residents, and the data could even feed into their Practice-Based Learning and Improvement milestones.