SUPERFORECASTING
Who Keeps Score of the Experts?
Tomorrow, an expert will tell you what the future holds. Open a newspaper, turn on television, or scroll through social media. Someone with impressive credentials will confidently predict what is going to happen. Economic collapse, climate catastrophe, international conflict, the next great disaster. The warnings arrive wrapped in elaborate explanations and frightening certainty.
The prophets of catastrophe have been spectacularly wrong before. In 1968, biologist Paul Ehrlich warned that hundreds of millions would starve during the 1970s. The authors of The Jupiter Effect predicted devastating earthquakes associated with a planetary alignment in 1982. One widely publicized scientific prediction suggested that the Arctic might be ice-free in summer by 2013. None of those specific forecasts came true.
That doesn’t mean famine, earthquakes, or climate change are imaginary dangers. Arctic sea ice has continued to decline. But a genuine danger does not make every prediction about its timing or consequences correct.
And mistaken authorities are not confined to predicting catastrophes. For decades, official dietary advice urged Americans to reduce their consumption of fat. Subsequent research revealed that the type of fat—and what replaced it—could matter far more than simply eating less of it.
Doctors once blamed stomach ulcers largely on stress and lifestyle. Then two Australian researchers, Barry Marshall and Robin Warren, demonstrated that a bacterium, Helicobacter pylori, caused most peptic ulcers. Their discovery earned them the Nobel Prize.
Wall Street’s experts pronounced thousands of mortgage-related investments exceptionally safe. Prestigious credit-rating agencies awarded them AAA ratings. Many subsequently suffered disastrous losses, contributing to the financial catastrophe of 2008.
And yet we continue to place extraordinary faith in the “expert.” Governments consult him. Corporations hire him. Television puts him before millions. We admire his credentials, trust his confidence, and repeat his conclusions.
But perhaps we’re confusing two very different things.
There’s the “expert” whose authority rests upon credentials, and the expert whose competence has been demonstrated. An astrologer may spend a lifetime mastering elaborate charts and calculations. Does that make his predictions accurate? Compare him with a surgeon who has performed a thousand successful operations. One has mastered a system of explanations; the other has a record of performance we can investigate.
Even a surgeon’s record requires examination. Difficult cases cannot be compared blindly with routine ones. But at least we’re asking the right question: What has this person actually accomplished?
We have been taught to ask, “What do the experts say?” How often do we ask, “How often were they right?”
Suppose we recorded their predictions: what they said, when they said it, and what actually happened.
How accurate do you suppose they would be?
Philip Tetlock decided to find out.
The Dart-Throwing Chimpanzee
The results were enough to make a chimpanzee famous. Over approximately twenty years, psychologist Philip Tetlock investigated the forecasting abilities of 284 specialists in political and economic affairs, examining more than 80,000 predictions against subsequent events.
On average, the experts performed only a little better than a hypothetical chimpanzee throwing darts at possible answers.
Imagine that! Years of university education, distinguished careers, impressive credentials—and a dart-throwing chimpanzee almost keeping up!
Of course, the chimpanzee was a metaphor for random guessing, not an actual participant. And the results were more nuanced than the joke suggests. Experts sometimes demonstrated genuine forecasting skill, particularly over shorter periods.
Even a broken clock is right twice a day. That doesn’t make it a timekeeper. An occasional correct prediction tells us little about the quality of someone’s judgment. We need a record, including the failures.
A baseball player cannot hide his batting average. Why should an expert hide his forecasting record? We evaluate athletes by performance. Yet a celebrated commentator may make confident predictions, explain away repeated failures, and return to television with undiminished authority.
We reward confidence and persuasive explanations, even when we have little evidence that the storyteller can predict what will happen.
But Tetlock’s most important discovery wasn’t that experts could be wrong.
It was that some people were considerably better at forecasting than others—and much of the difference lay in how they thought.
The Fox and the Hedgehog
The hedgehog knows one big thing. The fox knows many things. This ancient distinction became one of Tetlock’s most revealing findings.
The hedgehog possesses a grand explanation of the world. Economics, politics, history, human behavior—everything fits into an impressive intellectual framework. He speaks confidently, offers decisive answers, and seldom admits uncertainty.
Audiences love him.
But when events contradict his predictions, the hedgehog has a problem. Rather than reconsider his framework, he may explain away the evidence. The prediction wasn’t wrong; the timing was unfortunate. Unexpected events interfered. The theory survives.
The fox is less impressive—and often more accurate. Foxes examine competing explanations, borrow ideas from different disciplines, acknowledge uncertainty, and revise their judgments when the evidence changes.
They may be reluctant to announce what will happen. Instead, they consider alternatives and explain what might change their minds.
That doesn’t always make exciting television.
But Tetlock found that fox-like thinkers generally produced more accurate political and economic forecasts than their hedgehog counterparts.
What if the important difference isn’t intelligence, but the way we think?
And now the story becomes extraordinary.
The Amateurs Who Challenged the Intelligence Community
Ordinary volunteers, armed with public information, outforecast intelligence professionals who had access to government secrets.
Between 2011 and 2015, the U.S. intelligence community sponsored an ambitious forecasting experiment through its research agency, IARPA.
Researchers competed to develop better ways of predicting international events. Would a government fall? Would North Korea conduct another nuclear test? Would an international crisis escalate?
Tetlock and fellow researcher Barbara Mellers assembled the Good Judgment Project, recruiting volunteers from many different backgrounds. These were not necessarily intelligence professionals. They had no access to classified briefings, secret intercepts, or confidential government reports.
They worked with publicly available information.
Yet their best combined forecasts outperformed those of an intelligence-community prediction market whose participants could access classified information. An intelligence-community study compared 139 forecasting questions shared by the two systems. The Good Judgment Project’s strongest method produced significantly more accurate probability estimates.
Think about that.
The volunteers didn’t have better secrets. They had developed better methods for working with the information available to them.
This wasn’t proof that any amateur could defeat any intelligence analyst. It demonstrated something more useful: disciplined thinking, collaboration, and systematic correction can sometimes overcome an enormous informational advantage.
They didn’t win because they knew more. They won because they made better use of what they knew.
The Good Judgment Project identified its strongest forecasters, trained participants to recognize common errors in judgment, encouraged collaboration, and used mathematical techniques to combine predictions.
Some volunteers demonstrated exceptional forecasting ability. Tetlock called them superforecasters.
What distinguished them?
They broke complicated problems into smaller parts. They searched for contrary evidence, considered competing explanations, and distinguished what they knew from what they merely assumed.
They assigned probabilities instead of making grand pronouncements. They kept score. And when new information appeared, they changed their minds.
Superforecasters don’t possess crystal balls. They possess methods.
And here’s the remarkable part: you don’t need a security clearance, an advanced degree, or a position in an intelligence agency to begin cultivating those methods.
You can start today.
A Different Way of Thinking
Consider two commentators predicting a recession. The first announces, “Economic collapse is inevitable!” He presents a magnificent explanation, cites respected authorities, and dismisses anyone who disagrees.
The second examines the evidence and estimates a 35 percent probability of recession within the next twelve months. As new information appears, he adjusts his estimate.
Which is the better thinker?
You cannot tell from a single prediction. You must keep score. Both commentators might turn out to be right—or wrong. But repeated forecasts allow us to evaluate the quality of their judgments.
Someone who assigns a 70 percent probability to many different events should, over time, see roughly 70 percent of them occur. If the results consistently differ, his confidence needs correction.
Notice what has happened. We’ve moved from admiring the authority of the speaker to testing the effectiveness of his thinking.
That’s a revolution worth noticing.
Now Try It Yourself
Don’t merely admire the fox. Become one. Choose something you expect to happen in the coming weeks. Write down precisely what you predict, when it should happen, and the probability you assign to it.
When the time comes, check the result. Did the event occur? What did you overlook? Was your confidence justified?
Repeat the exercise. Keep a record. Over time, patterns will emerge.
You may discover something uncomfortable about your judgment.
Excellent! That’s what the exercise is for.
The Map Is Not the Territory
For thousands of years, oracles and experts have sold maps of a territory they’ve never visited: the future. Some maps proved useful. Others were elaborate, intellectually impressive, emotionally satisfying—and utterly worthless.
Tetlock did something revolutionary. He checked the maps against the territory. He recorded predictions, compared them with what actually happened, and identified methods that produced more accurate forecasts.
And that brings us to another foundational RISE² concept: The Map Is Not the Territory.
The most dangerous maps aren’t always the ones that look wrong. They’re the ones we desperately want to believe.
The Larger Discovery
We need education, accumulated knowledge, and genuine expertise. But none should grant immunity from examination or correction.
The methods cultivated by superforecasters are useful far beyond geopolitical predictions. Every important decision involves expectations about the future. We can defend our favorite explanations like hedgehogs or cultivate the curiosity, humility, and adaptability of foxes.
Tomorrow, another expert will predict the future for you. When you encounter him in a newspaper, on television, or on social media, listen, and it will feel good—but don’t surrender your judgment. Ask what evidence supports his prediction, what might prove him wrong, and whether anyone has kept score.
Then turn those same questions upon yourself.
The question isn’t whether the experts can be wrong. It’s whether you have learned how to discover when you are.
Sources and Further Exploration
-
Philip E. Tetlock and Dan Gardner, Superforecasting: The Art and Science of Prediction.
-
Philip E. Tetlock, Expert Political Judgment: How Good Is It? How Can We Know?
-
Seth Goldstein and colleagues, Assessing the Accuracy of Geopolitical Forecasts from the US Intelligence Community’s Prediction Market.
-
Barbara Mellers and colleagues, The Psychology of Intelligence Analysis.
-
Historical sources on The Population Bomb, The Jupiter Effect, and Arctic sea-ice predictions.
-
American Heart Association, The Facts on Fats.
-
Nobel Prize in Medicine, 2005 — Marshall and Warren.
-
U.S. Senate investigation into credit-rating agencies and the financial crisis.