Developing expert political judgment: The impact of training and practice on judgmental accuracy in geopolitical forecasting.

Published on Sep 1, 2016in Judgment and Decision Making
Welton Chang4
Estimated H-index: 4
,
Eva Chen6
Estimated H-index: 6
+ 1 AuthorsPhilip E. Tetlock99
Estimated H-index: 99
Sources
Abstract
The heuristics-and-biases research program highlights reasons for expecting people to be poor intuitive forecasters. This article tests the power of a cognitive-debiasing training module (“CHAMPS KNOW†) to improve probability judgments in a four-year series of geopolitical forecasting tournaments sponsored by the U.S. intelligence community. Although the training lasted less than one hour, it consistently improved accuracy (Brier scores) by 6 to 11% over the control condition. Cognitive ability and practice also made largely independent contributions to predictive accuracy. Given the brevity of the training tutorials and the heterogeneity of the problems posed, the observed effects are likely to be lower-bound estimates of what could be achieved by more intensive interventions. Future work should isolate which prongs of the multipronged CHAMPS KNOW training were most effective in improving judgment on which categories of problems.
📖 Papers frequently viewed together
189 Citations
118 Citations
2005
585 Citations
References87
Newest
#1Frank L. Schmidt (UI: University of Iowa)H-Index: 94
46 CitationsSource
#1Barbara A. Mellers (UPenn: University of Pennsylvania)H-Index: 56
#2Eric Stone (UPenn: University of Pennsylvania)H-Index: 5
Last. Philip E. Tetlock (UPenn: University of Pennsylvania)H-Index: 99
view all 12 authors...
Across a wide range of tasks, research has shown that people make poor probabilistic predictions of future events. Recently, the U.S. Intelligence Community sponsored a series of forecasting tournaments designed to explore the best strategies for generating accurate subjective probability estimates of geopolitical events. In this article, we describe the winning strategy: culling off top performers each year and assigning them into elite teams of superforecasters. Defying expectations of regress...
83 CitationsSource
#1Philip E. Tetlock (UPenn: University of Pennsylvania)H-Index: 99
#2Barbara A. Mellers (UPenn: University of Pennsylvania)H-Index: 56
Last. Eva Chen (UPenn: University of Pennsylvania)H-Index: 6
view all 4 authors...
Forecasting tournaments are level-playing-field competitions that reveal which individuals, teams, or algorithms generate more accurate probability estimates on which topics. This article describes a massive geopolitical tournament that tested clashing views on the feasibility of improving judgmental accuracy and on the best methods of doing so. The tournament’s winner, the Good Judgment Project, outperformed the simple average of the crowd by (a) designing new forms of cognitive-debiasing train...
47 CitationsSource
The Afghan and Iraqi conflicts, taken together, will be the most expensive wars in United States history, totaling somewhere between US4 to US trillion. This includes long-term medical care and disability compensation for service members, veterans and families, military replenishment, and social and economic costs. The largest portion of that bill is yet to be paid. Since 2001, the U.S. has expanded the quality, quantity, availability, and eligibility of benefits for military personnel and ve...
3 CitationsSource
#1Barbara A. Mellers (UPenn: University of Pennsylvania)H-Index: 56
#2Lyle H. Ungar (UPenn: University of Pennsylvania)H-Index: 80
Last. Philip E. Tetlock (UPenn: University of Pennsylvania)H-Index: 99
view all 13 authors...
Five university-based research groups competed to recruit forecasters, elicit their predictions, and aggregate those predictions to assign the most accurate probabilities to events in a 2-year geopolitical forecasting tournament. Our group tested and found support for three psychological drivers of accuracy: training, teaming, and tracking. Probability training corrected cognitive biases, encouraged forecasters to use reference classes, and provided forecasters with heuristics, such as averaging...
118 CitationsSource
#1Andrew C. Hafenbrack (Ad: INSEAD)H-Index: 8
#2Zoe Kinias (Ad: INSEAD)H-Index: 9
Last. Sigal G. Barsade (UPenn: University of Pennsylvania)H-Index: 28
view all 3 authors...
In the research reported here, we investigated the debiasing effect of mindfulness meditation on the sunk-cost bias. We conducted four studies (one correlational and three experimental); the results suggest that increased mindfulness reduces the tendency to allow unrecoverable prior costs to influence current decisions. Study 1 served as an initial correlational demonstration of the positive relationship between trait mindfulness and resistance to the sunk-cost bias. Studies 2a and 2b were labor...
152 CitationsSource
#1Pat Croskerry (Dal: Dalhousie University)H-Index: 32
#2Geeta Singhal (BCM: Baylor College of Medicine)H-Index: 8
Last. Sílvia Mamede (EUR: Erasmus University Rotterdam)H-Index: 25
view all 3 authors...
In a companion paper, we proposed that cognitive debiasing is a skill essential in developing sound clinical reasoning to mitigate the incidence of diagnostic failure. We reviewed the origins of cognitive biases and some proposed mechanisms for how debiasing processes might work. In this paper, we first outline a general schema of how cognitive change occurs and the constraints that may apply. We review a variety of individual factors, many of them biases themselves, which may be impediments to ...
178 CitationsSource
#1Pat Croskerry (Dal: Dalhousie University)H-Index: 32
#2Geeta Singhal (BCM: Baylor College of Medicine)H-Index: 8
Last. Sílvia Mamede (Erasmus University Medical Center)H-Index: 25
view all 3 authors...
Numerous studies have shown that diagnostic failure depends upon a variety of factors. Psychological factors are fundamental in influencing the cognitive performance of the decision maker. In this first of two papers, we discuss the basics of reasoning and the Dual Process Theory (DPT) of decision making. The general properties of the DPT model, as it applies to diagnostic reasoning, are reviewed. A variety of cognitive and affective biases are known to compromise the decision-making process. Th...
257 CitationsSource
#1Uriel Haran (BGU: Ben-Gurion University of the Negev)H-Index: 7
#2Ilana Ritov (HUJI: Hebrew University of Jerusalem)H-Index: 35
Last. Barbara A. Mellers (UPenn: University of Pennsylvania)H-Index: 56
view all 3 authors...
Errors in estimating and forecasting often result from the failure to collect and consider enough relevant information. We examine whether attributes associated with persistence in information acquisition can predict performance in an estimation task. We focus on actively open-minded thinking (AOT), need for cognition, grit, and the tendency to maximize or satisfice when making decisions. In three studies, participants made estimates and predictions of uncertain quantities, with varying levels o...
110 Citations
#1Mark L. Graber (RTI International)H-Index: 26
#2Stephanie M KissamH-Index: 2
Last. Hardeep SinghH-Index: 63
view all 10 authors...
Background: Errors in clinical reasoning occur in most cases in which the diagnosis is missed, delayed or wrong. The goal of this review was to identify interventions that might reduce the likelihood of these cognitive errors. Design: We searched PubMed and other medical and non-medical databases and identified additional literature through references from the initial data set and suggestions from subject matter experts. Articles were included if they either suggested a possible intervention or ...
249 CitationsSource
Cited By20
Newest
We investigate expert disagreement over the potential and limitations of deep learning. We conducted 25 expert interviews to reveal the reasons and arguments that underlie the disagreement about the limitations of deep learning, here evaluated in respect to high-level machine intelligence. Experts in our sample named 40 limitations of deep learning. Using interview data, we identify and explore five crucial, unresolved research subjects that underpin this scholarly disagreement: abstraction, gen...
Source
#1Bent Flyvbjerg (University of Oxford)H-Index: 68
#2Alexander Budzier (University of Oxford)H-Index: 11
Last. Daniel Lunn (University of Oxford)H-Index: 12
view all 3 authors...
The Olympic Games are the largest, highest-profile, and most expensive megaevent hosted by cities and nations. Average sports-related costs of hosting are $12.0 billion. Non-sports-related costs ar...
3 CitationsSource
#1Ilias Katsagounos (UoP: University of Peloponnese)H-Index: 1
#2Dimitrios D. Thomakos (UoP: University of Peloponnese)H-Index: 13
Last. Konstantinos Nikolopoulos (Durham University)H-Index: 23
view all 4 authors...
Abstract Superforecasting has drawn the attention of academics - despite earlier contradictory findings in the literature, arguing that humans can consistently and successfully forecast over long periods. It has also enthused practitioners, due to the major implications for improving forecast-driven decision-making. The evidence in support of the superforecasting hypothesis was provided via a 4-year project led by Tetlock and Mellers, which was based on an exhaustive experiment with more than 50...
Source
Geopolitical forecasting tournaments have stimulated the development of methods for improving probability judgments of real-world events. But these innovations have focused on easier-to quantify variables, like personnel selection, training, teaming, and crowd aggregation—and bypassed messier constructs, like qualitative properties of forecasters’ rationales. Here we adapt methods from natural language processing (NLP) and computational text analysis to identify distinctive reasoning strategies ...
1 CitationsSource
#1Haewon Yoon (IU: Indiana University)H-Index: 5
#2Irene Scopelliti (City University London)H-Index: 8
Last. Carey K. Morewedge (BU: Boston University)H-Index: 27
view all 3 authors...
Abstract Observational learning can debias judgment and decision making. One-shot observational learning-based training interventions (akin to “hot seating”) can produce reductions in cognitive biases in the laboratory (i.e., anchoring, representativeness, and social projection), and successfully teach a decision rule that increases advice taking in a weight on advice paradigm (i.e., the averaging principle). These interventions improve judgment, rule learning, and advice taking more than practi...
Source
#1Daniel M. Benjamin (McGill University)H-Index: 8
#2David R. MandelH-Index: 27
Last. Jonathan Kimmelman (McGill University)H-Index: 31
view all 7 authors...
Background Decisions about trial funding, ethical approval, or clinical practice guideline recommendations require expert judgments about the potential efficacy of new treatments. We tested whether individual and aggregated expert opinion of oncologists could predict reliably the efficacy of cancer treatments tested in randomized controlled trials. Materials and methods An international sample of 137 oncologists specializing in genitourinary, lung, and colorectal cancer provided forecasts on pri...
Source
#1Pavel AtanasovH-Index: 11
#2Jens Witkowski (Frankfurt School of Finance & Management)H-Index: 11
Last. Philip E. Tetlock (UPenn: University of Pennsylvania)H-Index: 99
view all 5 authors...
Abstract Laboratory research has shown that both underreaction and overreaction to new information pose threats to forecasting accuracy. This article explores how real-world forecasters who vary in skill attempt to balance these threats. We distinguish among three aspects of updating: frequency, magnitude, and confirmation propensity. Drawing on data from a four-year forecasting tournament that elicited over 400,000 probabilistic predictions on almost 500 geopolitical questions, we found that th...
6 CitationsSource
Abstract The paper addresses an issue largely discussed in the field of Forecasting and in many future-oriented scientific and professional disciplines, but less frequently considered in the Foresight literature, particularly in the technology foresight field- i.e. the extent to which biases of human experts influence the foresight process. The paper reviews the literature on cognitive biases and identifies the main areas of technology foresight in which biases are most likely to materialize. It...
10 CitationsSource
#1Randy Borum (USF: University of South Florida)H-Index: 60
Powered by advances in computing technology, a range of professions and business enterprises have moved toward a more science-driven approach to operations. Law enforcement has been no exception. In fact, the modern day idea of intelligence-led policing (ILP) emerged in the UK in the 1990s as the country was pushing all government services to operate on more of a data-informed, business process or managerial model. This trend led to the development of a British “National Intelligence Model” (NIM...
Source
#1Simon Beard (University of Cambridge)H-Index: 2
#2Thomas Rowe (VT: Virginia Tech)H-Index: 1
Last. James Fox (University of Oxford)H-Index: 2
view all 3 authors...
Abstract This paper examines and evaluates the range of methods that have been used to make quantified claims about the likelihood of Existential Hazards. In doing so, it draws on a comprehensive literature review of such claims that we present in an appendix. The paper uses an informal evaluative framework to consider the relative merits of these methods regarding their rigour, ability to handle uncertainty, accessibility for researchers with limited resources and utility for communication and ...
7 CitationsSource