Identifying Expertise to Extract the Wisdom of Crowds

Published on Feb 1, 2015in Management Science3.935
· DOI :10.1287/MNSC.2014.1909
David V. Budescu64
Estimated H-index: 64
(Fordham University),
Eva Chen6
Estimated H-index: 6
(UPenn: University of Pennsylvania)
Statistical aggregation is often used to combine multiple opinions within a group. Such aggregates outperform individuals, including experts, in various prediction and estimation tasks. This result is attributed to the “wisdom of crowds.” We seek to improve the quality of such aggregates by eliminating poorly performing individuals from the crowd. We propose a new measure of contribution to assess the judges' performance relative to the group and use positive contributors to build a weighting model for aggregating forecasts. In Study 1, we analyze 1,233 judges forecasting almost 200 current events to illustrate the superiority of our model over unweighted models and models weighted by measures of absolute performance. In Study 2, we replicate our findings by using economic forecasts from the European Central Bank and show how the method can be used to identify smaller crowds of the top positive contributors. We show that the model derives its power from identifying experts who consistently outperform the crowd. Data, as supplemental material, are available at . This paper was accepted by James Smith, decision analysis.
📖 Papers frequently viewed together
113 Citations
260 Citations
582 Citations
#1Albert E. MannesH-Index: 7
#2Jack B. Soll (Duke University)H-Index: 17
Last. Richard P. Larrick (Duke University)H-Index: 37
view all 3 authors...
Social psychologists have long recognized the power of statisticized groups. When individual judgments about some fact (e.g., the unemployment rate for next quarter) are averaged together, the average opinion is typically more accurate than most of the individual estimates, a pattern often referred to as the wisdom of crowds. The accuracy of averaging also often exceeds that of the individual perceived as most knowledgeable in the group. However, neither averaging nor relying on a single judge i...
113 CitationsSource
#1Brandon M. Turner (Stanford University)H-Index: 24
#2Mark Steyvers (UCI: University of California, Irvine)H-Index: 55
Last. Thomas S. Wallsten (UMD: University of Maryland, College Park)H-Index: 39
view all 5 authors...
It is known that the average of many forecasts about a future event tends to outperform the individual assessments. With the goal of further improving forecast performance, this paper develops and compares a number of models for calibrating and aggregating forecasts that exploit the well-known fact that individuals exhibit systematic biases during judgment and elicitation. All of the models recalibrate judgments or mean judgments via a two-parameter calibration function, and differ in terms of w...
39 CitationsSource
#1Clintin P. Davis-Stober (MU: University of Missouri)H-Index: 17
#2David V. Budescu (Fordham University)H-Index: 64
Last. Stephen B. Broomell (CMU: Carnegie Mellon University)H-Index: 13
view all 4 authors...
Numerous studies and anecdotes demonstrate the "wisdom of the crowd," the surprising accuracy of a group's aggregated judgments. Less is known, however, about the generality of crowd wisdom. For example, are crowds wise even if their members have systematic judgmental biases, or can influence each other before members render their judgments? If so, are there situations in which we can expect a crowd to be less accurate than skilled individuals? We provide a precise but general definition of crow...
63 CitationsSource
#1Victor Richmond R. Jose (Georgetown University)H-Index: 11
#2Yael Grushka-Cockayne (UVA: University of Virginia)H-Index: 11
Last. Kenneth C. Lichtendahl (UVA: University of Virginia)H-Index: 9
view all 3 authors...
We introduce an alternative to the popular linear opinion pool for combining individual probability forecasts. One of the well-known problems with the linear opinion pool is that it can be poorly calibrated. It tends toward underconfidence as the crowd's diversity increases, i.e., as the variance in the individuals' means increases. To address this calibration problem, we propose the exterior-trimmed opinion pool. To form this pool, forecasts with low and high means, or cumulative distribution f...
36 CitationsSource
#1Kenneth C. Lichtendahl (UVA: University of Virginia)H-Index: 9
#2Yael Grushka-Cockayne (UVA: University of Virginia)H-Index: 11
Last. Phillip E. Pfeifer (UVA: University of Virginia)H-Index: 24
view all 3 authors...
When several individuals are asked to forecast an uncertain quantity, they often face implicit or explicit incentives to be the most accurate. Despite the desire to elicit honest forecasts, such competition induces forecasters to report strategically and non-truthfully. The question we address is whether the competitive crowd's forecast (the average of strategic forecasts) is more accurate than the truthful crowd's forecast (the average of truthful forecasts from the same forecasters). We analyz...
23 CitationsSource
#1Stephen C. Hora (SC: University of Southern California)H-Index: 19
Last. Irving SuselH-Index: 2
view all 4 authors...
When multiple redundant probabilistic judgments are obtained from subject matter experts, it is common practice to aggregate their differing views into a single probability or distribution. Although many methods have been proposed for mathematical aggregation, no single procedure has gained universal acceptance. The most widely used procedure is simple arithmetic averaging, which has both desirable and undesirable properties. Here we propose an alternative for aggregating distribution functions ...
29 CitationsSource
#1Theodoros Evgeniou (Ad: INSEAD)H-Index: 28
#2Lily H. Fang (Ad: INSEAD)H-Index: 14
Last. Natalia Karelaia (Ad: INSEAD)H-Index: 16
view all 4 authors...
The outcomes in many competitive tasks depend upon both skill and luck. Behavioral theories on risk taking in tournaments indicate that low-skilled individuals may have incentives to take more risks than high-skilled ones. We build on these theories and suggest, in addition, that when luck is more important in determining outcomes, the increase in risk taking is larger for low-skilled than high-skilled individuals. We test this hypothesis by analyzing stock analysts' forecasts of companies' earn...
5 CitationsSource
#1Simon French (Warw.: University of Warwick)H-Index: 39
There are three contexts in which one might wish to combine expert judgments of uncertainty: the expert problem, the group decision problem, and the textbook problem. Much has been written on the first two, which have the focus of a single decision context, but little on the third. The textbook problem arises when one needs to draw together expert judgments into a decision analysis when their judgments were made originally in a context-free manner or perhaps for other decision contexts. In many ...
24 CitationsSource
#1Albert E. Mannes (CMU: Carnegie Mellon University)H-Index: 7
#2Richard P. Larrick (Duke University)H-Index: 37
Last. Jack B. Soll (Duke University)H-Index: 17
view all 3 authors...
74 Citations
#1Michael D. Lee (UCI: University of California, Irvine)H-Index: 80
#2Shunan Zhang (UCI: University of California, Irvine)H-Index: 11
Last. Jenny Shi (UCI: University of California, Irvine)H-Index: 2
view all 3 authors...
In The Price Is Right game show, players compete to win a prize, by placing bids on its price. We ask whether it is possible to achieve a “wisdom of the crowd” effect, by combining the bids to produce an aggregate price estimate that is superior to the estimates of individual players. Using data from the game show, we show that a wisdom of the crowd effect is possible, especially by using models of the decision-making processes involved in bidding. The key insight is that, because of the competi...
31 CitationsSource
Cited By128
In the setting where we want to aggregate people's subjective evaluations, plurality vote may be meaningless when a large amount of low-effort people always report "good" regardless of the true quality. "Surprisingly popular" method, picking the most surprising answer compared to the prior, handle this issue to some extent. However, it is still not fully robust to people's strategies. Here in the setting where a large number of people are asked to answer a small number of multi-choice questions ...
#1Yuqing KongH-Index: 8
Last. Jinzhao Wu (PKU: Peking University)
view all 5 authors...
System 1 vs. 2 theory describes two modes of thought, a fast, instinctive one and a slow, logical one. When we ask a question (e.g. A bat and ball cost \1.10. The bat costs \ more than the ball. How much does the ball cost?), with prior, we can identify fast/slow thinking (\.10/\05). But what if we do not have prior? A very clever method, surprisingly popular, additionally asks what percentage of other people answer \.10/\05 and selects the answer that is more popular than people predic...
#1Joshua Becker (UCL: University College London)H-Index: 6
#2Douglas Guilbeault (University of California, Berkeley)H-Index: 7
Last. Edward Smith (NU: Northwestern University)H-Index: 14
view all 3 authors...
Decades of research suggest that information exchange in groups and organizations can reliably improve judgment accuracy in tasks such as financial forecasting, market research, and medical decisio...
#1Hayley M. Geyle (CDU: Charles Darwin University)H-Index: 7
#2Conrad J. Hoskin (JCU: James Cook University)H-Index: 19
Last. Geoffrey W. Heard (CSU: Charles Sturt University)
view all 29 authors...
More than a third of the world’s amphibian species are listed as Threatened or Extinct, with a recent assessment identifying 45 Australian frogs (18.4% of the currently recognised species) as ‘Threatened’ based on IUCN criteria. We applied structured expert elicitation to 26 frogs assessed as Critically Endangered and Endangered to estimate their probability of extinction by 2040. We also investigated whether participant experience (measured as a self-assigned categorical score, i.e. ‘expert’ or...
#1Mark York (Harvard University)
#2Munther A. Dahleh (MIT: Massachusetts Institute of Technology)H-Index: 58
Last. David C. Parkes (Harvard University)H-Index: 63
view all 3 authors...
Access to capital is a major constraint for economic growth in the developing world. Yet those attempting to lend in this space face high defaults due to their inability to distinguish creditworthy borrowers from the rest. In this paper, we propose two novel scoring mechanisms that incentivize community members to truthfully report their signal on the creditworthiness of others in their community. We first design a truncated asymmetric scoring-rule for a setting where the lender has no liquidity...
#1Saul Estrin (Centre for Economic Performance)H-Index: 72
#2Susanna Khavul (SPbU: Saint Petersburg State University)H-Index: 15
Last. Mike Wright (Imperial College London)H-Index: 142
view all 3 authors...
#1Patrick Afflerbach (University of Augsburg)H-Index: 6
#2Christopher van Dun (University of Bayreuth)
Last. Johannes Seyfried (University of Augsburg)H-Index: 4
view all 5 authors...
Research has shown that aggregation of independent expert judgments significantly improves the quality of forecasts as compared to individual expert forecasts. This “wisdom of crowds” (WOC) has sparked substantial interest. However, previous studies on strengths and weaknesses of aggregation algorithms have been restricted by limited empirical data and analytical complexity. Based on a comprehensive analysis of existing knowledge on WOC and aggregation algorithms, this paper describes the design...
3 CitationsSource
#1Vincenz Frey (UG: University of Groningen)H-Index: 6
#2Arnout van de Rijt (UU: Utrecht University)H-Index: 14
Teams, juries, electorates, and committees must often select from various alternative courses of action what they judge to be the best option. The phenomenon that the central tendency of many indep...
3 CitationsSource
#1James A. Taylor (University of Oxford)H-Index: 83
#2Kathryn Taylor (University of Oxford)H-Index: 37
The COVID-19 pandemic has placed forecasting models at the forefront of health policy making. Predictions of mortality, cases and hospitalisations help governments meet planning and resource allocation challenges. In this paper, we consider the weekly forecasting of the cumulative mortality due to COVID-19 at the national and state level in the U.S. Optimal decision-making requires a forecast of a probability distribution, rather than just a single point forecast. Interval forecasts are also imp...
2 CitationsSource
#6Christopher Jackson (University of Cambridge)H-Index: 27
#8Alec Morton (University of Strathclyde)H-Index: 24
Background Many decisions in healthcare aim to maximise health, requiring judgements about interventions that may have higher health effects but potentially incur additional costs (cost-effectiveness framework). The evidence used to establish cost-effectiveness is typically uncertain and it is important that this uncertainty is characterised. In situations where evidence is uncertain, the experience of experts is essential. The process by which the beliefs of experts can be formally collected in...
2 CitationsSource