Some empirical results are more likely to be published than others. Such selective publication leads to biased estimates and distorted inference. This paper proposes two approaches for identifying the conditional probability of publication as a function of a study’s results, the first based on systematic replication studies and the second based on meta-studies. For known conditional publication probabilities, we propose median-unbiased estimators and associated confidence sets that correct for selective publication. We apply our methods to recent large-scale replication studies in experimental economics and psychology, and to meta-studies of the effects of minimum wages and de-worming programs.

#2Devin G. Pope(U of C: University of Chicago)H-Index: 27

We analyze how academic experts and nonexperts forecast the results of 15 piece-rate and behavioral treatments in a real-effort task. The average forecast of experts closely predicts the experimental results, with a strong wisdom-of-crowds effect: the average forecast outperforms 96 percent of individual forecasts. Citations, academic rank, field, and contextual experience do not correlate with accuracy. Experts as a group do better than nonexperts, but not if accuracy is defined as rank-orderin...

Meta-regression models were originally developed for the synthesis of experimental research where randomization ensures unbiased and consistent estimation of the effect of interest. Most economics research is, however, observational and specification searches may often result in estimates that are biased and inconsistent, for example, due to omitted-variable biases. We show that if the authors of primary studies search for statistically significant estimates in observational research, meta-regre...

Publication bias leads consumers of research to observe a selected sample of statistical estimates calculated by producers of research. We calculate critical values for statistical significance that could help to adjust after the fact for the distortions created by this selection effect, assuming that the only source of publication bias is file drawer bias. These adjusted critical values are easy to calculate and differ from unadjusted critical values by approximately 50%—rather than rejecting a...

#1Tomas Havranek(Charles University in Prague)H-Index: 25

I examine 2,735 estimates of the elasticity of intertemporal substitution in consumption (EIS) reported in 169 published studies. The literature shows strong selective reporting: researchers discard negative and insignificant estimates too often, which pulls the mean estimate up by about 0.5. The reporting bias dwarfs the effects of methods, with the exception of the choice between micro and macro data. When I correct the mean for the bias, for macro estimates I get zero, even though the reporte...

#1Benjamin A. Olken(MIT: Massachusetts Institute of Technology)H-Index: 40

The purpose of this paper is to help think through the advantages and costs of rigorous pre-specification of statistical analysis plans in economics. A pre-analysis plan pre-specifies in a precise way the analysis to be run before examining the data. A researcher can specify variables, data cleaning procedures, regression specifications, and so on. If the regressions are pre-specified in advance and researchers are required to report all the results they pre-specify, data-mining problems are gre...

#1Uri Simonsohn(UPenn: University of Pennsylvania)H-Index: 32

This article introduces a new approach for evaluating replication results. It combines effect-size estimation with hypothesis testing, assessing the extent to which the replication results are consistent with an effect size big enough to have been detectable in the original study. The approach is demonstrated by examining replications of three well-known findings. Its benefits include the following: (a) differentiating “unsuccessful” replication attempts (i.e., studies yielding p > .05) that are...

#1Uri Simonsohn(UPenn: University of Pennsylvania)H-Index: 32

#2Leif D. Nelson(University of California, Berkeley)H-Index: 36

Last. Joseph P. Simmons(UPenn: University of Pennsylvania)H-Index: 28

view all 3 authors...

Because scientists tend to report only studies (publication bias) or analyses (p-hacking) that "work," readers must ask, "Are these effects true, or do they merely reflect selective reporting?" We introduce p-curve as a way to answer this question. P-curve is the distribution of statistically significant p values for a set of studies (ps < .05). Because only true effects are expected to generate right-skewed p-curves-containing more low (.01s) than high (.04s) significant p values--only right-sk...

#2Patrick B. Ryan(Foundation for the National Institutes of Health)H-Index: 5

Often the literature makes assertions of medical product effects on the basis of ‘ p < 0.05’. The underlying premise is that at this threshold, there is only a 5% probability that the observed effect would be seen by chance when in reality there is no effect. In observational studies, much more than in randomized trials, bias and confounding may undermine this premise. To test this premise, we selected three exemplar drug safety studies from literature, representing a case–control, a cohort, and...

Card and Krueger's meta-analysis of the employment effects of minimum wages challenged existing theory. Unfortunately, their meta-analysis confused publication selection with the absence of a genuine empirical effect. We apply recently developed meta-analysis methods to 64 US minimum-wage studies and corroborate that Card and Krueger's findings were nevertheless correct. The minimum-wage effects literature is contaminated by publication selection bias, which we estimate to be slightly larger tha...

Summary There is increasing concern that most current published research fi ndings are false. The probability that a research claim is true may depend on study power and bias, the number of other studies on the same question, and, importantly, the ratio of true to no relationships among the relationships probed in each scientifi c fi eld. In this framework, a research fi nding is less likely to be true when the studies conducted in a fi eld are smaller; when effect sizes are smaller; when there ...

This paper performs a meta-analysis of the effect of financial development and liberalization on macroeconomic growth in Latin America and the Caribbean using a total of 233 estimates collected from 21 previous works. Meta-synthesis of the collected estimates demonstrates that it is probable that financial development and liberalization enhance economic growth in the region, and these policy measures have the potential to have a meaningful impact on the real economy. The synthesis results also r...

Surveillance studies for Covid-19 prevalence estimation are subject to sampling bias due to oversampling of symptomatic individuals and error-prone tests, particularly rapid antigen tests which are known to have high false negative rates for asymptomatic individuals. This results in naive estimators which can be very far from the truth.In this work, we present a method that removes these two sources of error directly. Moreover, our procedure can be easily extended to the stratified error situati...

A meta-analysis of published estimates shows that the social cost of carbon has increased as knowledge about climate change accumulates. Correcting for inflation and emission year and controlling for the discount rate, kernel density decomposition reveals a non-stationary distribution. Actual carbon prices are almost everywhere below its estimated value and should therefore go up.

This study attempts to solve the publication bias suggested by recent review articles in the tourism-growth literature. Publication bias is the tendency to report favourable and significant results. Method and data triangulation, and the Solow-Swan model are applied. A sample from 1995 to 2018 is considered with Tonga as a case study. The approach consists of multiple methods, data frequencies, exchange rates, structural breaks, and an overall tourism index developed using principal component an...

Suppose the 300+ published asset pricing factors are all spurious. How much p-hacking is required to produce these factors? If 10,000 researchers generate 8 factors every day, it takes hundreds of years. This is because dozens of published t-statistics exceed 6.0, while the corresponding p-value is infinitesimal, implying an astronomical amount of p-hacking in a general model. More structure implies p-hacking cannot address ≈100 published t-statistics that exceed 4.0, as they require an implausi...

Does the wish to convince others lead people to persuade themselves about the moral and factual superiority of their position? We investigate this question in the context of two international debating competitions, where persuasion goals (pro or contra a motion) are randomly assigned to debaters shortly before the debate. Using incentives for truthful reporting, we find evidence of self-persuasion in the form of (i) factual beliefs that become more conveniently aligned with the debater's side of...

This paper offers a meta-regression analysis of the literature on the drivers of financial development. Our results based on 1900 estimates suggest that institutional quality is positively correlated to both private sector credit and stock market capitalization (both as share of GDP). Domestic financial openness has a positive effect on both proxies for financial development, while trade openness seems only important for stock market capitalization. Inflation has an adverse effect on financial d...

We use cookies to improve your online experience. By continuing to use our website we assume you agree to the placement of these cookies. To learn more, you can find in our Privacy Policy.