Methods to select results to include in meta-analyses deserve more consideration in systematic reviews

Published on Nov 1, 2015in Journal of Clinical Epidemiology6.437
· DOI :10.1016/J.JCLINEPI.2015.02.009
Matthew J. Page42
Estimated H-index: 42
(Monash University),
Joanne E. McKenzie43
Estimated H-index: 43
(Monash University)
+ 2 AuthorsAndrew Forbes96
Estimated H-index: 96
(Monash University)
Sources
Abstract
Abstract Objectives To investigate how often systematic reviewers encounter multiple trial effect estimates that are available for inclusion in a particular meta-analysis (multiplicity of results) and the methods they use to select effect estimates. Study Design and Setting We randomly sampled Cochrane and MEDLINE-indexed non-Cochrane reviews published between January 2010 and January 2012. The first presented meta-analysis of an effect measure for a continuous outcome in each review was identified, and methods to select results to include in this meta-analysis were extracted from review protocols and reviews. All effect estimates that were available for inclusion in the meta-analyses were extracted from trial reports. Results We examined 44 reviews. Multiplicity of results was common, occurring in 49% of trial reports ( n  = 210). Prespecification of decision rules to select results from multiple measurement scales and intervention/control groups (in multi-arm trials) was uncommon (19% and 14% of 21 review protocols, respectively). Overall, 70% of reviews included at least one randomized controlled trial with multiplicity of results, but this occurred less frequently in reviews with a protocol (risk difference, −25%; 95% confidence interval: −52%, 1%). Conclusion Systematic reviewers are likely to encounter multiplicity of results in the included trials. We recommend that systematic reviewers always consider predefining methods to select results to include in meta-analyses. Methods focusing on selection of measurement scales and how to deal with multi-arm trials would be most valuable.
📖 Papers frequently viewed together
20112.30BMJ
2017
References29
Newest
#1Paul Glasziou (Bond University)H-Index: 139
#2Iain ChalmersH-Index: 72
Last. Susan Michie (UCL: University College London)H-Index: 110
view all 4 authors...
Paul Glasziou and colleagues discuss methods to guide selection of an intervention from individual trials within a systematic review. Please see later in the article for the Editors' Summary.
Source
#1Maarten Boers (VUmc: VU University Medical Center)H-Index: 99
#2John R. Kirwan (BRI: Bristol Royal Infirmary)H-Index: 76
Last. Peter Tugwell (U of O: University of Ottawa)H-Index: 149
view all 15 authors...
Abstract Background Lack of standardization of outcome measures limits the usefulness of clinical trial evidence to inform health care decisions. This can be addressed by agreeing on a minimum core set of outcome measures per health condition, containing measures relevant to patients and decision makers. Since 1992, the Outcome Measures in Rheumatology (OMERACT) consensus initiative has successfully developed core sets for many rheumatologic conditions, actively involving patients since 2002. It...
Source
#1Jamie J Kirkham (University of Liverpool)H-Index: 37
#2Maarten BoersH-Index: 99
Last. Paula R Williamson (University of Liverpool)H-Index: 92
view all 5 authors...
Background The development and application of standardised sets of outcomes to be measured and reported in clinical trials have the potential to increase the efficiency and value of research. One of the most notable of the current outcome sets began nearly 20 years ago: the World Health Organization and International League of Associations for Rheumatology core set of outcomes for rheumatoid arthritis clinical trials, originating from the OMERACT (Outcome Measures in Rheumatology) Initiative. Th...
Source
#1Matthew J. Page (Monash University)H-Index: 42
#2Joanne E. McKenzie (Monash University)H-Index: 43
Last. Andrew Forbes (Monash University)H-Index: 96
view all 3 authors...
Abstract Objective To collate and categorize the ways in which selective inclusion and reporting can occur in randomized controlled trials (RCTs) and systematic reviews. Study Design and Setting Searches of the Cochrane Methodology Register, PubMed, and PsycInfo were conducted in April 2011. Methodological reports describing empirically investigated or hypothetical examples of selective inclusion or reporting were eligible for inclusion. Examples were extracted from the reports by one author and...
Source
#1Matthew J. Page (Monash University)H-Index: 42
#2Joanne E. McKenzie (Monash University)H-Index: 43
Last. Andrew Forbes (Monash University)H-Index: 96
view all 4 authors...
Background Systematic reviewers may encounter a multiplicity of outcome data in the reports of randomised controlled trials included in the review (for example, multiple measurement instruments measuring the same outcome, multiple time points, and final and change from baseline values). The primary objectives of this study are to investigate in a cohort of systematic reviews of randomised controlled trials of interventions for rheumatoid arthritis, osteoarthritis, depressive disorders and anxiet...
Source
#2Jill EdenH-Index: 8
#3Laura LevitH-Index: 3
Last. Sally C. MortonH-Index: 85
view all 4 authors...
Healthcare decision makers in search of reliable information that compares health interventions increasingly turn to systematic reviews for the best summary of the evidence. Systematic reviews identify, select, assess, and synthesize the findings of similar but separate studies, and can help clarify what is known and not known about the potential benefits and harms of drugs, devices, and other healthcare services. Systematic reviews can be helpful for clinicians who want to integrate research fi...
#1Lucy Turner (Ottawa Hospital Research Institute)H-Index: 20
#2James Galipeau (Ottawa Hospital Research Institute)H-Index: 15
Last. David Moher (Ottawa Hospital Research Institute)H-Index: 101
view all 7 authors...
Background Systematic reviews (SRs) are abundant. The optimal reporting of SRs is critical to enable clinicians to use their findings to make informed treatment decisions. Complementary and alternative medicine (CAM) therapies are widely used therefore it is critical that conduct and reporting of systematic research in this field be of high quality. Here, methodological and reporting characteristics of a sample of CAM-related SRs and a sample of control SRs are evaluated and compared.
Source
#1Britta Tendal (Cochrane Collaboration)H-Index: 18
#2Eveline NüeschH-Index: 40
Last. Peter C GøtzscheH-Index: 97
view all 5 authors...
Objectives To examine the extent of multiplicity of data in trial reports and to assess the impact of multiplicity on meta-analysis results. Design Empirical study on a cohort of Cochrane systematic reviews. Data sources All Cochrane systematic reviews published from issue 3 in 2006 to issue 2 in 2007 that presented a result as a standardised mean difference (SMD). We retrieved trial reports contributing to the first SMD result in each review, and downloaded review protocols. We used these SMDs ...
Source
#1Britta Tendal (Cochrane Collaboration)H-Index: 18
#2Julian P T Higgins (UoB: University of Bristol)H-Index: 134
Last. Peter C Gøtzsche (Cochrane Collaboration)H-Index: 97
view all 11 authors...
Objective To study the inter-observer variation related to extraction of continuous and numerical rating scale data from trial reports for use in meta-analyses. Design Observer agreement study. Data sources A random sample of 10 Cochrane reviews that presented a result as a standardised mean difference (SMD), the protocols for the reviews and the trial reports (n=45) were retrieved. Data extraction Five experienced methodologists and five PhD students independently extracted data from the trial ...
Source
Cited By15
Newest
#4Zhaoli Dai (Macquarie University)H-Index: 11
ObjectivesTo investigate the extent of multiplicity of results in study reports of nutrition research, and the methods specified in systematic reviews to select results for inclusion in meta-analyses. null MethodsMEDLINE and Epistemonikos were searched (January 2018 - June 2019) to identify systematic reviews with meta-analysis of the association between food/diet and health-related outcomes. A random sample of these reviews was selected, and for the first presented ( index) meta-analysis, rules...
Source
#1Hou-Ren TsaiH-Index: 1
#2Jing-Wun LuH-Index: 1
Last. Tai-Li ChenH-Index: 1
view all 4 authors...
Janus kinase (JAK) inhibitors are promising treatments for atopic dermatitis (AD). The aim of this study was to assess the efficacy and safety of JAK inhibitors for AD treatment via the “Grading of Recommendations Assessment, Development, and Evaluation” approach. We identified 15 randomized controlled trials comparing oral or topical JAK inhibitors against placebo to treat AD. A random-effects meta-analysis was performed, and the numbers-needed-to-treat (NNTs)/numbers-needed-to-harm (NNHs) were...
Source
#1My-Linh Nguyen Luong (University of Melbourne)H-Index: 5
#2Michelle Hall (University of Melbourne)H-Index: 16
Last. Rana S. Hinman (University of Melbourne)H-Index: 68
view all 6 authors...
OBJECTIVE To evaluate the effects of financial incentives on physical activity (PA). DATA SOURCES MEDLINE, Embase, 7 other databases, and 2 trial registries until July 17, 2019. STUDY INCLUSION AND EXCLUSION CRITERIA Randomized controlled trials with adults aged ≥18 years assessing the effect of financial incentives on PA. Any comparator was eligible provided the only difference between groups was the incentive strategy. DATA EXTRACTION Two independent reviewers extracted data and assessed study...
Source
#1Matthew J. Page (Monash University)H-Index: 42
#2Lisa Bero (USYD: University of Sydney)H-Index: 82
Last. Joanne E. McKenzie (Monash University)H-Index: 43
view all 7 authors...
Background: Dietary guidelines should be informed by systematic reviews (SRs) of the available scientific evidence. However, if the SRs that underpin dietary guidelines are flawed in their design, conduct or reporting, the recommendations contained therein may be misleading or harmful. To date there has been little empirical investigation of bias due to selective inclusion of results, and bias due to missing results, in SRs of food/diet-outcome relationships. Objectives: To explore in SRs with m...
Source
#1Tianjing Li (Johns Hopkins University)H-Index: 36
#2Evan Mayo-Wilson (Johns Hopkins University)H-Index: 37
Last. Kay Dickersin (Johns Hopkins University)H-Index: 71
view all 5 authors...
Clinical trials and systematic reviews of clinical trials inform healthcare decisions. There is growing concern, however, about results from clinical trials that cannot be reproduced. Reasons for nonreproducibility include that outcomes are defined in multiple ways, results can be obtained using multiple methods of analysis, and trial findings are reported in multiple sources (“multiplicity”). Multiplicity combined with selective reporting can influence dissemination of trial findings and decisi...
Source
#1José A. López-López (UoB: University of Bristol)H-Index: 23
#2Matthew J. Page (UoB: University of Bristol)H-Index: 42
Last. Julian P T Higgins (UoB: University of Bristol)H-Index: 134
view all 4 authors...
: Systematic reviews often encounter primary studies that report multiple effect sizes based on data from the same participants. These have the potential to introduce statistical dependency into the meta-analytic data set. In this paper, we provide a tutorial on dealing with effect size multiplicity within studies in the context of meta-analyses of intervention and association studies, recommending a three-step approach. The first step is to define the research question and consider the extent t...
Source
#1Fernanda S. Tonin (UFPR: Federal University of Paraná)H-Index: 16
#2Laiza M. Steimbach (UFPR: Federal University of Paraná)H-Index: 7
Last. Fernando Fernandez-Llimos (University of Lisbon)H-Index: 29
view all 6 authors...
Background Network meta-analysis (NMA) is a new tool developed to overcome some limitations of pairwise meta-analyses. NMAs provide evidence on more than two comparators simultaneously. This study aimed to map the characteristics of the published NMAs on drug therapy comparisons. Methods A systematic review of NMAs comparing pharmacological interventions was performed. Searches in Medline (PubMed) and Scopus along with manual searches were conducted. The main characteristics of NMAs were systema...
Source
#1Matthew J. Page (Monash University)H-Index: 42
#2Larissa Shamseer (Ottawa Hospital Research Institute)H-Index: 33
Last. Andrea C. Tricco (U of T: University of Toronto)H-Index: 63
view all 3 authors...
The International Prospective Register of Systematic Reviews (PROSPERO) was launched in February 2011 to increase transparency of systematic reviews (SRs). There have been few investigations of the content and use of the database. We aimed to investigate the number of PROSPERO registrations from inception to 2017, and website usage in the last year. We also aimed to explore the epidemiological characteristics of and completeness of primary outcome pre-specification in a sample of PROSPERO record...
Source
#1Evan Mayo-Wilson (Johns Hopkins University)H-Index: 37
#2Tianjing Li (Johns Hopkins University)H-Index: 36
Last. Kay Dickersin (Johns Hopkins University)H-Index: 71
view all 22 authors...
Abstract Objectives The objective of this study was to determine whether disagreements among multiple data sources affect systematic reviews of randomized clinical trials (RCTs). Study Design and Setting Eligible RCTs examined gabapentin for neuropathic pain and quetiapine for bipolar depression, reported in public (e.g., journal articles) and nonpublic sources (clinical study reports [CSRs] and individual participant data [IPD]). Results We found 21 gabapentin RCTs (74 reports, 6 IPD) and 7 que...
Source
#1Evan Mayo-Wilson (Johns Hopkins University)H-Index: 37
#2Nicole Fusco (Johns Hopkins University)H-Index: 6
Last. Kay Dickersin (Johns Hopkins University)H-Index: 71
view all 6 authors...
Abstract Objective To identify variations in outcomes and results across reports of randomized clinical trials (RCTs). Study Design and Setting Eligible RCTs examined gabapentin for neuropathic pain and quetiapine for bipolar depression, reported in public (e.g., journal articles) and nonpublic (e.g., clinical study reports) sources by 2015. We prespecified outcome domains. From each source, we collected “outcomes” (i.e., domain, measure, metric, method of aggregation, and time point); “treatmen...
Source
This website uses cookies.
We use cookies to improve your online experience. By continuing to use our website we assume you agree to the placement of these cookies.
To learn more, you can find in our Privacy Policy.