Design Characteristics Influence Performance of Clinical Prediction Rules in Validation: A Meta-Epidemiological Study.

Published on Jan 5, 2016in PLOS ONE3.24
路 DOI :10.1371/JOURNAL.PONE.0145779
Jong-Wook Ban3
Estimated H-index: 3
(University of Oxford),
Jose I. Emparanza18
Estimated H-index: 18
(Ciber)
+ 1 AuthorsAmanda Burls46
Estimated H-index: 46
(RMIT: RMIT University)
Sources
Abstract
BACKGROUND: Many new clinical prediction rules are derived and validated. But the design and reporting quality of clinical prediction research has been less than optimal. We aimed to assess whether design characteristics of validation studies were associated with the overestimation of clinical prediction rules' performance. We also aimed to evaluate whether validation studies clearly reported important methodological characteristics. METHODS: Electronic databases were searched for systematic reviews of clinical prediction rule studies published between 2006 and 2010. Data were extracted from the eligible validation studies included in the systematic reviews. A meta-analytic meta-epidemiological approach was used to assess the influence of design characteristics on predictive performance. From each validation study, it was assessed whether 7 design and 7 reporting characteristics were properly described. RESULTS: A total of 287 validation studies of clinical prediction rule were collected from 15 systematic reviews (31 meta-analyses). Validation studies using case-control design produced a summary diagnostic odds ratio (DOR) 2.2 times (95% CI: 1.2-4.3) larger than validation studies using cohort design and unclear design. When differential verification was used, the summary DOR was overestimated by twofold (95% CI: 1.2 -3.1) compared to complete, partial and unclear verification. The summary RDOR of validation studies with inadequate sample size was 1.9 (95% CI: 1.2 -3.1) compared to studies with adequate sample size. Study site, reliability, and clinical prediction rule was adequately described in 10.1%, 9.4%, and 7.0% of validation studies respectively. CONCLUSION: Validation studies with design shortcomings may overestimate the performance of clinical prediction rules. The quality of reporting among studies validating clinical prediction rules needs to be improved.
馃摉 Papers frequently viewed together
References59
Newest
#1Gary S. Collins (University of Oxford)H-Index: 68
#2Joris A H de GrootH-Index: 21
Last. Douglas G. Altman (University of Oxford)H-Index: 273
view all 11 authors...
Background Before considering whether to use a multivariable (diagnostic or prognostic) prediction model, it is essential that its performance be evaluated in data that were not used to develop the model (referred to as external validation). We critically appraised the methodological conduct and reporting of external validation studies of multivariable prediction models.
Source
#1Penny Whiting (UoB: University of Bristol)H-Index: 57
#2Anne W S Rutjes (University of Bern)H-Index: 58
Last. Susan Mallett (University of Oxford)H-Index: 60
view all 4 authors...
Abstract Objective To classify the sources of bias and variation and to provide an updated summary of the evidence of the effects of each source of bias and variation. Study Design and Setting We conducted a systematic review of studies of any design with the main objective of addressing bias or variation in the results of diagnostic accuracy studies. We searched MEDLINE, EMBASE, BIOSIS, the Cochrane Methodology Register, and Database of Abstracts of Reviews of Effects (DARE) from 2001 to Octobe...
Source
#1J. M. T. Hendriksen (Oklahoma State University Center for Health Sciences)H-Index: 1
#2Geert-Jan GeersingH-Index: 19
Last. J. A. H. de GrootH-Index: 1
view all 4 authors...
Summary Risk prediction models can be used to estimate the probability of either having (diagnostic model) or developing a particular disease or outcome (prognostic model). In clinical practice, these models are used to inform patients and guide therapeutic management. Examples from the field of venous thrombo-embolism (VTE) include the Wells rule for patients suspected of deep venous thrombosis and pulmonary embolism, and more recently prediction rules to estimate the risk of recurrence after a...
Source
#1Jelena Savovi膰 (UoB: University of Bristol)H-Index: 32
#2Hayley E Jones (UoB: University of Bristol)H-Index: 23
Last. Jonathan A C Sterne (UoB: University of Bristol)H-Index: 124
view all 18 authors...
Published evidence suggests that aspects of trial design lead to biased intervention effect estimates, but findings from different studies are inconsistent. This study combined data from 7 meta-epidemiologic studies and removed overlaps to derive a final data set of 234 unique meta-analyses containing 1973 trials. Outcome measures were classified as "mortality," "other objective," "or subjective," and Bayesian hierarchical models were used to estimate associations of trial characteristics with a...
Source
#1Walter Bouwmeester (UU: Utrecht University)H-Index: 6
#2Nicolaas P.A. Zuithoff (UU: Utrecht University)H-Index: 29
Last. Karel G.M. Moons (UU: Utrecht University)H-Index: 117
view all 8 authors...
textabstractBackground: We investigated the reporting and methods of prediction studies, focusing on aims, designs, participant selection, outcomes, predictors, statistical power, statistical methods, and predictive performance measures. Methods and Findings: We used a full hand search to identify all prediction studies published in 2008 in six high impact general medical journals. We developed a comprehensive item list to systematically score conduct and reporting of the studies, based on recen...
Source
#1Penny Whiting (UoB: University of Bristol)H-Index: 57
#2Anne W S RutjesH-Index: 58
Last. Patrick M.M. BossuytH-Index: 150
view all 9 authors...
In 2003, the QUADAS tool for systematic reviews of diagnostic accuracy studies was developed. Experience, anecdotal reports, and feedback suggested areas for improvement; therefore, QUADAS-2 was developed. This tool comprises 4 domains: patient selection, index test, reference standard, and flow and timing. Each domain is assessed in terms of risk of bias, and the first 3 domains are also assessed in terms of concerns regarding applicability. Signalling questions are included to help judge risk ...
Source
#1Emma Wallace (RCSI: Royal College of Surgeons in Ireland)H-Index: 24
#2Susan M Smith (RCSI: Royal College of Surgeons in Ireland)H-Index: 56
Last. Tom Fahey (RCSI: Royal College of Surgeons in Ireland)H-Index: 75
view all 9 authors...
Clinical Prediction Rules (CPRs) are tools that quantify the contribution of symptoms, clinical signs and available diagnostic tests, and in doing so stratify patients according to the probability of having a target outcome or need for a specified treatment. Most focus on the derivation stage with only a minority progressing to validation and very few undergoing impact analysis. Impact analysis studies remain the most efficient way of assessing whether incorporating CPRs into a decision making p...
Source
#1Gary S. Collins (University of Oxford)H-Index: 68
#2Susan Mallett (University of Oxford)H-Index: 60
Last. Ly-Mee Yu (University of Oxford)H-Index: 58
view all 4 authors...
Background The World Health Organisation estimates that by 2030 there will be approximately 350 million people with type 2 diabetes. Associated with renal complications, heart disease, stroke and peripheral vascular disease, early identification of patients with undiagnosed type 2 diabetes or those at an increased risk of developing type 2 diabetes is an important challenge. We sought to systematically review and critically assess the conduct and reporting of methods used to develop risk predict...
Source
Randomised trials use the play of chance to assign participants to comparison groups. The unpredictability of the process, if not subverted, should prevent systematic differences between comparison groups (selection bias). Differences due to chance will still occur and these are minimised by randomising a sufficiently large number of people.
Source
#1Alex J. Mitchell (Leicester Royal Infirmary)H-Index: 89
#2Victoria Bird (Centre for Mental Health)H-Index: 29
Last. Nicholas Meader (Centre for Mental Health)H-Index: 25
view all 4 authors...
Background The Geriatric Depression Scale (GDS) has been evaluated in individual studies, but its validity and added value in medical settings and nursing homes is uncertain. Therefore, the authors conducted a meta-analysis, analyzing the diagnostic accuracy of long, short, and ultrashort versions of the GDS and stratified this into those with and without cognitive impairment. Methods A comprehensive search identified 69 studies that measured the diagnostic validity of the GDS against a semistru...
Source
Cited By8
Newest
#1Kacper Lechowicz (Pomeranian Medical University)H-Index: 6
#2Aleksandra Szyli艅ska (Pomeranian Medical University)H-Index: 11
Last. Katarzyna Kotfis (Pomeranian Medical University)H-Index: 17
view all 7 authors...
Background Postoperative delirium (POD) is a serious complication of cardiac surgery. It is an acute neuropsychiatric syndrome. The aim of this study was to analyze the CARDEL Index, composed of advancing age, preoperative glycated hemoglobin and the platelet-to-WBC ratio (PWR) previously described and calculated, using a different patient database, to assess its usefulness as a marker for predicting postoperative delirium after coronary artery by-pass grafting (CABG). Methods A retrospective an...
Source
#1Jong-Wook BanH-Index: 3
#2M S Chan (University of Oxford)H-Index: 2
Last. Rafael PereraH-Index: 70
view all 6 authors...
Abstract Objective To evaluate design, methods, and reporting of impact studies of cardiovascular clinical prediction rules (CPRs). Study Design and Setting We conducted a systematic review. Impact studies of cardiovascular CPRs were identified by forward citation and electronic database searches. We categorized the design of impact studies as appropriate for randomized and non-randomized experiments, excluding uncontrolled before-after study. For impact studies with appropriate study design, we...
Source
#1Niamh C. Murphy (RCSI: Royal College of Surgeons in Ireland)H-Index: 5
#2Naomi Burke (RCSI: Royal College of Surgeons in Ireland)H-Index: 5
Last. Fionnuala Breathnach (RCSI: Royal College of Surgeons in Ireland)H-Index: 13
view all 11 authors...
Abstract Objective The ability to predict the need for emergency Cesarean delivery holds the potential to facilitate birth choices. The objective of the RECIPE study (Reducing Emergency Cesarean delivery and Improving the Primiparous Experience) was to externally validate a Cesarean delivery risk prediction model. This model, developed by the Genesis study, identified five key predictive factors for emergency Cesarean delivery: maternal age, maternal height, BMI, fetal head circumference (HC) an...
Source
#1Thomas P.A. Debray (UU: Utrecht University)H-Index: 31
#2Johanna A A G Damen (UU: Utrecht University)H-Index: 11
Last. Karel G.M. Moons (UU: Utrecht University)H-Index: 117
view all 8 authors...
It is widely recommended that any developed鈥攄iagnostic or prognostic鈥攑rediction model is externally validated in terms of its predictive performance measured by calibration and discrimination. When...
Source
#1Bradley E. Belsher (USU: Uniformed Services University of the Health Sciences)H-Index: 9
#2Derek J. Smolenski (Defense Health Agency)H-Index: 27
Last. Nancy A. Skopp (Defense Health Agency)H-Index: 20
view all 10 authors...
Importance Suicide prediction models have the potential to improve the identification of patients at heightened suicide risk by using predictive algorithms on large-scale data sources. Suicide prediction models are being developed for use across enterprise-level health care systems including the US Department of Defense, US Department of Veterans Affairs, and Kaiser Permanente. Objectives To evaluate the diagnostic accuracy of suicide prediction models in predicting suicide and suicide attempts ...
Source
#1Johanna A A G Damen (UU: Utrecht University)H-Index: 11
#2Thomas P.A. Debray (UU: Utrecht University)H-Index: 31
Last. Lotty Hooft (UU: Utrecht University)H-Index: 39
view all 7 authors...
Objectives To empirically assess the relation between study characteristics and prognostic model performance in external validation studies of multivariable prognostic models. Design Meta-epidemiological study. Data sources and study selection On 16 October 2018, we searched electronic databases for systematic reviews of prognostic models. Reviews from non-overlapping clinical fields were selected if they reported common performance measures (either the concordance (c)-statistic or the ratio of ...
Source
#1Yudai Iijima (Institute of Medical Science)H-Index: 1
#2Yasuyuki OkumuraH-Index: 14
Last. Atsushi NishidaH-Index: 23
view all 15 authors...
Abstract Background Anxiety disorders are prevalent among adolescents; however, without objective behavioral markers, anxiety disorders in adolescent populations may often go undiagnosed. Response inhibition is considered as a possible behavioral marker, based on the results with two-gate design, which can aid in early detection of anxiety disorders. The aim of this study was to investigate the relationship between response inhibition and anxiety using a large-scale population-based adolescent s...
Source
#1Jong-Wook Ban (University of Oxford)H-Index: 3
#2Richard Stevens (University of Oxford)H-Index: 46
Last. Rafael Perera (University of Oxford)H-Index: 70
view all 3 authors...
Clinical prediction rules (CPRs) should be externally validated by independent researchers. Although there are many cardiovascular CPRs, most have not been externally validated. It is not known why some CPRs are externally validated by independent researchers and others are not. We analyzed cardiovascular risk CPRs included in a systematic review. Independent external validations were identified by forward citation searches of derivation studies. Time between the publication of a cardiovascular ...
Source
This website uses cookies.
We use cookies to improve your online experience. By continuing to use our website we assume you agree to the placement of these cookies.
To learn more, you can find in our Privacy Policy.