Sample size considerations for the external validation of a multivariable prognostic model: a resampling study.

Published on Jan 30, 2016in Statistics in Medicine2.373
· DOI :10.1002/SIM.6787
Gary S. Collins68
Estimated H-index: 68
(University of Oxford),
Emmanuel O. Ogundimu7
Estimated H-index: 7
(University of Oxford),
Douglas G. Altman273
Estimated H-index: 273
(University of Oxford)
Sources
Abstract
After developing a prognostic model, it is essential to evaluate the performance of the model in samples independent from those used to develop the model, which is often referred to as external validation. However, despite its importance, very little is known about the sample size requirements for conducting an external validation. Using a large real data set and resampling methods, we investigate the impact of sample size on the performance of six published prognostic models. Focussing on unbiased and precise estimation of performance measures (e.g. the c-index, D statistic and calibration), we provide guidance on sample size for investigators designing an external validation study. Our study suggests that externally validating a prognostic model requires a minimum of 100 events and ideally 200 (or more) events. © 2015 The Authors. Statistics in Medicine Published by John Wiley and Sons Ltd.
Figures & Tables
Download
📖 Papers frequently viewed together
References51
Newest
#1Thomas P.A. Debray (UU: Utrecht University)H-Index: 31
#2Yvonne Vergouwe (EUR: Erasmus University Rotterdam)H-Index: 35
Last. Karel G.M. Moons (UU: Utrecht University)H-Index: 117
view all 6 authors...
Objectives It is widely acknowledged that the performance of diagnostic and prognostic prediction models should be assessed in external validation studies with independent data from “different but related” samples as compared with that of the development sample. We developed a framework of methodological steps and statistical methods for analyzing and enhancing the interpretation of results from external validation studies of prediction models. Study Design and Setting We propose to quantify the...
Source
#1Gary S. Collins (University of Oxford)H-Index: 68
#3Douglas G. Altman (University of Oxford)H-Index: 273
Last. Karel G.M. Moons (UU: Utrecht University)H-Index: 117
view all 4 authors...
Prediction models are developed to aid health-care providers in estimating the probability or risk that a specific disease or condition is present (diagnostic models) or that a specific event will occur in the future (prognostic models), to inform their decision making. However, the overwhelming evidence shows that the quality of reporting of prediction model studies is poor. Only with full and clear reporting of information on all aspects of a prediction model can risk of bias and potential use...
Source
The TRIPOD (Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis) Statement includes a 22-item checklist, which aims to improve the reporting of studies developing, validating, or updating a prediction model, whether for diagnostic or prognostic purposes. The TRIPOD Statement aims to improve the transparency of the reporting of a prediction model study regardless of the study methods used. This explanation and elaboration document describes the rational...
Source
#1George C.M. Siontis (UoI: University of Ioannina)H-Index: 29
#2Ioanna Tzoulaki (Imperial College London)H-Index: 72
Last. John P. A. Ioannidis (Stanford University)H-Index: 205
view all 4 authors...
Abstract Objectives To evaluate how often newly developed risk prediction models undergo external validation and how well they perform in such validations. Study Design and Setting We reviewed derivation studies of newly proposed risk models and their subsequent external validations. Study characteristics, outcome(s), and models' discriminatory performance [area under the curve, (AUC)] in derivation and validation studies were extracted. We estimated the probability of having a validation, chang...
Source
#1Gary S. Collins (University of Oxford)H-Index: 68
#2Joris A H de GrootH-Index: 21
Last. Douglas G. Altman (University of Oxford)H-Index: 273
view all 11 authors...
Background Before considering whether to use a multivariable (diagnostic or prognostic) prediction model, it is essential that its performance be evaluated in data that were not used to develop the model (referred to as external validation). We critically appraised the methodological conduct and reporting of external validation studies of multivariable prediction models.
Source
#1Gary S. Collins (University of Oxford)H-Index: 68
#2Douglas G. Altman (University of Oxford)H-Index: 273
: Early identification of ovarian cancer is an unresolved challenge and the predictive value of single symptoms is limited. We evaluated the performance of QCancer(®) (Ovarian) prediction model for predicting the risk of ovarian cancer in a UK cohort of general practice patients. A total of 1.1 million patients registered with a general practice surgery between 1 January 2000 and 30 June 2008, aged 30-84 years with 735 ovarian cancer cases, were included in the analysis. Ovarian cancer was defin...
Source
#1Gary S. Collins (University of Oxford)H-Index: 68
#2Douglas G. Altman (University of Oxford)H-Index: 273
Abstract Introduction : To evaluate the performance of QCancer ® (Renal) for predicting the absolute risk of renal tract cancer in a large independent UK cohort of patients from general practice records. Materials and methods : Open cohort study to validate QCancer ® (Renal) prediction model. Record from 365 practices from United Kingdom contributing to The Health Improvement Network (THIN) database. 2.1 million patients registered with a general practice surgery between 01 January 2000 and 30 J...
Source
#1Patrick Royston (UCL: University College London)H-Index: 95
#2Douglas G. Altman (University of Oxford)H-Index: 273
Background A prognostic model should not enter clinical practice unless it has been demonstrated that it performs a useful role. External validation denotes evaluation of model performance in a sample independent of that used to develop the model. Unlike for logistic regression models, external validation of Cox models is sparsely treated in the literature. Successful validation of a model means achieving satisfactory discrimination and calibration (prediction accuracy) in the validation sample....
Source
#1Gary S. Collins (University of Oxford)H-Index: 68
#2Douglas G. Altman (University of Oxford)H-Index: 273
Abstract Objective To evaluate the performance of QCancer® (Gastro-Oesophageal) for predicting the risk of undiagnosed gastro-oesophageal cancer in an independent UK cohort of patients from general practice records. Design Open cohort study to validate QCancer® (Gastro-Oesophageal) prediction model. Three hundred sixty-five practices from the United Kingdom contributing to The Health Improvement Network database. 2.1 million patients registered with a general practice surgery between 01 January ...
Source
#1Ewout W. Steyerberg (EUR: Erasmus University Rotterdam)H-Index: 161
#2Karel G.M. MoonsH-Index: 117
Last. Douglas G. Altman (University of Oxford)H-Index: 273
view all 9 authors...
Prognostic models are abundant in the medical literature yet their use in practice seems limited. In this article, the third in the PROGRESS series, the authors review how such models are developed and validated, and then address how prognostic models are assessed for their impact on practice and patient outcomes, illustrating these ideas with examples.
Source
Cited By250
Newest
#1Yohei Okada (Kyoto University)H-Index: 8
#2Asami OkadaH-Index: 2
Last. Tadahiro Goto (UTokyo: University of Tokyo)H-Index: 17
view all 5 authors...
Abstract null null Background null The POP score was developed as an easy screening tool for predicting obstetrics and gynecological (OBGYN) diseases in the emergency department (ED), and consists of three predictors, each representing one point: past history of OBGYN diseases, no fever or digestive symptoms, and peritoneal irritation signs). However, its external validity has not yet been evaluated. We aimed to perform the external validation of the POP score. null null null Methods null This i...
Source
Source
#1E. O'Dowd (NUH: Nottingham University Hospitals NHS Trust)H-Index: 8
#2Kevin ten Haaf (EUR: Erasmus University Rotterdam)H-Index: 22
Last. David R Baldwin (NUH: Nottingham University Hospitals NHS Trust)H-Index: 46
view all 12 authors...
Lung cancer screening is effective if offered to people at increased risk of the disease. Currently, direct contact with potential participants is required for evaluating risk. A way to reduce the number of ineligible people contacted might be to apply risk-prediction models directly to digital primary care data, but model performance in this setting is unknown. null Method null The Clinical Practice Research Datalink, a computerised, longitudinal primary care database, was used to evaluate the ...
Source
#1Daniel M. Goldenholz (Harvard University)H-Index: 15
#2Haoqi Sun (Harvard University)H-Index: 12
Last. M. Brandon Westover (Harvard University)H-Index: 39
view all 4 authors...
OBJECTIVE: Before integrating new machine learning (ML) into clinical practice, algorithms must undergo validation. Validation studies require sample size estimates. Unlike hypothesis testing studies seeking a p-value, the goal of validating predictive models is obtaining estimates of model performance. Our aim was to provide a standardized, data distribution- and model-agnostic approach to sample size calculations for validation studies of predictive ML models. MATERIALS AND METHODS: Sample Siz...
Source
#1C. Yang (EUR: Erasmus University Rotterdam)
#2Jan A. Kors (Erasmus University Medical Center)H-Index: 70
Last. Peter R. Rijnbeek (Erasmus University Medical Center)H-Index: 28
view all 10 authors...
Objectives This systematic review aims to provide further insights into the conduct and reporting of clinical prediction model development and validation over time. We focus on assessing the reporting of information necessary to enable external validation by other investigators. Materials and Methods We searched Embase, Medline, Web-of-Science, Cochrane Library and Google Scholar to identify studies that developed one or more multivariable prognostic prediction models using electronic health rec...
Source
#1Daniel Murphy (UMN: University of Minnesota)H-Index: 3
#2Scott Reule (Veterans Health Administration)H-Index: 9
Last. Paul E. Drawz (UMN: University of Minnesota)H-Index: 24
view all 4 authors...
Abstract null Rationale & Objective null Risk-factors for acute kidney injury (AKI) in the hospital have been well studied. Yet, risk-factors for identifying high-risk patients for AKI occurring and managed in the outpatient setting are unknown and may differ. null Study Design null Predictive model development and external validation using observational electronic health record data. null Setting & Participants null Patients aged 18-90 years with recurrent primary care encounters, known baselin...
Source
Intraoperative hypotension (IOH) is common during major surgery and is associated with a poor postoperative outcome. Hypotension Prediction Index (HPI) is an algorithm derived from machine learning that uses the arterial waveform to predict IOH. The aim of this study was to assess the diagnostic ability of HPI working with non-invasive ClearSight system in predicting impending hypotension in patients undergoing major gynaecologic oncologic surgery (GOS). In this retrospective analysis hemodynami...
Source
Source
#1Erkan Kalafat (METU: Middle East Technical University)H-Index: 15
#2Smriti Prasad (St George's, University of London)H-Index: 1
Last. Asma Khalil (St George's, University of London)H-Index: 56
view all 26 authors...
Background null Pregnant women are at an increased risk of mortality and morbidity owing to COVID-19. Many studies have reported on the association of COVID-19 with pregnancy-specific adverse outcomes, but prediction models utilizing large cohorts of pregnant women are still lacking for estimating the risk of maternal morbidity and other adverse events. null null null Objective null The main aim of this study was to develop a prediction model to quantify the risk of progression to critical COVID...
Source
#3Mina Fazel (Oxford Health NHS Foundation Trust)H-Index: 29
Background There has been a rapid growth in the publication of new prediction models relevant to child and adolescent mental health. However, before their implementation into clinical services, it is necessary to appraise the quality of their methods and reporting. We conducted a systematic review of new prediction models in child and adolescent mental health, and examined their development and validation. Method We searched five databases for studies developing or validating multivariable predi...
Source
This website uses cookies.
We use cookies to improve your online experience. By continuing to use our website we assume you agree to the placement of these cookies.
To learn more, you can find in our Privacy Policy.