External validation and recalibration of the Brock model to predict probability of cancer in pulmonary nodules using NLST data

Published on Jun 1, 2019in Thorax8.834
· DOI :10.1136/THORAXJNL-2018-212413
Audrey Winter1
Estimated H-index: 1
(UC: University of California),
Denise R. Aberle64
Estimated H-index: 64
(UC: University of California),
William Hsu14
Estimated H-index: 14
(UC: University of California)
Introduction We performed an external validation of the Brock model using the National Lung Screening Trial (NLST) data set, following strict guidelines set forth by the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis statement. We report how external validation results can be interpreted and highlight the role of recalibration and model updating. Materials and methods We assessed model discrimination and calibration using the NLST data set. Adhering to the inclusion/exclusion criteria reported by McWilliams et al , we identified 7879 non-calcified nodules discovered at the baseline low-dose CT screen with 2 years of follow-up. We characterised differences between Pan-Canadian Early Detection of Lung Cancer Study and NLST cohorts. We calculated the slope on the prognostic index and the intercept coefficient by fitting the original Brock model to NLST. We also assessed the impact of model recalibration and the addition of new covariates such as body mass index, smoking status, pack-years and asbestos. Results While the area under the curve (AUC) of the model was good, 0.905 (95% CI 0.882 to 0.928), a histogram plot showed that the model poorly differentiated between benign and malignant cases. The calibration plot showed that the model overestimated the probability of cancer. In recalibrating the model, the coefficients for emphysema, spiculation and nodule count were updated. The updated model had an improved calibration and achieved an optimism-corrected AUC of 0.912 (95% CI 0.891 to 0.932). Only pack-year history was found to be significant (p Conclusion While the Brock model achieved a high AUC when validated on the NLST data set, the model benefited from updating and recalibration. Nevertheless, covariates used in the model appear to be insufficient to adequately discriminate malignant cases.
📖 Papers frequently viewed together
3 Citations
11 Citations
#1Kaman Chung (Radboud University Nijmegen)H-Index: 9
#2Onno M. Mets (UU: Utrecht University)H-Index: 18
view all 10 authors...
Objective To assess the performance of the Brock malignancy risk model for pulmonary nodules detected in routine clinical setting. Methods In two academic centres in the Netherlands, we established a list of patients aged ≥40 years who received a chest CT scan between 2004 and 2012, resulting in 16 850 and 23 454 eligible subjects. Subsequent diagnosis of lung cancer until the end of 2014 was established through linking with the National Cancer Registry. A nested case–control study was performed...
20 CitationsSource
#1Anton Schreuder (AG: Analysis Group)H-Index: 3
Last. Bram van Ginneken (AG: Analysis Group)H-Index: 23
view all 6 authors...
Background All lung cancer CT screening trials used fixed follow-up intervals, which may not be optimal. We developed new lung cancer risk models for personalising screening intervals to 1 year or 2 years, and compared these with existing models. Methods We included participants in the CT arm of the National Lung Screening Trial (2002–2010) who underwent a baseline scan and a first annual follow-up scan and were not diagnosed with lung cancer in the first year. True and false positives and the a...
22 CitationsSource
#1Bumhee YangH-Index: 6
#2Byung Woo JhunH-Index: 17
Last. Kyungjong Lee (SMC: Samsung Medical Center)H-Index: 14
view all 11 authors...
OBJECTIVE: Four commonly used clinical models for predicting the probability of malignancy in pulmonary nodules were compared. While three of the models (Mayo Clinic, Veterans Association [VA], and Brock University) are based on clinical and computed tomography (CT) characteristics, one model (Herder) additionally includes the 18F-fluorodeoxyglucose (FDG) uptake value among the positron emission tomography (PET) characteristics. This study aimed to compare the predictive power of these four mode...
5 CitationsSource
#1Sarah J. van Riel (Radboud University Nijmegen)H-Index: 10
#2Francesco Ciompi (Radboud University Nijmegen)H-Index: 23
Last. Bram van GinnekenH-Index: 81
view all 18 authors...
Purpose: To compare human observers to a mathematically derived computer model for differentiation between malignant and benign pulmonary nodules detected on baseline screening computed tomography (CT) scans. Methods: A case-cohort study design was chosen. The study group consisted of 300 chest CT scans from the Danish Lung Cancer Screening Trial (DLCST). It included all scans with proven malignancies (n = 62) and two subsets of randomly selected baseline scans with benign nodules of all sizes (...
21 CitationsSource
#1Viswam S. Nair (USF: University of South Florida)H-Index: 17
#2Vandana Sundaram (Stanford University)H-Index: 25
Last. Michael K. Gould (KP: Kaiser Permanente)H-Index: 67
view all 4 authors...
16 CitationsSource
#1Yunlang She (Tongji University)H-Index: 12
#2Lilan Zhao (Tongji University)H-Index: 4
Last. Chang Chen (Tongji University)H-Index: 19
view all 15 authors...
Objectives To develop and validate a nomogram to estimate the pretest probability of malignancy in Chinese patients with solid solitary pulmonary nodule (SPN). Materials and Methods A primary cohort of 1798 patients with pathologically confirmed solid SPNs after surgery was retrospectively studied at five institutions from January 2014 to December 2015. A nomogram based on independent prediction factors of malignant solid SPN was developed. Predictive performance also was evaluated using the cal...
19 CitationsSource
#1Simone PerandiniH-Index: 9
Last. Stefania MontemezziH-Index: 15
view all 12 authors...
Objectives To achieve multicentre external validation of the Herder and Bayesian Inference Malignancy Calculator (BIMC) models.
12 CitationsSource
#2Simone PerandiniH-Index: 9
Last. Stefania MontemezziH-Index: 6
view all 10 authors...
Objectives To provide multicentre external validation of the Bayesian Inference Malignancy Calculator (BIMC) model by assessing diagnostic accuracy in a cohort of solitary pulmonary nodules (SPNs) collected in a clinic-based setting. To assess model impact on SPN decision analysis and to compare findings with those obtained via the Mayo Clinic model.
12 CitationsSource
#1Charles S. WhiteH-Index: 35
#2Ekta DharaiyaH-Index: 10
Last. Lilla BoroczkyH-Index: 1
view all 4 authors...
Receiver operating characteristic analysis showed that the full prediction model based on a National Lung Screening Trial data subset resulted in high discriminant value between benign and malignant nodules, similar to that in the Vancouver study.
22 CitationsSource
#1A Talwar (University of Oxford)H-Index: 6
#2Najib M. Rahman (University of Oxford)H-Index: 46
Last. Fergus V. Gleeson (University of Oxford)H-Index: 58
view all 5 authors...
Aim To estimate the probability of malignancy in small pulmonary nodules (PNs) based on clinical and radiological characteristics in a non-screening population that includes patients with a prior history of malignancy using three validated models. Materials and methods Retrospective data on clinical and radiological characteristics was collected from the medical records of 702 patients (379 men, 323 women; range 19–94 years) with PNs ≤12 mm in diameter at a single centre. The final diagnosis was...
7 CitationsSource
Cited By8
#1Hyungjin Kim (SNU: Seoul National University)H-Index: 26
#2Hyae Young KimH-Index: 24
Last. Yeol KimH-Index: 15
view all 4 authors...
To validate and compare the performance of the Brock model and Lung CT Screening Reporting and Data System (Lung-RADS) on nodules detected by baseline CT screening. We performed a secondary analysis of the Korean Lung Cancer Screening Project (K-LUCAS; ClinicalTrials.gov , NCT03394703), a nationwide, multicenter, prospective cohort study. From April 2017 to December 2018, low-dose CT screening was performed on high-risk subjects. Discrimination and calibration of Brock models 2a and 2b (i.e., fu...
1 CitationsSource
#1Zixing Wang (Peking Union Medical College)H-Index: 9
#2Ning Li (Peking Union Medical College)H-Index: 1
Last. Jingmei Jiang (Peking Union Medical College)H-Index: 12
view all 12 authors...
BACKGROUND The timeliness of diagnostic testing after positive screening remains suboptimal because of limited evidence and methodology, leading to delayed diagnosis of lung cancer and over-examination. We propose a radiomics approach to assist with planning of the diagnostic testing interval in lung cancer screening. METHODS From an institute-based lung cancer screening cohort, we retrospectively selected 92 patients with pulmonary nodules with diameters ≥ 3 mm at baseline (61 confirmed as lung...
#1Dongdong Gu (Hunan University)H-Index: 2
#2Guocai Liu (Hunan University)H-Index: 1
Last. Zhong XueH-Index: 22
view all 3 authors...
Abstract Computed tomography (CT) screening is an effective way for early detection of lung cancer in order to improve the survival rate of such a deadly disease. For more than two decades, image processing techniques such as nodule detection, segmentation, and classification have been extensively studied to assist physicians in identifying nodules from hundreds of CT slices to measure shapes and HU distributions of nodules automatically and to distinguish their malignancy. Thanks to new paralle...
2 CitationsSource
#1Maria Tsakok (John Radcliffe Hospital)H-Index: 7
#1Maria Tsakok (John Radcliffe Hospital)H-Index: 2
Last. Fergus V. Gleeson (John Radcliffe Hospital)H-Index: 58
view all 6 authors...
Abstract Purpose To determine how implementation of an artificial intelligence nodule algorithm, the Lung Cancer Prediction Convolutional Neural Network (LCP-CNN), at the point of incidental nodule detection would have influenced further investigation and management using a series of threshold scores at both the benign and malignant end of the spectrum. Method An observational retrospective study was performed in the assessment of nodules between 5-15 mm (158 benign, 32 malignant) detected on CT...
#1Sungmin Zo (SMC: Samsung Medical Center)
#2Sook-Young Woo (SMC: Samsung Medical Center)H-Index: 17
Last. Kyungjong Lee (SMC: Samsung Medical Center)H-Index: 14
view all 10 authors...
The next diagnostic step in cases of indeterminate radial probe endobronchial ultrasound (radial EBUS)-guided biopsy results remains uncertain. This study aimed to identify risk factors for malignancy based on clinical findings, chest computed tomography (CT), and radial EBUS images, and to estimate the risk of malignancy in lung nodules that showed indeterminate radial EBUS-guided biopsy results by constructing a nomogram. This retrospective study included 157 patients with indeterminate result...
#1Spencer C. DyerH-Index: 1
#2Brian J. BartholmaiH-Index: 33
Last. Chi Wan KooH-Index: 8
view all 3 authors...
Lung cancer remains the leading cause of cancer death in the United States. Screening with low-dose computed tomography (LDCT) has been proven to aid in early detection of lung cancer and reduce disease specific mortality. In 2014, the American College of Radiology (ACR) released version 1.0 of the Lung CT Screening Reporting and Data System (Lung-RADS) as a quality tool to standardize the reporting of lung cancer screening LDCT. In 2019, 5 years after the implementation of Lung-RADS version 1.0...
8 CitationsSource
#1Eui Jin Hwang (SNU: Seoul National University)H-Index: 17
#2Hyungjin Kim (SNU: Seoul National University)H-Index: 26
Last. Chang Min Park (New Generation University College)H-Index: 48
view all 5 authors...
OBJECTIVES To evaluate the calibration of a deep learning (DL) model in a diagnostic cohort and to improve model's calibration through recalibration procedures. METHODS Chest radiographs (CRs) from 1135 consecutive patients (M:F = 582:553; mean age, 52.6 years) who visited our emergency department were included. A commercialized DL model was utilized to identify abnormal CRs, with a continuous probability score for each CR. After evaluation of the model calibration, eight different methods were ...
#1E. O'Dowd (NUH: Nottingham University Hospitals NHS Trust)H-Index: 7
#2Kevin ten Haaf (EUR: Erasmus University Rotterdam)H-Index: 18
Two large randomised controlled trials of screening for lung cancer with low-dose CT (LDCT)—the National Lung Screening Trial (NLST) and the Dutch-Belgian lung cancer screening trial (Nederlands-Leuvens Longkanker Screenings Onderzoek (NELSON) trial)—have both shown substantial lung cancer mortality reduction in the LDCT arm.1 Although there is now strong evidence that screening for lung cancer with LDCT reduces lung cancer mortality, concerns have been raised about the potential costs and harms...
1 CitationsSource