External Testing of a Deep Learning Model for Lung Cancer Risk from Low-Dose Chest CT

Background: Sybil, an open-source deep learning model that uses low-dose CT (LDCT) for lung cancer prediction, requires rigorous external testing to confirm generalizability. Additionally, its utility in identifying individuals with high risk who never smoked or have light smoking histories remains unanswered. Purpose: To externally test Sybil for identifying individuals with high risk for lung cancer within an Asian health checkup cohort. Materials and Methods: This retrospective study analyzed LDCT scans from a single medical checkup facility in a study sample of individuals aged 50–80 years, collected between January 2004 and December 2021, with at least one follow-up scan. The predictive performance of the model for lung cancer risk over a 6-year period was assessed using the time-dependent area under the receiver operating characteristic curve (AUC). These evaluations were conducted in the overall study sample and within subgroups of patients with heavy (at least 20 pack-years) and never- or light smoking histories (ie, ever smoking [median, 2 pack-years]; ineligible for lung cancer screening per 2021 U.S. Preventive Services Task Force recommendations). Additionally, performance was evaluated according to the visibility of lung cancers on baseline LDCT scans. Results: Among 18 057 individuals (median age, 56 years [IQR, 52–61 years]; 11 267 male), 92 lung cancers were diagnosed (0.5%) within 6 years. Of these, 2848 had heavy smoking histories and 9943 had never- or light smoking histories, with 24 (0.8%) and 41 (0.4%) lung cancers, respectively. Sybil achieved AUCs of 0.91 for 1-year risk and 0.74 for 6-year risk. In the heavy-smoking subgroup, 1-year AUC was 0.94 (for visible lung cancers) and 6-year AUC was 0.70 (for future lung cancers). For the never- or light-smoking subgroup, Sybil had an AUC of 0.89 for visible lung cancers and 0.56 for future lung cancers. Conclusion: Sybil demonstrated excellent discriminative performance for visible lung cancers and acceptable performance for future lung cancers in Asian individuals with heavy smoking history but demonstrated poor performance for future lung cancers in a never- or light-smoking subgroup.