Deep learning for detecting pulmonary tuberculosis via chest radiography: an international study across 10 countries

Purpose: Tuberculosis (TB) is one of the top 10 causes of death worldwide and disproportionately affects low-to-middle-income-countries. Though the WHO recommends chest radiographs (CXRs) to facilitate TB screening efforts and the means of acquiring CXRs are generally accessible, expertise in CXR interpretation poses a challenge to broad implementation of TB screening efforts in many parts of the world. To help mitigate this challenge, we developed a generalizable deep learning system (DLS) to help detect active TB and compared its performance to 14 radiologists, from both endemic (India) and non-endemic (US) practice settings. Materials and Methods: We trained a DLS using CXRs from 9 countries spanning Africa, Asia, and Europe. To improve generalization, we incorporated large-scale CXR pretraining, attention pooling, and semi-supervised learning via “noisy student”. The DLS was evaluated on a combined test set spanning sites in China, India, US, and Zambia, with all positives confirmed via microbiology or nucleic acid amplification testing (NAAT). The India test set was independent of those used in training. Another independent test set from a mining population in South Africa was also used to further evaluate the model. Given WHO targets of 90% sensitivity and 70% specificity, the DLS’s operating point was prespecified to favor sensitivity over specificity. Results: Across the combined test set spanning 4 countries, the DLS’s receiver operating characteristic (ROC) curve was above all 9 India-based radiologists (where TB is endemic), with an area under the curve (AUC) of 0.90 (95%CI 0.87-0.92). At the prespecified operating point, the DLS’s sensitivity (88%) was higher than the India-based radiologists (mean sensitivity: 75%, range 69-87%, p<0.001 for superiority), and the DLS’s specificity (79%) was non-inferior to these radiologists (mean specificity: 84%, range 78-88%, p=0.004). Similar trends were observed within HIV positive and sputum smear positive sub-groups, and in the additional South Africa test set. We additionally found that 5 USbased radiologists (where TB is not endemic) who also reviewed the cases were more sensitive but less specific than the India-based radiologists. The DLS was similarly non-inferior to this second cohort of radiologists at the same prespecified operating point. Depending on the simulated setting and prevalence, use of the DLS as a prioritization tool for NAAT could reduce the cost per positive TB case detected by 40-80% compared to the use of NAAT alone. Conclusion: We developed a DLS to detect active pulmonary TB on CXRs, which generalized to patient populations from 5 different regions of the world, and merits prospective evaluation to assist cost-effective screening efforts in settings with scarce access to radiologists. Operating point flexibility may permit customization of the DLS to account for site-specific factors such as TB prevalence, demographics, clinical resources, and customary practice patterns. Introduction Globally, 1 in 4 people are infected with Mycobacterium tuberculosis, and 5-10% of these individuals will develop active tuberculosis (TB) disease in their lifetime1,2. In 2019, the estimated TB mortality was 1.4 million, including 200,000 people who were human immunodeficiency virus (HIV) positive, and an estimated 2.9 million people who contracted TB were not formally reported due to a combination of underreporting, underdiagnosis, and pretreatment loss to follow up. Almost 90% of the active TB cases occur in a few dozen “high-burden” countries, many with scarce resources needed to tackle this public health problem.3 The anticipated rising burden of drug resistant TB poses an increased threat to both endemic and non-endemic parts of the world.4 Lastly, the COVID-19 pandemic that has caused devastation around the world has also disrupted efforts to combat TB: globally, 21% fewer (1.4 million) people received care for TB in 2020 than in 2019.5 In the past decade, there has been steady global support to combat this health crisis through the World Health Organization (WHO)’s End TB Strategy, the United Nations (UN)’s Sustainable Development Goals, and the Global Fund to fight AIDS, TB and malaria.6 Cost effective pulmonary TB screening using CXR has the potential to increase equity in access to healthcare, particularly in difficult-to-reach populations.7 In light of high patient volumes and limited access to timely expert interpretation of CXRs in many regions, there has been active research into using artificial intelligence to screen with a CXR followed by a corroborating diagnostic test;8–20,21 Such artificial intelligence-based triaging followed by GeneXpert testing for a confirmatory diagnosis was shown to be cost-effective compared to GeneXpert alone, and also substantially increased patient throughput.13 As part of their recently-published 2021 guidance, the WHO evaluated three independent computer-aided detection (CAD) software systems, and determined that the diagnostic accuracy and performance of CAD software was similar to human readers.7,9,13,17 Given the scarcity of experienced readers, as an alternative to human interpretation of CXR, the WHO now recommends CAD for both screening and triage in individuals 15 years or older.7 However, the WHO emphasized the importance of using a performant CAD system that has been tested on a population that is representative of the target population. In this study, we developed a deep learning system (DLS) to interpret CXRs for imaging features of active TB. Developing a universal TB classifier can be challenging not only due to the array of potential imaging features but because prevailing imaging features, severity of disease at presentation, and prevalence of TB and HIV can differ broadly on locale. Therefore, we validated our DLS using an aggregate of datasets from China, India, US, and Zambia that together reflect different regions, race/ethnicities, and local disease prevalence. We evaluated the DLS under two conditions: having a single prespecified operating point across all datasets, and when customized to radiologists’ performance in each locale. As diagnostic performance may be influenced by disease prevalence, we compared the DLS with two different cohorts of radiologists: one based in a TB-endemic region (India) and one based in a TB non-endemic region (United States). An analysis of HIV positive and sputum smear positive subgroups was also performed. Finally, we estimate cost savings for using this DLS as a triaging solution for nucleic acid amplification testing (NAAT) in screening settings.