Pitfalls in machine learning‐based assessment of tumor‐infiltrating lymphocytes in breast cancer: A report of the International Immuno‐Oncology Biomarker Working Group on Breast Cancer

The clinical significance of the tumor‐immune interaction in breast cancer is now established, and tumor‐infiltrating lymphocytes (TILs) have emerged as predictive and prognostic biomarkers for patients with triple‐negative (estrogen receptor, progesterone receptor, and HER2‐negative) breast cancer and HER2‐positive breast cancer. How computational assessments of TILs might complement manual TIL assessment in trial and daily practices is currently debated. Recent efforts to use machine learning (ML) to automatically evaluate TILs have shown promising results. We review state‐of‐the‐art approaches and identify pitfalls and challenges of automated TIL evaluation by studying the root cause of ML discordances in comparison to manual TIL quantification. We categorize our findings into four main topics: (1) technical slide issues, (2) ML and image analysis aspects, (3) data challenges, and (4) validation issues. The main reason for discordant assessments is the inclusion of false‐positive areas or cells identified by performance on certain tissue patterns or design choices in the computational implementation. To aid the adoption of ML for TIL assessment, we provide an in‐depth discussion of ML and image analysis, including validation issues that need to be considered before reliable computational reporting of TILs can be incorporated into the trial and routine clinical management of patients with triple‐negative breast cancer. © 2023 The Authors. The Journal of Pathology published by John Wiley & Sons Ltd on behalf of The Pathological Society of Great Britain and Ireland.

Rajarsi R. Gupta | N. Halama | J. A. van der Laak | N. Rajpoot | M. Salto‐Tellez | J. Hartman | E. Thompson | C. Sotiriou | Marvin Lerousseau | S. Loi | D. Larsimont | L. Pusztai | P. V. van Diest | A. Madabhushi | H. Horlings | J. Reis-Filho | A. Vincent-Salomon | E. Hytopoulos | D. Rimm | M. Amgad | L. Cooper | E. Balslev | S. Dudgeon | Yinyin Yuan | K. AbdulJabbar | P. Savas | F. Ciompi | D. Moore | J. Lennerz | Pawan Kirtani | G. Pruneri | S. Demaria | S. Adams | S. Loibl | Z. Kos | M. Hanna | S. Michiels | R. Salgado | A. Hida | A. Grigoriadis | A. Laenkholm | B. Ács | E. Bellolio | G. Broeckx | J. Giltnane | K. Siziopikou | K. Blenman | K. Korski | S. Irshad | S. Fineberg | Wentao Yang | W. Tran | Z. Husain | T. Taxter | A. Dahl | S. Tejpar | F. Symmans | J. Saltz | S. Hart | T. Rau | A. Harbhajanka | A. Coosemans | S. Sayed | E. Janssen | S. Gnjatic | Germán Corredor | T. Tramm | W. Gallagher | J. Teuwen | Arman Rahman | C. Jahangir | I. Alvarado-Cabrero | A. Khramtsov | N. Wahab | A. Kovács | Shachi Mittal | B. Rapoport | L. Kodach | A. Ly | Guray Akturk | D. Marks | Thomas Walter | F. Penault-Llorca | Søren Hauberg | T. Ebstrup | A. Roslind | E. McDonald | M. Sughayer | Xiaoxian Li | Sara Verbandt | T. Kataoka | Daniel G Sur | K. Kawaguchi | G. Acosta Haab | H. Wen | Reena Khiroya | M. Kahila | T. Papathomas | G. Viale | Y. Waumans | Umay Kiraz | O. Burgués | Sunao Tanaka | Najat Bouchmaa | S. Badve | S. Fox | R. Zin | C. Lang-Schwarz | C. Pinard | Durga Kharidehal | Ravi Mehrotra | C. Denkert | V. Bheemaraju | F. Minhas | J. Thagaard | Rashindrie Perera | Farid Azmoudeh-Ardalan | Shamim Mushtaq | Jonas S. Almeida | F. Deman | Paula I. González-Ericsson | S. Maley | Stephen M. Hewitt | Elisabeth Specht Stovgaard | A. Stenzinger | Claudio Fernandez-Martín | M. Vieth | D. B. Page | John M. S. Bartlett | Rajarsi Gupta | Nurkhairul Bariyah Baharun | Luciana Botinelly Mendonça Fujimoto | Alexandros Chardas | Maggie Chon U Cheang | Flavio Luis Dantas Portela | Johan Doré Hansen | Mahmoud Elghazawy | Zaheed Husain | Vidya Manur Narasimhamurthy | Hussain Nighat | Juan Carlos Pinto-Cardenas | J. M. Ribeiro | Ely Scott | G. Verghese | E. Scott | Paula I. Gonzalez-Ericsson | Konstanty Korski | Corinna Lang-Schwarz | D. Page

[1]  Sung-Bae Kim,et al.  Abstract PD9-05: Stromal tumor-infiltrating lymphocytes identify early-stage triple-negative breast cancer patients with favorable outcomes at 10-year follow-up in the absence of systemic therapy: a pooled analysis of 1835 patients , 2023, Cancer Research.

[2]  Mohamed Atef AlMoslemany,et al.  NuCLS: A scalable crowdsourcing approach and dataset for nucleus classification and segmentation in breast cancer , 2022, GigaScience.

[3]  Rajarsi R. Gupta,et al.  Spatial Characterization of Tumor-Infiltrating Lymphocytes and Breast Cancer Progression , 2022, Cancers.

[4]  Rajarsi R. Gupta,et al.  Deep Learning-Based Mapping of Tumor Infiltrating Lymphocytes in Whole Slide Images of 23 Types of Cancer , 2022, Frontiers in Oncology.

[5]  E. Provenzano,et al.  Intra-Tumour Heterogeneity Is One of the Main Sources of Inter-Observer Variation in Scoring Stromal Tumour Infiltrating Lymphocytes in Triple Negative Breast Cancer , 2021, Cancers.

[6]  J. Hartman,et al.  What do we still need to learn on digitally assessed biomarkers? , 2021, EBioMedicine.

[7]  Jakob Nikolas Kather,et al.  The impact of site-specific digital histology signatures on deep learning model accuracy and bias , 2021, Nature Communications.

[8]  Shuoyu Xu,et al.  A Computational Tumor-Infiltrating Lymphocyte Assessment Method Comparable with Visual Reporting Guidelines for Triple-Negative Breast Cancer , 2021, EBioMedicine.

[9]  Mohsin Bilal,et al.  Semantic annotation for computational pathology: multidisciplinary experience and best practice recommendations , 2021, The journal of pathology. Clinical research.

[10]  J. Hartman,et al.  An Open-Source, Automated Tumor-Infiltrating Lymphocyte Algorithm for Prognosis in Triple-Negative Breast Cancer , 2021, Clinical Cancer Research.

[11]  E. S. Stovgaard,et al.  Automated Quantification of sTIL Density with H&E-Based Digital Image Analysis Has Prognostic Potential in Triple-Negative Breast Cancers , 2021, Cancers.

[12]  G. Litjens,et al.  Deep learning in histopathology: the path to the clinic , 2021, Nature Medicine.

[13]  Jeroen van der Laak,et al.  Residual cyclegan for robust domain transformation of histopathological tissue slides , 2021, Medical Image Anal..

[14]  Ny,et al.  NuCLS: A scalable crowdsourcing approach and dataset for nucleus classification and segmentation in breast cancer , 2021, GigaScience.

[15]  G. Litjens,et al.  Optimized tumour infiltrating lymphocyte assessment for triple negative breast cancer prognostics , 2021, Breast.

[16]  D. Kerr,et al.  Designing deep learning studies in cancer diagnostics , 2021, Nature Reviews Cancer.

[17]  Australia,et al.  A Pathologist-Annotated Dataset for Validating Artificial Intelligence: A Project Description and Pilot Study , 2020, Journal of pathology informatics.

[18]  Yi-long Wu,et al.  Quantitative multiplex immunofluorescence analysis identifies infiltrating PD1+CD8+ and CD8+ T cells as predictive of response to neoadjuvant chemotherapy in breast cancer , 2020, Thoracic cancer.

[19]  Alexi Baidoshvili,et al.  Impact of rescanning and normalization on convolutional neural network performance in multi-center, whole-slide classification of prostate cancer , 2020, Scientific Reports.

[20]  Andrew H. Beck,et al.  Pitfalls in assessing stromal tumor infiltrating lymphocytes (sTILs) in breast cancer , 2020, npj Breast Cancer.

[21]  Andrew H. Beck,et al.  Report on computational assessment of Tumor Infiltrating Lymphocytes from the International Immuno-Oncology Biomarker Working Group , 2020, npj Breast Cancer.

[22]  Jacqueline A. Hall,et al.  Application of a risk-management framework for integration of stromal tumor-infiltrating lymphocytes in clinical trials , 2020, npj Breast Cancer.

[23]  Zhi Han,et al.  Deep-Learning–Based Characterization of Tumor-Infiltrating Lymphocytes in Breast Cancers From Histopathology Images and Multiomics Data , 2020, JCO clinical cancer informatics.

[24]  Roger R. Wang,et al.  Multi-panel immunofluorescence analysis of tumor infiltrating lymphocytes in triple negative breast cancer: Evolution of tumor immune profiles and patient prognosis , 2020, PloS one.

[25]  Geert J. S. Litjens,et al.  Efficient Out-of-Distribution Detection in Digital Pathology Using Multi-Head Convolutional Neural Networks , 2020, MIDL.

[26]  Anne L. Martel,et al.  Deep neural network models for computational histopathology: A survey , 2019, Medical Image Anal..

[27]  Reika Takamatsu,et al.  Quantitative digital image analysis of tumor-infiltrating lymphocytes in HER2-positive breast cancer , 2019, Virchows Archiv.

[28]  Geert J. S. Litjens,et al.  Learning to detect lymphocytes in immunohistochemistry with deep learning , 2019, Medical Image Anal..

[29]  Andrew H. Beck,et al.  Computational pathology definitions, best practices, and recommendations for regulatory guidance: a white paper from the Digital Pathology Association , 2019, The Journal of pathology.

[30]  Claes Lundström,et al.  A Closer Look at Domain Shift for Deep Learning in Histopathology , 2019, ArXiv.

[31]  Ashish Sharma,et al.  Utilizing Automated Breast Cancer Detection to Identify Spatial Distributions of Tumor Infiltrating Lymphocytes in Invasive Breast Cancer , 2019, The American journal of pathology.

[32]  Consuelo Morigi,et al.  Highlights of the 16th St Gallen International Breast Cancer Conference, Vienna, Austria, 20–23 March 2019: personalised treatments for patients with early breast cancer , 2019, Ecancermedicalscience.

[33]  Geert J. S. Litjens,et al.  Quantifying the effects of data augmentation and stain color normalization in convolutional neural networks for computational pathology , 2019, Medical Image Anal..

[34]  Jonathan D. Beezley,et al.  Structured crowdsourcing enables convolutional segmentation of histology images , 2019, Bioinform..

[35]  L. Russo,et al.  Locally advanced breast cancer: Tumor-infiltrating lymphocytes as a predictive factor of response to neoadjuvant chemotherapy. , 2019, European journal of surgical oncology : the journal of the European Society of Surgical Oncology and the British Association of Surgical Oncology.

[36]  Anindya Sarkar,et al.  Joint region and nucleus segmentation for characterization of tumor infiltrating lymphocytes in breast cancer , 2019, Medical Imaging.

[37]  Heikki Joensuu,et al.  Tumor-Infiltrating Lymphocytes and Prognosis: A Pooled Individual Patient Analysis of Early-Stage Triple-Negative Breast Cancers. , 2019, Journal of clinical oncology : official journal of the American Society of Clinical Oncology.

[38]  Timo Kohlberger,et al.  Whole-Slide Image Focus Quality: Automatic Assessment and Impact on AI Cancer Detection , 2019, Journal of pathology informatics.

[39]  Navid Farahani,et al.  A Practical Guide to Whole Slide Imaging: A White Paper From the Digital Pathology Association. , 2018, Archives of pathology & laboratory medicine.

[40]  Meyke Hermsen,et al.  1399 H&E-stained sentinel lymph node sections of breast cancer patients: the CAMELYON dataset , 2018, GigaScience.

[41]  P. Heikkilä,et al.  Reproducibility and predictive value of scoring stromal tumour infiltrating lymphocytes in triple-negative breast cancer: a multi-institutional study , 2018, Breast Cancer Research and Treatment.

[42]  Rajarsi R. Gupta,et al.  Spatial Organization and Molecular Correlation of Tumor-Infiltrating Lymphocytes Using Deep Learning on Pathology Images. , 2018, Cell reports.

[43]  S. Sleijfer,et al.  Breast cancer genomics and immuno-oncological markers to guide immune therapies. , 2017, Seminars in cancer biology.

[44]  Ross B. Girshick,et al.  Mask R-CNN , 2017, 1703.06870.

[45]  Zhen-ning Wang,et al.  Mechanism of immune evasion in breast cancer , 2017, OncoTargets and therapy.

[46]  Bram van Ginneken,et al.  A survey on deep learning in medical image analysis , 2017, Medical Image Anal..

[47]  Peter Bankhead,et al.  QuPath: Open source software for digital pathology image analysis , 2017, Scientific Reports.

[48]  Jianxu Chen,et al.  Automatic Lymphocyte Detection in H&E Images with Deep Neural Networks , 2016, ArXiv.

[49]  M. Disis,et al.  Clinical significance of tumor-infiltrating lymphocytes in breast cancer , 2016, Journal of Immunotherapy for Cancer.

[50]  S. Adams,et al.  Variation in the Incidence and Magnitude of Tumor-Infiltrating Lymphocytes in Breast Cancer Subtypes: A Systematic Review. , 2016, JAMA oncology.

[51]  Carsten Denkert,et al.  Clinical relevance of host immunity in breast cancer: from TILs to the clinic , 2016, Nature Reviews Clinical Oncology.

[52]  Andrew Janowczyk,et al.  Deep learning for digital pathology image analysis: A comprehensive tutorial with selected use cases , 2016, Journal of pathology informatics.

[53]  T. Nielsen,et al.  The evaluation of tumor-infiltrating lymphocytes (TILs) in breast cancer: recommendations by an International TILs Working Group 2014. , 2015, Annals of oncology : official journal of the European Society for Medical Oncology.

[54]  Siew Hooi Tan,et al.  Kuala Lumpur, Malaysia , 2012 .

[55]  F. Markowetz,et al.  Quantitative Image Analysis of Cellular Heterogeneity in Breast Tumors Complements Genomic Profiling , 2012, Science Translational Medicine.

[56]  Gyan Bhanot,et al.  Computerized Image-Based Detection and Grading of Lymphocytic Infiltration in HER2+ Breast Cancer Histopathology , 2010, IEEE Transactions on Biomedical Engineering.

[57]  Stephen B Fox,et al.  Quantification of regulatory T cells enables the identification of high-risk breast cancer patients and those at risk of late relapse. , 2006, Journal of clinical oncology : official journal of the American Society of Clinical Oncology.

[58]  L. R. Macfarlane,et al.  Department of pathology , 1961, Journal of the Royal Army Medical Corps.

[59]  M. Cigéhn,et al.  Quality control of whole-slide images through multi-class semantic segmentation of artifacts , 2021 .

[60]  Søren Hauberg,et al.  Can You Trust Predictive Uncertainty Under Real Dataset Shifts in Digital Pathology? , 2020, MICCAI.

[61]  Amal Al-Omari Clinical Research Management at the King Hussein Cancer Center, Amman, Jordan , 2017 .

[62]  Robert W. Lucky,et al.  The Grand Challenge , 1993 .

[63]  K. Jenpanich,et al.  [Drug administration]. , 1976, Thai journal of nursing.