Automated multi-class ground-truth labeling of H&E images for deep learning using multiplexed fluorescence microscopy

Manual annotation of Hematoxylin and Eosin (H&E) stained tissue images for deep learning classification is difficult, time consuming, and error-prone particularly for multi-class and rare-class problems. Chemical probes in immunohistochemistry (IHC) or immunofluorescence (IF) can automatically tag cellular structures; however, chemical labeling is difficult to use in training a deep classifier for H&E images (e.g. through serial sectioning and registration). In this work, we leverage the novel Multiplexed Immuno-Fluorescencent (MxIF) microscopy method developed by General Electric Global Research Center (GE GRC) which allows sequential, stain-image-bleach (SSB) application of protein markers on formalin-fixed, paraffin-embedded(FFPE) samples followed by traditional H&E staining to build chemically-annotated tissue maps of nuclei, cytoplasm, and cell membranes. This allows us to automate the creation of ground truth class-label maps for training an H&E-based tissue classifier. In this study, a tissue microarray consisting of 149 breast cancer and normal tissue cores were stained using MxIF for our three analytes, followed by traditional H&E staining. The MxIF stains for each TMA core were combined to create a “Virtual H&E” image, which is registered with the corresponding real H&E images. Each MxIF stained spot was segmented to obtain a class-label map for each analyte, which was then applied to the real H&E image to build a dataset consisting of the three analytes. A convolutional neural network (CNN) was then trained to classify this dataset. This system achieved an overall accuracy of 70%, suggesting that the MxIF system can provide useful labels for identifying hard to distinguish structures. A U-net was also trained to generate pseudo-IF stains from H&E and resulted in similar results.