Visinity: Visual Spatial Neighborhood Analysis for Multiplexed Tissue Imaging Data

New multiplexed tissue imaging technologies have enabled the study of normal and diseased tissues in unprecedented detail. These methods are increasingly being applied to understand how cancer cells and immune response change during tumor development, progression, and metastasis as well as following treatment. Yet, existing analysis approaches focus on investigating small tissue samples on a per-cell basis, not taking into account the spatial proximity of cells, which indicates cell-cell interaction and specific biological processes in the larger cancer microenvironment. We present Visinity, a scalable visual analytics system to analyze cell interaction patterns across cohorts of whole-slide multiplexed tissue images. Our approach is based on a fast regional neighborhood computation, leveraging unsupervised learning to quantify, compare, and group cells by their surrounding cellular neighborhood. These neighborhoods can be visually analyzed in an exploratory and confirmatory workflow. Users can explore spatial patterns present across tissues through a scalable image viewer and coordinated views highlighting the neighborhood composition and spatial arrangements of cells. To verify or refine existing hypotheses, users can query for specific patterns to determine their presence and statistical significance. Findings can be interactively annotated, ranked, and compared in the form of small multiples. In two case studies with biomedical experts, we demonstrate that Visinity can identify common biological processes within a human tonsil and uncover novel white-blood networks and immune-tumor interactions.

[1]  Cody N. Heiser,et al.  Multiplexed 3D atlas of state transitions and immune interactions in colorectal cancer , 2021, bioRxiv.

[2]  E. Eisemann,et al.  Incorporating Texture Information into Dimensionality Reduction for High-Dimensional Images , 2022, 2022 IEEE 15th Pacific Visualization Symposium (PacificVis).

[3]  Fabian J Theis,et al.  Spatial components of molecular tissue biology , 2022, Nature Biotechnology.

[4]  Alyce A. Chen,et al.  The Spatial Landscape of Progression and Immunoediting in Primary Melanoma at Single-Cell Resolution , 2021, bioRxiv.

[5]  Jeffrey M. Spraggins,et al.  Viv: multiscale visualization of high-resolution multiplexed bioimaging data on the web , 2020, Nature Methods.

[6]  Hanspeter Pfister,et al.  Scope2Screen: Focus+Context Techniques for Pathology Tumor Assessment in Multivariate Image Data , 2021, IEEE Transactions on Visualization and Computer Graphics.

[7]  Shixia Liu,et al.  Revisiting Dimensionality Reduction Techniques for Visual Cluster Analysis: An Empirical Study , 2021, IEEE Transactions on Visualization and Computer Graphics.

[8]  Kurt Keutzer,et al.  Region Similarity Representation Learning , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[9]  C. Rudin,et al.  Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges , 2021, Statistics Surveys.

[10]  Samouil L. Farhi,et al.  MCMICRO: a scalable, modular image-processing pipeline for multiplexed tissue imaging , 2021, Nature Methods.

[11]  D. Kobak,et al.  Initialization is critical for preserving global data structure in both t-SNE and UMAP , 2021, Nature Biotechnology.

[12]  Frits Koning,et al.  ImaCytE: Visual Exploration of Cellular Micro-Environments for Imaging Mass Cytometry Data , 2021, IEEE Transactions on Visualization and Computer Graphics.

[13]  Tim Oates,et al.  Bringing UMAP Closer to the Speed of Light with GPU Acceleration , 2020, AAAI.

[14]  Thomas Höllt,et al.  Visual cohort comparison for spatial single-cell omics-data , 2020, IEEE Transactions on Visualization and Computer Graphics.

[15]  Jeff Johnson,et al.  Billion-Scale Similarity Search with GPUs , 2017, IEEE Transactions on Big Data.

[16]  Hanspeter Pfister,et al.  Minerva: a light-weight, narrative image browser for multiplexed tissue images , 2020, J. Open Source Softw..

[17]  Philipp Berens,et al.  Attraction-Repulsion Spectrum in Neighbor Embeddings , 2020, J. Mach. Learn. Res..

[18]  Q. Nguyen,et al.  stLearn: integrating spatial location, tissue morphology and gene expression to find cell types, cell-cell interactions and spatial trajectories within undissociated tissues , 2020, bioRxiv.

[19]  Ahmed Mahfouz,et al.  SCHNEL: Scalable clustering of high dimensional single-cell data , 2020, bioRxiv.

[20]  Jeremy L. Muhlich,et al.  Online narrative guides for illuminating tissue atlas data and digital pathology images , 2020, bioRxiv.

[21]  Roland Eils,et al.  Cell segmentation-free inference of cell types from in situ transcriptomics data , 2019, Nature Communications.

[22]  Sanjiv Kumar,et al.  Accelerating Large-Scale Inference with Anisotropic Vector Quantization , 2019, ICML.

[23]  Elmar Eisemann,et al.  GPGPU Linear Complexity t-SNE Optimization , 2018, IEEE Transactions on Visualization and Computer Graphics.

[24]  Brandy E. Olin,et al.  CytoMAP: A Spatial Analysis Toolbox Reveals Features of Myeloid Cell Organization in Lymphoid Tissues , 2019, bioRxiv.

[25]  P. Sorger,et al.  Facetto: Combining Unsupervised and Supervised Learning for Hierarchical Phenotype Analysis in Multi-Channel Image Data , 2019, bioRxiv.

[26]  Eric J. Ma,et al.  Peax: Interactive Visual Pattern Search in Sequential Data Using Unsupervised Deep Representation Learning , 2019, bioRxiv.

[27]  Kei-Hoi Cheung,et al.  Reporting and connecting cell type names and gating definitions through ontologies , 2019, BMC Bioinformatics.

[28]  Raphaël Marée,et al.  Cytomine: Toward an Open and Collaborative Software Platform for Digital Pathology Bridged to Molecular Investigations , 2018, Proteomics. Clinical applications.

[29]  Philipp Berens,et al.  The art of using t-SNE for single-cell transcriptomics , 2018, Nature Communications.

[30]  Zena Werb,et al.  Roles of the immune system in cancer: from tumor initiation to metastatic progression , 2018, Genes & development.

[31]  Guocheng Yuan,et al.  Identification of spatially associated subpopulations by combining scRNA-seq and sequential fluorescence in situ hybridization data , 2018, Nature Biotechnology.

[32]  Sandro Santagata,et al.  Highly multiplexed immunofluorescence imaging of human tissues and tumors using t-CyCIF and conventional optical microscopes , 2018, eLife.

[33]  Lai Guan Ng,et al.  Evaluation of UMAP as an alternative to t-SNE for single-cell data , 2018, bioRxiv.

[34]  Boudewijn P F Lelieveldt,et al.  Interactive Visual Exploration of 3D Mass Spectrometry Imaging Data Using Hierarchical Stochastic Neighbor Embedding Reveals Spatiomolecular Structures at Full Data Resolution , 2018, Journal of proteome research.

[35]  Leland McInnes,et al.  UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction , 2018, ArXiv.

[36]  Salil S. Bhate,et al.  Deep Profiling of Mouse Splenic Architecture with CODEX Multiplexed Imaging , 2017, Cell.

[37]  Yalong Yang,et al.  Many-to-Many Geographically-Embedded Flow Visualisation: An Evaluation , 2019, IEEE Transactions on Visualization and Computer Graphics.

[38]  Elmar Eisemann,et al.  Visual analysis of mass cytometry data by hierarchical stochastic neighbour embedding reveals rare cell types , 2017, Nature Communications.

[39]  Tina Tremel,et al.  VESPa 2.0: Data-Driven Behavior Models for Visual Analytics of Movement Sequences , 2017, 2017 International Symposium on Big Data Visual Analytics (BDVA).

[40]  Sarah A. Teichmann,et al.  Faculty Opinions recommendation of histoCAT: analysis of cell phenotypes and interactions in multiplex image cytometry data. , 2017 .

[41]  Benjamin M. Gyori,et al.  From word models to executable models of signaling networks using automated assembly , 2017, bioRxiv.

[42]  Elmar Eisemann,et al.  Approximated and User Steerable tSNE for Progressive Visual Analytics , 2015, IEEE Transactions on Visualization and Computer Graphics.

[43]  David H. Laidlaw,et al.  Colorgorical: Creating discriminable and preferable color palettes for information visualization , 2017, IEEE Transactions on Visualization and Computer Graphics.

[44]  P. Sorger,et al.  Cyclic Immunofluorescence (CycIF), A Highly Multiplexed Method for Single‐cell Imaging , 2016, Current protocols in chemical biology.

[45]  Martin Wattenberg,et al.  How to Use t-SNE Effectively , 2016 .

[46]  Boudewijn P F Lelieveldt,et al.  Data-driven identification of prognostic tumor subpopulations using spatially mapped t-SNE of mass spectrometry imaging data , 2016, Proceedings of the National Academy of Sciences.

[47]  Chih-Fong Tsai,et al.  The distance function effect on k-nearest neighbor classification for medical datasets , 2016, SpringerPlus.

[48]  M. Ebert,et al.  Spatial Autocorrelation in Mass Spectrometry Imaging. , 2016, Analytical chemistry.

[49]  Piotr Jankowski,et al.  Scalable and privacy-respectful interactive discovery of place semantics from human mobility traces , 2016, Inf. Vis..

[50]  Chris Allan,et al.  Metadata management for high content screening in OMERO , 2016, Methods.

[51]  Andreas Kerren,et al.  MobilityGraphs: Visual Analysis of Mass Mobility Dynamics via Spatio-Temporal Graphs and Clustering , 2016, IEEE Transactions on Visualization and Computer Graphics.

[52]  Jitendra Malik,et al.  Region-Based Convolutional Networks for Accurate Object Detection and Segmentation , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[53]  Siu Kwan Lam,et al.  Numba: a LLVM-based Python JIT compiler , 2015, LLVM '15.

[54]  Sean C. Bendall,et al.  Data-Driven Phenotypic Dissection of AML Reveals Progenitor-like Cells that Correlate with Prognosis , 2015, Cell.

[55]  Simon Li,et al.  OMERO and Bio-Formats 5: flexible access to large bioimaging datasets at scale , 2015, Medical Imaging.

[56]  Michael S. Bernstein,et al.  ImageNet Large Scale Visual Recognition Challenge , 2014, International Journal of Computer Vision.

[57]  Max Welling,et al.  Auto-Encoding Variational Bayes , 2013, ICLR.

[58]  Tamara Munzner,et al.  A Multi-Level Typology of Abstract Visualization Tasks , 2013, IEEE Transactions on Visualization and Computer Graphics.

[59]  Qing Li,et al.  Highly multiplexed single-cell analysis of formalin-fixed, paraffin-embedded cancer tissue , 2013, Proceedings of the National Academy of Sciences.

[60]  Luis Pizarro,et al.  Hyperspectral visualization of mass spectrometry imaging data. , 2013, Analytical chemistry.

[61]  Pascal Vincent,et al.  Representation Learning: A Review and New Perspectives , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[62]  Nadia Hashem,et al.  "I" and "others" , 2013 .

[63]  Tamara Munzner,et al.  Design Study Methodology: Reflections from the Trenches and the Stacks , 2012, IEEE Transactions on Visualization and Computer Graphics.

[64]  Miin-Shen Yang,et al.  A robust EM clustering algorithm for Gaussian mixture models , 2012, Pattern Recognit..

[65]  David S. Ebert,et al.  A correlative analysis process in a visual analytics environment , 2012, 2012 IEEE Conference on Visual Analytics Science and Technology (VAST).

[66]  Chun-Hsi Huang,et al.  Biological network motif detection: principles and practice , 2012, Briefings Bioinform..

[67]  P. Cochat,et al.  Et al , 2008, Archives de pediatrie : organe officiel de la Societe francaise de pediatrie.

[68]  W. Marsden I and J , 2012 .

[69]  Jason Dykes,et al.  Visualizing the Dynamics of London's Bicycle-Hire Scheme , 2011, Cartogr. Int. J. Geogr. Inf. Geovisualization.

[70]  Helga Thorvaldsdóttir,et al.  Molecular signatures database (MSigDB) 3.0 , 2011, Bioinform..

[71]  Timo Ropinski,et al.  Survey of glyph-based visualization techniques for spatial multivariate medical data , 2011, Comput. Graph..

[72]  Gaël Varoquaux,et al.  Scikit-learn: Machine Learning in Python , 2011, J. Mach. Learn. Res..

[73]  Jun S. Liu,et al.  The EM Algorithm and the Rise of Computational Biology , 2010, 1104.2180.

[74]  Mark V. Janikas,et al.  Spatial Statistics in ArcGIS , 2010 .

[75]  S. Johansson,et al.  Interactive Dimensionality Reduction Through User-defined Combinations of Quality Metrics , 2009, IEEE Transactions on Visualization and Computer Graphics.

[76]  Diansheng Guo,et al.  Flow Mapping and Multivariate Visualization of Large Spatial Interaction Data , 2009, IEEE Transactions on Visualization and Computer Graphics.

[77]  Samy Bengio,et al.  Large Scale Online Learning of Image Similarity Through Ranking , 2009, J. Mach. Learn. Res..

[78]  Tobias Schreck,et al.  A System for Interactive Visual Analysis of Large Graphs Using Motifs in Graph Editing and Aggregation , 2009, VMV.

[79]  M. Sheelagh T. Carpendale,et al.  Evaluating Information Visualizations , 2008, Information Visualization.

[80]  Geoffrey E. Hinton,et al.  Visualizing Data using t-SNE , 2008 .

[81]  W. Eric L. Grimson,et al.  Spatial Latent Dirichlet Allocation , 2007, NIPS.

[82]  J.C. Roberts,et al.  State of the Art: Coordinated & Multiple Views in Exploratory Visualization , 2007, Fifth International Conference on Coordinated and Multiple Views in Exploratory Visualization (CMV 2007).

[83]  P. Hanrahan,et al.  Flow map layout , 2005, IEEE Symposium on Information Visualization, 2005. INFOVIS 2005..

[84]  Cynthia A. Brewer,et al.  ColorBrewer.org: An Online Tool for Selecting Colour Schemes for Maps , 2003 .

[85]  Stefan Berchtold,et al.  Similarity clustering of dimensions for an enhanced visualization of multidimensional data , 1998, Proceedings IEEE Symposium on Information Visualization (Cat. No.98TB100258).

[86]  J. Hooton,et al.  Randomization tests: statistics for experimenters. , 1991, Computer methods and programs in biomedicine.

[87]  B. Ripley The second-order analysis of stationary point processes , 1976, Journal of Applied Probability.

[88]  W. Tobler A Computer Movie Simulating Urban Growth in the Detroit Region , 1970 .

[89]  R. Geary,et al.  The Contiguity Ratio and Statistical Mapping , 1954 .

[90]  P. Moran Notes on continuous stochastic phenomena. , 1950, Biometrika.