Activation Landscapes as a Topological Summary of Neural Network Performance

We use topological data analysis (TDA) to study how data transforms as it passes through successive layers of a deep neural network (DNN). We compute the persistent homology of the activation data for each layer of the network and summarize this information using persistence landscapes. The resulting feature map provides both an informative visualization of the network and a kernel for statistical analysis and machine learning. A statistical test shows that it correlates with classification accuracy. We observe that the topological complexity often increases with training and that the topological complexity does not decrease with each layer.

[1]  Minsuk Kahng,et al.  Visual Analytics in Deep Learning: An Interrogative Survey for the Next Frontiers , 2018, IEEE Transactions on Visualization and Computer Graphics.

[2]  Andreas Uhl,et al.  Deep Learning with Topological Signatures , 2017, NIPS.

[3]  Leonidas J. Guibas,et al.  A Topology Layer for Machine Learning , 2019, AISTATS.

[4]  Pierre Baldi,et al.  Deep Architectures and Deep Learning in Chemoinformatics: The Prediction of Aqueous Solubility for Drug-Like Molecules , 2013, J. Chem. Inf. Model..

[5]  Sayan Mukherjee,et al.  Local homology transfer and stratification learning , 2012, SODA.

[6]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[7]  Rob Fergus,et al.  Visualizing and Understanding Convolutional Networks , 2013, ECCV.

[8]  Chao Chen,et al.  TopoGAN: A Topology-Aware Generative Adversarial Network , 2020, ECCV.

[9]  Yuriy Mileyko Another look at recovering local homology from samples of stratified sets , 2021, J. Appl. Comput. Topol..

[10]  Erich Elsen,et al.  Deep Speech: Scaling up end-to-end speech recognition , 2014, ArXiv.

[11]  Rickard Brüel Gabrielsson,et al.  Exposition and Interpretation of the Topology of Neural Networks , 2018, 2019 18th IEEE International Conference On Machine Learning And Applications (ICMLA).

[12]  Ulrich Bauer,et al.  Ripser: efficient computation of Vietoris–Rips persistence barcodes , 2019, Journal of Applied and Computational Topology.

[13]  Yi-Hsuan Yang,et al.  Applying Topological Persistence in Convolutional Neural Network for Music Audio Signals , 2016, ArXiv.

[14]  Afra Zomorodian,et al.  Computing Persistent Homology , 2004, SCG '04.

[15]  Mathieu Carrière,et al.  PersLay: A Neural Network Layer for Persistence Diagrams and New Graph Topological Signatures , 2020, AISTATS.

[16]  Dong Yu,et al.  Conversational Speech Transcription Using Context-Dependent Deep Neural Networks , 2012, ICML.

[17]  Peter Bubenik,et al.  Statistical topological data analysis using persistence landscapes , 2012, J. Mach. Learn. Res..

[18]  Facundo Mémoli,et al.  Topological Methods for the Analysis of High Dimensional Data Sets and 3D Object Recognition , 2007, PBG@Eurographics.

[19]  Herbert Edelsbrunner,et al.  Topological Persistence and Simplification , 2000, Proceedings 41st Annual Symposium on Foundations of Computer Science.

[20]  Lek-Heng Lim,et al.  Topology of deep neural networks , 2020, J. Mach. Learn. Res..

[21]  Ruslan Salakhutdinov,et al.  On Characterizing the Capacity of Neural Networks using Algebraic Topology , 2018, ArXiv.

[22]  Alberto Dassatti,et al.  giotto-tda: A Topological Data Analysis Toolkit for Machine Learning and Data Exploration , 2020, J. Mach. Learn. Res..

[23]  Levent Burak Kara,et al.  TopologyGAN: Topology Optimization Using Generative Adversarial Networks Based on Physical Fields Over the Initial Domain , 2020, Journal of Mechanical Design.

[24]  Yuhei Umeda,et al.  Topological Data Analysis for Arrhythmia Detection through Modular Neural Networks , 2019, Canadian AI.

[25]  Geoffrey E. Hinton,et al.  Speech recognition with deep recurrent neural networks , 2013, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing.

[26]  Gunnar E. Carlsson,et al.  Topological Approaches to Deep Learning , 2018, Topological Data Analysis.

[27]  Jose A. Perea,et al.  A Comparative Study of Machine Learning Methods for Persistence Diagrams , 2021, Frontiers in Artificial Intelligence.

[28]  Natalia Gimelshein,et al.  PyTorch: An Imperative Style, High-Performance Deep Learning Library , 2019, NeurIPS.

[29]  Larry Wasserman,et al.  PLLay: Efficient Topological Layer based on Persistent Landscapes , 2020, NeurIPS.

[30]  Pierre Baldi,et al.  Deep architectures for protein contact map prediction , 2012, Bioinform..

[31]  Hod Lipson,et al.  Understanding Neural Networks Through Deep Visualization , 2015, ArXiv.

[32]  Konrad P. Körding,et al.  Toward an Integration of Deep Learning and Neuroscience , 2016, bioRxiv.

[33]  C. Joslyn,et al.  Local homology of abstract simplicial complexes , 2018, 1805.11547.

[34]  Marc Niethammer,et al.  Connectivity-Optimized Representation Learning via Persistent Homology , 2019, ICML.

[35]  Stefano Ermon,et al.  Evaluating the Disentanglement of Deep Generative Models through Manifold Topology , 2020, ICLR.

[36]  Geraint Rees,et al.  Clinically applicable deep learning for diagnosis and referral in retinal disease , 2018, Nature Medicine.

[37]  Hayato Yamana,et al.  Topological Measurement of Deep Neural Networks Using Persistent Homology , 2020, ISAIM.

[38]  Abhishek Das,et al.  Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization , 2016, 2017 IEEE International Conference on Computer Vision (ICCV).

[39]  Yuhei Umeda,et al.  Time Series Classification via Topological Data Analysis , 2017, Inf. Media Technol..

[40]  Karsten M. Borgwardt,et al.  Neural Persistence: A Complexity Measure for Deep Neural Networks Using Algebraic Topology , 2018, ICLR.

[41]  Vidit Nanda,et al.  Local Cohomology and Stratification , 2017, Foundations of Computational Mathematics.

[42]  P Baldi,et al.  Enhanced Higgs boson to τ(+)τ(-) search with deep learning. , 2014, Physical review letters.

[43]  Jack L. Gallant,et al.  The DeepTune framework for modeling and characterizing neurons in visual cortex area V4 , 2018, bioRxiv.

[44]  Nithin Chalapathi,et al.  TopoAct: Visually Exploring the Shape of Activations in Deep Learning , 2019, Comput. Graph. Forum.

[45]  Ulrich Bauer,et al.  Phat - Persistent Homology Algorithms Toolbox , 2014, J. Symb. Comput..