Neural Architecture Search of SPD Manifold Networks

In this paper, we propose a new neural architecture search (NAS) problem of Symmetric Positive Definite (SPD) manifold networks. Unlike the conventional NAS problem, our problem requires to search for a unique computational cell called the SPD cell. This SPD cell serves as a basic building block of SPD neural architectures. An efficient solution to our problem is important to minimize the extraneous manual effort in the SPD neural architecture design. To accomplish this goal, we first introduce a geometrically rich and diverse SPD neural architecture search space for an efficient SPD cell design. Further, we model our new NAS problem using the supernet strategy which models the architecture search problem as a one-shot training process of a single supernet. Based on the supernet modeling, we exploit a differentiable NAS algorithm on our relaxed continuous search space for SPD neural architecture search. Statistical evaluation of our method on drone, action, and emotion recognition tasks mostly provides better results than the state-of-the-art SPD networks and NAS algorithms. Empirical results show that our algorithm excels in discovering better SPD network design, and providing models that are more than 3 times lighter than searched by state-of-the-art NAS algorithms.

[1]  Baba C. Vemuri,et al.  Recursive Computation of the Fréchet Mean on Non-positively Curved Riemannian Manifolds with Applications , 2016 .

[2]  Kirthevasan Kandasamy,et al.  Neural Architecture Search with Bayesian Optimisation and Optimal Transport , 2018, NeurIPS.

[3]  Geoffrey J. Gordon,et al.  DeepArchitect: Automatically Designing and Training Deep Architectures , 2017, ArXiv.

[4]  Kalyanmoy Deb,et al.  A fast and elitist multiobjective genetic algorithm: NSGA-II , 2002, IEEE Trans. Evol. Comput..

[5]  Larry S. Davis,et al.  Covariance discriminative learning: A natural and efficient approach to image set classification , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[6]  Maher Moakher,et al.  A Differential Geometric Approach to the Geometric Mean of Symmetric Positive-Definite Matrices , 2005, SIAM J. Matrix Anal. Appl..

[7]  Chi Zhang,et al.  Deep Manifold Learning of Symmetric Positive Definite Matrices with Application to Face Recognition , 2017, AAAI.

[8]  Yiming Yang,et al.  DARTS: Differentiable Architecture Search , 2018, ICLR.

[9]  Yunde Jia,et al.  A Robust Distance Measure for Similarity-Based Classification on the SPD Manifold , 2020, IEEE Transactions on Neural Networks and Learning Systems.

[10]  Vikas Singh,et al.  Dilated Convolutional Neural Networks for Sequential Manifold-Valued Data , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[11]  Jun Wang,et al.  Off-Policy Reinforcement Learning for Efficient and Effective GAN Architecture Search , 2020, ECCV.

[12]  Frank Hutter,et al.  Neural Architecture Search: A Survey , 2018, J. Mach. Learn. Res..

[13]  H. Karcher.  Riemannian center of mass and mollifier smoothing , 1977 .

[14]  Lorenzo Torresani,et al.  Connectivity Learning in Multi-Branch Networks , 2017, ArXiv.

[15]  Dawn Xiaodong Song,et al.  Differentiable Neural Network Architecture Search , 2018, ICLR.

[16]  Tido Röder,et al.  Documentation Mocap Database HDM05 , 2007 .

[17]  Lei Wang,et al.  DeepKSPD: Learning Kernel-matrix-based SPD Representation for Fine-grained Image Recognition , 2017, ECCV.

[18]  Luc Van Gool,et al.  Dense Non-Rigid Structure from Motion: A Manifold Viewpoint , 2020, ArXiv.

[19]  Quoc V. Le,et al.  Efficient Neural Architecture Search via Parameter Sharing , 2018, ICML.

[20]  Anoop Cherian,et al.  Scalable Dense Non-rigid Structure-from-Motion: A Grassmannian Perspective , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[21]  Mehrtash Tafazzoli Harandi,et al.  From Manifold to Manifold: Geometry-Aware Dimensionality Reduction for SPD Matrices , 2014, ECCV.

[22]  Yuandong Tian,et al.  FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[23]  Oriol Vinyals,et al.  Hierarchical Representations for Efficient Architecture Search , 2017, ICLR.

[24]  H. Wechsler,et al.  Micro-Doppler effect in radar: phenomenon, model, and simulation study , 2006, IEEE Transactions on Aerospace and Electronic Systems.

[25]  Alok Aggarwal,et al.  Regularized Evolution for Image Classifier Architecture Search , 2018, AAAI.

[26]  Theodore Lim,et al.  SMASH: One-Shot Model Architecture Search through HyperNetworks , 2017, ICLR.

[27]  Stephen P. Boyd,et al.  Differentiable Convex Optimization Layers , 2019, NeurIPS.

[28]  Lei Zhang,et al.  G2DeNet: Global Gaussian Distribution Embedding Network and Its Application to Visual Recognition , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[29]  Shiguang Shan,et al.  Learning Euclidean-to-Riemannian Metric for Point-to-Set Classification , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[30]  Xavier Pennec,et al.  Manifold-valued image processing with SPD matrices , 2020, Riemannian Geometric Statistics in Medical Image Analysis.

[31]  Suryansh Kumar,et al.  Jumping Manifolds: Geometry Aware Dense Non-Rigid Structure From Motion , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[32]  Bo Zhang,et al.  ScarletNAS: Bridging the Gap Between Scalability and Fairness in Neural Architecture Search , 2019, ArXiv.

[33]  Luc Van Gool,et al.  Building Deep Networks on Grassmann Manifolds , 2016, AAAI.

[34]  Vijay Vasudevan,et al.  Learning Transferable Architectures for Scalable Image Recognition , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[35]  Luc Van Gool,et al.  Deep Learning on Lie Groups for Skeleton-Based Action Recognition , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[36]  Ramón Fernández Astudillo,et al.  From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classification , 2016, ICML.

[37]  Luc Brun,et al.  A Neural Network Based on SPD Manifold Learning for Skeleton-Based Hand Gesture Recognition , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[38]  Quoc V. Le,et al.  Neural Architecture Search with Reinforcement Learning , 2016, ICLR.

[39]  Rudrasis Chakraborty,et al.  ManifoldNorm: Extending normalizations on Riemannian Manifolds , 2020, ArXiv.

[40]  Christian Jutten,et al.  Multiclass Brain–Computer Interface Classification by Riemannian Geometry , 2012, IEEE Transactions on Biomedical Engineering.

[41]  Rudrasis Chakraborty,et al.  ManifoldNet: A Deep Neural Network for Manifold-Valued Data With Applications , 2020, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[42]  Vladlen Koltun,et al.  Convolutional Sequence Modeling Revisited , 2018, ICLR.

[43]  Xavier Pennec,et al.  A Riemannian Framework for Tensor Computing , 2005, International Journal of Computer Vision.

[44]  Matthieu Cord,et al.  Riemannian batch normalization for SPD neural networks , 2019, NeurIPS.

[45]  Yu Qiao,et al.  Frame Attention Networks for Facial Expression Recognition in Videos , 2019, 2019 IEEE International Conference on Image Processing (ICIP).

[46]  Li Fei-Fei,et al.  Progressive Neural Architecture Search , 2017, ECCV.

[47]  Silvere Bonnabel,et al.  Stochastic Gradient Descent on Riemannian Manifolds , 2011, IEEE Transactions on Automatic Control.

[48]  Ludovic Denoyer,et al.  Learning Time/Memory-Efficient Deep Architectures with Budgeted Super Networks , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[49]  Bo Zhang,et al.  SCARLET-NAS: Bridging the Gap between Stability and Scalability in Weight-sharing Neural Architecture Search , 2019, 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW).

[50]  Jack J. Dongarra,et al.  Accelerating the SVD two stage bidiagonal reduction and divide and conquer using GPUs , 2017, Parallel Comput..

[51]  Jack J. Dongarra,et al.  Optimizing the SVD Bidiagonalization Process for a Batch of Small Matrices , 2017, ICCS.

[52]  Frank Hutter,et al.  Simple And Efficient Architecture Search for Convolutional Neural Networks , 2017, ICLR.

[53]  Jakob Verbeek,et al.  Convolutional Neural Fabrics , 2016, NIPS.

[54]  Shiguang Shan,et al.  Log-Euclidean Metric Learning on Symmetric Positive Definite Manifold with Application to Image Set Classification , 2015, ICML.

[55]  Wei Pan,et al.  BayesNAS: A Bayesian Approach for Neural Architecture Search , 2019, ICML.

[56]  Tamás D. Gedeon,et al.  Emotion Recognition In The Wild Challenge 2014: Baseline, Data and Protocol , 2014, ICMI.

[57]  Quoc V. Le,et al.  Understanding and Simplifying One-Shot Architecture Search , 2018, ICML.

[58]  Shiyu Chang,et al.  AutoGAN: Neural Architecture Search for Generative Adversarial Networks , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[59]  Yong Yu,et al.  Efficient Architecture Search by Network Transformation , 2017, AAAI.

[60]  Luc Van Gool,et al.  Neural Architecture Search as Sparse Supernet , 2020, AAAI.

[61]  Luc Van Gool,et al.  A Riemannian Network for SPD Matrix Learning , 2016, AAAI.

[62]  Lei Zhang,et al.  Deep CNNs Meet Global Covariance Pooling: Better Representation and Generalization , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[63]  Shifeng Zhang,et al.  DARTS+: Improved Differentiable Architecture Search with Early Stopping , 2019, ArXiv.

[64]  R. Bhatia,et al.  Riemannian geometry and matrix geometric means , 2006 .

[65]  Naiyan Wang,et al.  You Only Search Once: Single Shot Neural Architecture Search via Direct Sparse Optimization , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[66]  Fatih Murat Porikli,et al.  Region Covariance: A Fast Descriptor for Detection and Classification , 2006, ECCV.

[67]  Mehrtash Harandi,et al.  Dimensionality Reduction on SPD Manifolds: The Emergence of Geometry-Aware Methods , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[68]  Rongrong Ji,et al.  Multinomial Distribution Learning for Effective Neural Architecture Search , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[69]  Bo Zhang,et al.  Fair DARTS: Eliminating Unfair Advantages in Differentiable Architecture Search , 2020, ECCV.

[70]  Vikas Singh,et al.  A Statistical Recurrent Model on the Manifold of Symmetric Positive Definite Matrices , 2018, NeurIPS.

[71]  Ramesh Raskar,et al.  Accelerating Neural Architecture Search using Performance Prediction , 2017, ICLR.

[72]  Fatih Murat Porikli,et al.  Pedestrian Detection via Classification on Riemannian Manifolds , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.