VPNet: Variable Projection Networks

In this paper, we introduce VPNet, a novel model-driven neural network architecture based on variable projection (VP). Applying VP operators to neural networks results in learnable features, interpretable parameters, and compact network structures. This paper discusses the motivation and mathematical background of VPNet and presents experiments. The VPNet approach was evaluated in the context of signal processing, where we classified a synthetic dataset and real electrocardiogram (ECG) signals. Compared to fully connected and one-dimensional convolutional networks, VPNet offers fast learning ability and good accuracy at a low computational cost of both training and inference. Based on these advantages and the promising results obtained, we anticipate a profound impact on the broader field of signal processing, in particular on classification, regression and clustering problems.

[1]  Rob Fergus,et al.  Visualizing and Understanding Convolutional Networks , 2013, ECCV.

[2]  Carsten Peterson,et al.  Clustering ECG complexes using Hermite functions and self-organizing maps , 2000, IEEE Trans. Biomed. Eng..

[3]  D. Rumelhart.  Learning Internal Representations by Error Propagation, Parallel Distributed Processing , 1986 .

[4]  Raquel Urtasun,et al.  Understanding the Effective Receptive Field in Deep Convolutional Neural Networks , 2016, NIPS.

[5]  José Cristóbal Riquelme Santos,et al.  An Experimental Review on Deep Learning Architectures for Time Series Forecasting , 2020, Int. J. Neural Syst..

[6]  Péter Kovács,et al.  Ensemble Learning for Heartbeat Classification Using Adaptive Orthogonal Transformations , 2019, EUROCAST.

[7]  Luca Antiga,et al.  Automatic differentiation in PyTorch , 2017 .

[8]  Dimitrios I. Fotiadis,et al.  Epileptic Seizure Detection in EEGs Using Time–Frequency Analysis , 2009, IEEE Transactions on Information Technology in Biomedicine.

[9]  J. Zico Kolter,et al.  OptNet: Differentiable Optimization as a Layer in Neural Networks , 2017, ICML.

[10]  Sundeep Rangan,et al.  AMP-Inspired Deep Networks for Sparse Linear Inverse Problems , 2016, IEEE Transactions on Signal Processing.

[11]  Moncef Gabbouj,et al.  Real-Time Patient-Specific ECG Classification by 1-D Convolutional Neural Networks , 2016, IEEE Transactions on Biomedical Engineering.

[12]  Marco Paluszny,et al.  Recovery of relaxation rates in MRI T2-weighted brain images via exponential fitting , 2010 .

[13]  João Paulo Papa,et al.  FEMa: a finite element machine for fast learning , 2019, Neural Computing and Applications.

[14]  Alexios Balatsoukas-Stimming,et al.  Deep Unfolding for Communications Systems: A Survey and Some New Directions , 2019, 2019 IEEE International Workshop on Signal Processing Systems (SiPS).

[15]  C. Burrus,et al.  Introduction to Wavelets and Wavelet Transforms: A Primer , 1997 .

[16]  Dianne P. O'Leary,et al.  Variable projection for nonlinear least squares problems , 2012, Computational Optimization and Applications.

[17]  Pavlos I. Lazaridis,et al.  Discrete orthogonal Gauss–Hermite transform for optical pulse propagation analysis , 2003 .

[18]  Onur Avci,et al.  1D Convolutional Neural Networks and Applications: A Survey , 2019, Mechanical Systems and Signal Processing.

[19]  Gene H. Golub,et al.  The differentiation of pseudo-inverses and non-linear least squares problems whose variables separate , 1972, Milestones in Matrix Computation.

[20]  Mario Huemer,et al.  ECG segmentation using adaptive hermite functions , 2017, 2017 51st Asilomar Conference on Signals, Systems, and Computers.

[21]  Rene F. Swarttouw,et al.  Orthogonal polynomials , 2020, NIST Handbook of Mathematical Functions.

[22]  Jonathan Le Roux,et al.  Deep Unfolding: Model-Based Inspiration of Novel Deep Architectures , 2014, ArXiv.

[23]  Hojjat Adeli,et al.  Enhanced probabilistic neural network with local decision circles: A robust classifier , 2010, Integr. Comput. Aided Eng..

[24]  Thomas Pock,et al.  Learning a variational network for reconstruction of accelerated MRI data , 2017, Magnetic resonance in medicine.

[25]  Zhixun Su,et al.  Learning PDEs for Image Restoration via Optimal Control , 2010, ECCV.

[26]  Karl Kunisch,et al.  A Bilevel Optimization Approach for Parameter Learning in Variational Models , 2013, SIAM J. Imaging Sci..

[27]  Mario Huemer,et al.  Waveform Modeling by Adaptive Weighted Hermite Functions , 2019, ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[28]  R. Merletti,et al.  Hermite expansions of compact support waveforms: applications to myoelectric signals , 1994, IEEE Transactions on Biomedical Engineering.

[29]  Bin Liu,et al.  Unilateral sensorineural hearing loss identification based on double-density dual-tree complex wavelet transform and multinomial logistic regression , 2019, Integr. Comput. Aided Eng..

[30]  Ferenc Schipp,et al.  Adaptive Rational Transformations in Biomedical Signal Processing , 2019, Progress in Industrial Mathematics at ECMI 2018.

[31]  Jürgen Schmidhuber,et al.  Deep learning in neural networks: An overview , 2014, Neural Networks.

[32]  Andrei Paun,et al.  Simplified and Yet Turing Universal Spiking Neural P Systems with Communication on Request , 2018, Int. J. Neural Syst..

[33]  Philip de Chazal,et al.  Automatic classification of heartbeats using ECG morphology and heartbeat interval features , 2004, IEEE Transactions on Biomedical Engineering.

[34]  Sándor Fridli,et al.  Generalized Rational Variable Projection With Application in ECG Compression , 2020, IEEE Transactions on Signal Processing.

[35]  Jeffrey M. Hausdorff,et al.  Physionet: Components of a New Research Resource for Complex Physiologic Signals". Circu-lation Vol , 2000 .

[36]  Graham W. Taylor,et al.  Dataset Augmentation in Feature Space , 2017, ICLR.

[37]  Gian Luca Foresti,et al.  A Neural Network for Image Anomaly Detection with Deep Pyramidal Representations and Dynamic Routing , 2020, Int. J. Neural Syst..

[38]  Irena Orovic,et al.  On the parameterization of Hermite transform with application to the compression of QRS complexes , 2017, Signal Process..

[39]  Min Gan,et al.  Insights Into Algorithms for Separable Nonlinear Least Squares Problems , 2021, IEEE Transactions on Image Processing.

[40]  Abdul V. Roudsari,et al.  Automation bias: a systematic review of frequency, effect mediators, and mitigators , 2012, J. Am. Medical Informatics Assoc..

[41]  G. Golub,et al.  Separable nonlinear least squares: the variable projection method and its applications , 2003 .

[42]  U. Rajendra Acharya,et al.  Computer aided diagnosis of atrial arrhythmia using dimensionality reduction methods on transform domain representation , 2014, Biomed. Signal Process. Control..

[43]  Sándor Fridli,et al.  Heartbeat Classification of ECG Signals Using Rational Function Systems , 2017, EUROCAST.

[44]  Mattias Ohlsson,et al.  Detecting acute myocardial infarction in the 12-lead ECG using Hermite expansions and neural networks , 2004, Artif. Intell. Medicine.

[45]  Gergo Bognár,et al.  ECG heartbeat classification by means of variable rational projection , 2020, Biomed. Signal Process. Control..

[46]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[47]  E Weinan,et al.  A Proposal on Machine Learning via Dynamical Systems , 2017, Communications in Mathematics and Statistics.

[48]  José R. Dorronsoro,et al.  Deep support vector neural networks , 2020, Integr. Comput. Aided Eng..

[49]  Pablo Laguna,et al.  ECG Beat Representation and Delineation by Means of Variable Projection , 2021, IEEE Transactions on Biomedical Engineering.

[50]  Pascal Vincent,et al.  Representation Learning: A Review and New Perspectives , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[51]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[52]  W. Gautschi.  Orthogonal Polynomials: Computation and Approximation , 2004 .

[53]  A. Krylov,et al.  Two-dimensional hermite S-method for high-resolution inverse synthetic aperture radar imaging applications , 2010 .

[54]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[55]  M. Kubát.  An Introduction to Machine Learning , 2017, Springer International Publishing.

[56]  Cindy Cappelle,et al.  Multi-object tracking with discriminant correlation filter based deep learning tracker , 2019, Integr. Comput. Aided Eng..

[57]  Haikun Wei,et al.  A Deep Fourier Neural Network for Seizure Prediction Using Convolutional Neural Network and Ratios of Spectral Power , 2021, Int. J. Neural Syst..

[58]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[59]  Weidong Zhou,et al.  Automatic Seizure Detection Based on S-Transform and Deep Convolutional Neural Network , 2020, Int. J. Neural Syst..

[60]  Jelena Kovacevic,et al.  Efficient Compression of QRS Complexes Using Hermite Expansion , 2012, IEEE Transactions on Signal Processing.

[61]  Maciej Ławryńczuk,et al.  Practical nonlinear predictive control algorithms for neural Wiener models , 2013 .

[62]  Hojjat Adeli,et al.  Parallel backpropagation learning algorithms on CRAY Y-MP8/864 supercomputer , 1993, Neurocomputing.

[63]  Yandong Tang,et al.  Identification of Nonlinear Dynamic Systems Using Hammerstein-Type Neural Network , 2014 .

[64]  Gergo Bognár,et al.  ECG Segmentation by Adaptive Rational Transform , 2019, EUROCAST.

[65]  Maciej Lawrynczuk,et al.  Computationally efficient nonlinear predictive control based on neural Wiener models , 2010, Neurocomputing.

[66]  Eldad Haber,et al.  Reversible Architectures for Arbitrarily Deep Residual Neural Networks , 2017, AAAI.

[67]  Hojjat Adeli,et al.  A dynamic ensemble learning algorithm for neural networks , 2019, Neural Computing and Applications.

[68]  Gregory S. Chirikjian,et al.  A fast Hermite transform , 2008, Theor. Comput. Sci..

[69]  Rong-Ching Wu,et al.  Alternative Diagnosis of Epilepsy in Children Without Epileptiform Discharges Using Deep Convolutional Neural Networks , 2020, Int. J. Neural Syst..

[70]  Boris Escalante-Ramírez,et al.  Optical flow estimation in cardiac CT images using the steered Hermite transform , 2013, Signal Process. Image Commun..

[71]  Moncef Gabbouj,et al.  Epileptic Seizure Classification of EEG Time-Series Using Rational Discrete Short-Time Fourier Transform , 2015, IEEE Transactions on Biomedical Engineering.

[72]  Linqiang Pan,et al.  Spiking Neural P Systems With Communication on Request and Mute Rules , 2017, IEEE Transactions on Parallel and Distributed Systems.

[73]  Antonio Fernández-Caballero,et al.  Deep Support Vector Machines for the Identification of Stress Condition from Electrodermal Activity , 2020, Int. J. Neural Syst..

[74]  Geoffrey Ye Li,et al.  Model-Driven Deep Learning for Physical Layer Communications , 2018, IEEE Wireless Communications.

[75]  John Suckling,et al.  Ensemble Deep Learning on Large, Mixed-Site fMRI Datasets in Autism and Other Tasks , 2020, Int. J. Neural Syst..

[76]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[77]  Tadashi Wadayama,et al.  Trainable ISTA for Sparse Signal Recovery , 2018, IEEE Transactions on Signal Processing.

[78]  Hojjat Adeli,et al.  Object-oriented backpropagation and its application to structural design , 1994, Neurocomputing.

[79]  V. Pereyraa,et al.  Variable projections neural network training , 2022 .

[80]  Remco J. Renken,et al.  Scaled Subprofile Modeling and Convolutional Neural Networks for the Identification of Parkinson's Disease in 3D Nuclear Imaging Data , 2019, Int. J. Neural Syst..

[81]  Razvan Pascanu,et al.  Deep Learners Benefit More from Out-of-Distribution Examples , 2011, AISTATS.

[82]  G.B. Moody,et al.  The impact of the MIT-BIH Arrhythmia Database , 2001, IEEE Engineering in Medicine and Biology Magazine.

[83]  U. Rajendra Acharya,et al.  Current methods in electrocardiogram characterization , 2014, Comput. Biol. Medicine.

[84]  Sebastián Ventura,et al.  Performing Multi-Target Regression via a Parameter Sharing-Based Deep Network , 2019, Int. J. Neural Syst..

[85]  Yunjin Chen,et al.  Trainable Nonlinear Reaction Diffusion: A Flexible Framework for Fast and Effective Image Restoration , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[86]  William Robson Schwartz,et al.  ECG-based heartbeat classification for arrhythmia detection: A survey , 2016, Comput. Methods Programs Biomed..

[87]  Geoffrey E. Hinton,et al.  Reducing the Dimensionality of Data with Neural Networks , 2006, Science.

[88]  Ami Wiesel,et al.  Learning to Detect , 2018, IEEE Transactions on Signal Processing.

[89]  Yoram Singer,et al.  Adaptive Subgradient Methods for Online Learning and Stochastic Optimization , 2011, J. Mach. Learn. Res..