A Novel Recurrent Neural Network-Based Ultra-Fast, Robust, and Scalable Solver for Inverting a “Time-Varying Matrix”

The concept presented in this paper is based on previous dynamical methods to realize a time-varying matrix inversion. It is essentially a set of coupled ordinary differential equations (ODEs) which does indeed constitute a recurrent neural network (RNN) model. The coupled ODEs constitute a universal modeling framework for realizing a matrix inversion provided the matrix is invertible. The proposed model does converge to the inverted matrix if the matrix is invertible, otherwise it converges to an approximated inverse. Although various methods exist to solve a matrix inversion in various areas of science and engineering, most of them do assume that either the time-varying matrix inversion is free of noise or they involve a denoising module before starting the matrix inversion computation. However, in the practice, the noise presence issue is a very serious problem. Also, the denoising process is computationally expensive and can lead to a violation of the real-time property of the system. Hence, the search for a new ‘matrix inversion’ solving method inherently integrating noise-cancelling is highly demanded. In this paper, a new combined/extended method for time-varying matrix inversion is proposed and investigated. The proposed method is extending both the gradient neural network (GNN) and the Zhang neural network (ZNN) concepts. Our new model has proven that it has exponential stability according to Lyapunov theory. Furthermore, when compared to the other previous related methods (namely GNN, ZNN, Chen neural network, and integration-enhanced Zhang neural network or IEZNN) it has a much better theoretical convergence speed. To finish, all named models (the new one versus the old ones) are compared through practical examples and both their respective convergence and error rates are measured. It is shown/observed that the novel/proposed method has a better practical convergence rate when compared to the other models. Regarding the amount of noise, it is proven that there is a very good approximation of the matrix inverse even in the presence of noise.

[1]  James Hardy Wilkinson,et al.  Error Analysis of Direct Methods of Matrix Inversion , 1961, JACM.

[2]  Soo-Young Lee,et al.  An Optimization Network for Matrix Inversion , 1987, NIPS.

[3]  Lin-Bao Yang,et al.  Cellular neural networks: theory , 1988 .

[4]  Leon O. Chua,et al.  Neural networks for nonlinear programming , 1988 .

[5]  Abdesselam Bouzerdoum,et al.  Neural network for quadratic optimization with bound constraints , 1993, IEEE Trans. Neural Networks.

[6]  Tamás Roska,et al.  The CNN universal machine: an analogic array computer , 1993 .

[7]  Antonio Bicchi,et al.  Optimal design of multivariate sensors , 1994 .

[8]  Jun Wang,et al.  Recurrent Neural Networks for Computing Pseudoinverses of Rank-Deficient Matrices , 1997, SIAM J. Sci. Comput..

[9]  James T. Dobbins,et al.  Applications of matrix inversion tomosynthesis , 2000, Medical Imaging.

[10]  Benesty,et al.  Adaptive eigenvalue decomposition algorithm for passive acoustic source localization , 2000, The Journal of the Acoustical Society of America.

[11]  Lei Xu,et al.  A PCA approach for fast retrieval of structural patterns in attributed graphs , 2001, IEEE Trans. Syst. Man Cybern. Part B.

[12]  Klaus Mainzer.  CNN and the evolution of complex information systems in nature and technology , 2002, Proceedings of the 2002 7th IEEE International Workshop on Cellular Neural Networks and Their Applications.

[13]  J. J. Hopfield,et al.  “Neural” computation of decisions in optimization problems , 1985, Biological Cybernetics.

[14]  Fuye Feng,et al.  A Recurrent Neural Network for Extreme Eigenvalue Problem , 2005, ICIC.

[15]  Shuzhi Sam Ge,et al.  Design and analysis of a general recurrent neural network model for time-varying matrix inversion , 2005, IEEE Transactions on Neural Networks.

[16]  Yunong Zhang.  Towards Piecewise-Linear Primal Neural Networks for Optimization and Redundant Robotics , 2006, 2006 IEEE International Conference on Networking, Sensing and Control.

[17]  Johan Eilert,et al.  Efficient Complex Matrix Inversion for MIMO Software Defined Radio , 2007, 2007 IEEE International Symposium on Circuits and Systems.

[18]  Yiguang Liu,et al.  A recurrent neural network computing the largest imaginary or real part of eigenvalues of real matrices , 2007, Comput. Math. Appl..

[19]  W. Koshak,et al.  General Matrix Inversion Technique for the Calibration of Electric Field Sensor Arrays on Aircraft Platforms , 2007 .

[20]  Ryan Kastner,et al.  An FPGA Design Space Exploration Tool for Matrix Inversion Architectures , 2008, 2008 Symposium on Application Specific Processors.

[21]  Dierk Schröder,et al.  Comments on "Backpropagation Algorithms for a Broad Class of Dynamic Networks" , 2009, IEEE Trans. Neural Networks.

[22]  David B. Dunson,et al.  Multitask Compressive Sensing , 2009, IEEE Transactions on Signal Processing.

[23]  Ke Chen,et al.  Performance Analysis of Gradient Neural Network Exploited for Online Time-Varying Matrix Inversion , 2009, IEEE Transactions on Automatic Control.

[24]  Binghuang Cai,et al.  From Zhang Neural Network to Newton Iteration for Matrix Inversion , 2009, IEEE Transactions on Circuits and Systems I: Regular Papers.

[25]  Jianping Li,et al.  Another neural network based approach for computing eigenvalues and eigenvectors of real skew-symmetric matrices , 2010, Comput. Math. Appl..

[26]  K. Kyamakya,et al.  Potential contribution of CNN-based solving of stiff ODEs & PDEs to enabling real-time Computational Engineering , 2010, 2010 12th International Workshop on Cellular Nanoscale Networks and their Applications (CNNA 2010).

[27]  Chenfu Yi,et al.  Improved gradient-based neural networks for online solution of Lyapunov matrix equation , 2011, Inf. Process. Lett..

[28]  Lei Ma,et al.  QR Decomposition-Based Matrix Inversion for High Performance Embedded MIMO Receivers , 2011, IEEE Transactions on Signal Processing.

[29]  Yunong Zhang,et al.  Complex-valued Zhang neural network for online complex-valued time-varying matrix inversion , 2011, Appl. Math. Comput..

[30]  Carlos H. Llanos,et al.  A suitable FPGA implementation of floating-point matrix inversion based on Gauss-Jordan elimination , 2011, 2011 VII Southern Conference on Programmable Logic (SPL).

[31]  Kyandoghere Kyamakya,et al.  CNN based high performance computing for real time image processing on GPU , 2011, Proceedings of the Joint INDS'11 & ISTET'11.

[32]  Dongsheng Guo,et al.  Zhang neural network, Getz-Marsden dynamic system, and discrete-time algorithms for time-varying matrix inversion with application to robots' kinematic control , 2012, Neurocomputing.

[33]  Chenfu Yi,et al.  Improved neural solution for the Lyapunov matrix equation based on gradient search , 2013, Inf. Process. Lett..

[34]  Bin Gu,et al.  Feasibility and Finite Convergence Analysis for Accurate On-Line $\nu$-Support Vector Machine , 2013, IEEE Transactions on Neural Networks and Learning Systems.

[35]  Joseph R. Cavallaro,et al.  Approximate matrix inversion for high-throughput data detection in the large-scale MIMO uplink , 2013, 2013 IEEE International Symposium on Circuits and Systems (ISCAS2013).

[36]  Harry Leib,et al.  Sphere Decoding for MIMO Systems with Newton Iterative Matrix Inversion , 2013, IEEE Communications Letters.

[37]  James T Dobbins,et al.  The effect of averaging adjacent planes for artifact reduction in matrix inversion tomosynthesis. , 2013, Medical physics.

[38]  Ke Chen.  Recurrent implicit dynamics for online matrix inversion , 2013, Appl. Math. Comput..

[39]  Predrag S. Stanimirovic,et al.  Recurrent Neural Network for Computing the Drazin Inverse , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[40]  Yong Wang,et al.  Locating Multiple Optimal Solutions of Nonlinear Equation Systems Based on Multiobjective Optimization , 2015, IEEE Transactions on Evolutionary Computation.

[41]  Kyandoghere Kyamakya,et al.  A Universal Concept Based on Cellular Neural Networks for Ultrafast and Flexible Solving of Differential Equations , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[42]  Pei-Yun Tsai,et al.  Matrix-Inversion-Free Compressed Sensing With Variable Orthogonal Multi-Matching Pursuit Based on Prior Information for ECG Signals , 2016, IEEE Transactions on Biomedical Circuits and Systems.

[43]  Sundeep Prabhakar Chepuri,et al.  Sensor Selection for Estimation with Correlated Measurement Noise , 2015, IEEE Transactions on Signal Processing.

[44]  Dongsheng Guo,et al.  The application of Li-function activated RNN to acceleration-level robots' kinematic control via time-varying matrix inversion , 2016, 2016 Chinese Control and Decision Conference (CCDC).

[45]  Flora Amato,et al.  Recommendation in Social Media Networks , 2017, 2017 IEEE Third International Conference on Multimedia Big Data (BigMM).

[46]  Liang Xiao,et al.  Wider and Deeper, Cheaper and Faster: Tensorized LSTMs for Sequence Learning , 2017, NIPS.

[47]  Flora Amato,et al.  Multimedia summarization using social media content , 2017, Multimedia Tools and Applications.

[48]  Franz J. Dutz,et al.  Iterative matrix algorithm for high precision temperature and force decoupling in multi-parameter FBG sensing. , 2018, Optics express.

[49]  F. Smarandache,et al.  Exploring the Extension of Natural Operations on Intervals, Matrices and Complex Numbers , 2020 .