Robustness Analysis on Dual Neural Network-based $k$ WTA With Input Noise

This paper studies the effects of uniform input noise and Gaussian input noise on the dual neural network-based <inline-formula> <tex-math notation="LaTeX">$k$ </tex-math></inline-formula>WTA (DNN-<inline-formula> <tex-math notation="LaTeX">$k$ </tex-math></inline-formula>WTA) model. We show that the state of the network (under either uniform input noise or Gaussian input noise) converges to one of the equilibrium points. We then derive a formula to check if the network produce correct outputs or not. Furthermore, for the uniformly distributed inputs, two lower bounds (one for each type of input noise) on the probability that the network produces the correct outputs are presented. Besides, when the minimum separation amongst inputs is given, we derive the condition for the network producing the correct outputs. Finally, experimental results are presented to verify our theoretical results. Since random drift in the comparators can be considered as input noise, our results can be applied to the random drift situation.

[1]  Andrew Chi-Sing Leung,et al.  A Fault-Tolerant Regularizer for RBF Networks , 2008, IEEE Transactions on Neural Networks.

[2]  Andrew Chi-Sing Leung,et al.  Analysis for a class of winner-take-all model , 1999, IEEE Trans. Neural Networks.

[3]  Andrew Chi-Sing Leung,et al.  Properties and Performance of Imperfect Dual Neural Network-Based $k$ WTA Networks , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[4]  Pavlo V. Tymoshchuk.  A fast analogue K-winners-take-all neural circuit , 2013, The 2013 International Joint Conference on Neural Networks (IJCNN).

[5]  Daniela Danciu,et al.  Gradient Like Behavior and High Gain Design of KWTA Neural Networks , 2009, IWANN.

[6]  Degang Chen,et al.  Analyses of Static and Dynamic Random Offset Voltages in Dynamic Comparators , 2009, IEEE Transactions on Circuits and Systems I: Regular Papers.

[7]  J. A. Nossek,et al.  Winner-take-all cellular neural networks , 1993 .

[8]  Justin K. Romberg,et al.  Convergence and Rate Analysis of Neural Networks for Sparse Approximation , 2011, IEEE Transactions on Neural Networks and Learning Systems.

[9]  R. Lippmann,et al.  An introduction to computing with neural nets , 1987, IEEE ASSP Magazine.

[10]  Lachlan L. H. Andrew.  Improving the robustness of winner-take-all cellular neural networks , 1996 .

[11]  Michalis E. Zervakis,et al.  KWTA networks and their applications , 1995, Multidimens. Syst. Signal Process..

[12]  Xiaodong Liu,et al.  Stability analysis for neural networks with time-varying delay , 2008, 2008 47th IEEE Conference on Decision and Control.

[13]  Pavlo V. Tymoshchuk.  A discrete-time dynamic K-winners-take-all neural circuit , 2009, Neurocomputing.

[14]  Jun Wang,et al.  Parametric Sensitivity and Scalability of k-Winners-Take-All Networks with a Single State Variable and Infinity-Gain Activation Functions , 2010, ISNN.

[15]  Yiran Chen,et al.  Memristor crossbar based hardware realization of BSB recall function , 2012, The 2012 International Joint Conference on Neural Networks (IJCNN).

[16]  John Lazzaro,et al.  Winner-Take-All Networks of O(N) Complexity , 1988, NIPS.

[17]  Fernando Pardo,et al.  Selective Change Driven Vision Sensor With Continuous-Time Logarithmic Photoreceptor and Winner-Take-All Circuit for Pixel Selection , 2015, IEEE Journal of Solid-State Circuits.

[18]  Shukai Duan,et al.  Memristor-Based Cellular Nonlinear/Neural Network: Design, Analysis, and Applications , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[19]  G. L. Dempsey,et al.  Circuit implementation of a peak detector neural network , 1993 .

[20]  Wayne P. Burleson,et al.  Rank-order filtering algorithms: A comparison of VLSI implementations , 1993, 1993 IEEE International Symposium on Circuits and Systems.

[21]  Xiaolin Hu,et al.  An Improved Dual Neural Network for Solving a Class of Quadratic Programming Problems and Its $k$-Winners-Take-All Application , 2008, IEEE Transactions on Neural Networks.

[22]  Yangming Li,et al.  A class of finite-time dual neural networks for solving quadratic programming problems and its k-winners-take-all application , 2013, Neural Networks.

[23]  Chi-Sing Leung,et al.  Analysis on the Convergence Time of Dual Neural Network-Based $k{\rm WTA}$ , 2012, IEEE Transactions on Neural Networks and Learning Systems.

[24]  F. Clarke.  Optimization And Nonsmooth Analysis , 1983 .

[25]  Michel Steyaert,et al.  Measurement of EMI induced input offset voltage of an operational amplifier , 2007 .

[26]  Jun Wang,et al.  Analysis and Design of a $k$ -Winners-Take-All Model With a Single State Variable and the Heaviside Step Activation Function , 2010, IEEE Transactions on Neural Networks.

[27]  Shuai Li,et al.  Selective Positive–Negative Feedback Produces the Winner-Take-All Competition in Recurrent Neural Networks , 2013, IEEE Transactions on Neural Networks and Learning Systems.

[28]  Corneliu A. Marinov,et al.  New Accurate and Flexible Design Procedure for a Stable KWTA Continuous Time Network , 2011, IEEE Transactions on Neural Networks.

[29]  Andrew Chi-Sing Leung,et al.  Effect of Input Noise and Output Node Stochastic on Wang's k WTA , 2013, IEEE Transactions on Neural Networks and Learning Systems.

[30]  Huaguang Zhang,et al.  A Comprehensive Review of Stability Analysis of Continuous-Time Recurrent Neural Networks , 2014, IEEE Transactions on Neural Networks and Learning Systems.

[31]  Peter Zörnig,et al.  4 Karush–Kuhn–Tucker conditions and duality , 2014 .

[32]  Lipo Wang,et al.  Noise injection into inputs in sparsely connected Hopfield and winner-take-all neural networks , 1997, IEEE Trans. Syst. Man Cybern. Part B.

[33]  Ting Wang,et al.  The design of low noise chopper operational amplifier with inverter , 2015, 2015 IEEE 16th International Conference on Communication Technology (ICCT).