Generalization ability of Boolean functions implemented in feedforward neural networks

Abstract We introduce a measure for the complexity of Boolean functions that is highly correlated with the generalization ability that could be obtained when the functions are implemented in feedforward neural networks. The measure, based on the calculation of the number of neighbour examples that differ in their output value, can be simply computed from the definition of the functions, independently of their implementation. Numerical simulations performed on different architectures show a good agreement between the estimated complexity and the generalization ability and training times obtained. The proposed measure could help as a useful tool for carrying a systematic study of the computational capabilities of network architectures by classifying in an easy and reliable way the Boolean functions. Also, based on the fact that the average generalization ability computed over the whole set of Boolean functions is 0.5, a very complex set of functions was found for which the generalization ability is lower than for random functions.

[1]  Simon Haykin,et al.  Neural Networks: A Comprehensive Foundation , 1998 .

[2]  Saburo Muroga,et al.  Threshold logic and its applications , 1971 .

[3]  David Haussler,et al.  What Size Net Gives Valid Generalization? , 1989, Neural Computation.

[4]  Jacob Feldman,et al.  Minimization of Boolean complexity in human concept learning , 2000, Nature.

[5]  Alberto L. Sangiovanni-Vincentelli,et al.  LSAT-an algorithm for the synthesis of two level threshold gate networks , 1991, 1991 IEEE International Conference on Computer-Aided Design Digest of Technical Papers.

[6]  John E. Hopcroft,et al.  Synthesis of Minimal Threshold Logic Networks , 1965, IEEE Trans. Electron. Comput..

[7]  Robert A. Legenstein,et al.  Foundations for a Circuit Complexity Theory of Sensory Processing , 2000, NIPS.

[8]  Ian Parberry,et al.  Circuit complexity and neural networks , 1994 .

[9]  J. Stephen Judd,et al.  Neural network design and the complexity of learning , 1990, Neural network modeling and connectionism.

[10]  Alberto L. Sangiovanni-Vincentelli,et al.  Learning Complex Boolean Functions: Algorithms and Applications , 1993, NIPS.

[11]  Noam Nisan,et al.  Constant depth circuits, Fourier transform, and learnability , 1993, JACM.

[12]  Peter Grassberger,et al.  Information and Complexity Measures in Dynamical Systems , 1991 .

[13]  Lech Jozwiak,et al.  New approach to learning noisy Boolean functions , 1998 .

[14]  Lech Jozwiak,et al.  An Effective and Efficient Method for Functional Decomposition of Boolean Functions Based on Information Relationships Measures. , 2000 .

[15]  Thomas Kailath,et al.  Depth-Size Tradeoffs for Neural Computation , 1991, IEEE Trans. Computers.

[16]  J. Rice.  Mathematical Statistics and Data Analysis , 1988 .

[17]  David H. Wolpert,et al.  The Mathematics of Generalization: The Proceedings of the SFI/CNLS Workshop on Formal Approaches to Supervised Learning , 1994 .

[18]  Wolfgang Kinzel,et al.  Antipredictable Sequences: Harder to Predict Than Random Sequences , 1998, Neural Computation.

[19]  Michael L. Dertouzos,et al.  Threshold Logic: A Synthesis Approach , 1965 .

[20]  Nathan Linial,et al.  The influence of variables on Boolean functions , 1988, [Proceedings 1988] 29th Annual Symposium on Foundations of Computer Science.

[21]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[22]  László Lovász,et al.  INFORMATION AND COMPLEXITY ( HOW TO MEASURE THEM ? ) , 2022 .

[23]  Chris Thornton,et al.  Parity: The Problem that Won't Go Away , 1996, Canadian Conference on AI.

[24]  Peter J. W. Rayner,et al.  Generalization and PAC learning: some new results for the class of generalized single-layer networks , 1995, IEEE Trans. Neural Networks.

[25]  Ingo Wegener,et al.  The complexity of Boolean functions , 1987 .

[26]  Robert O. Winder,et al.  Threshold logic , 1971, IEEE Spectrum.

[27]  Leonardo Franco,et al.  Generalization properties of modular networks: implementing the parity function , 2001, IEEE Trans. Neural Networks.

[28]  Ronald L. Rivest,et al.  Training a 3-node neural network is NP-complete , 1988, COLT '88.

[29]  José M. Bravo,et al.  Role of Function Complexity and Network Size in the Generalization Ability of Feedforward Networks , 2005, IWANN.

[30]  Mathukumalli Vidyasagar,et al.  A Theory of Learning and Generalization , 1997 .

[31]  Mathukumalli Vidyasagar,et al.  A Theory of Learning and Generalization: With Applications to Neural Networks and Control Systems , 1997 .

[32]  Mariusz Rawski,et al.  Functional decomposition with an efficient input support selection for sub-functions based on information relationship measures , 2001, J. Syst. Archit..

[33]  Jirí Síma,et al.  Back-propagation is not Efficient , 1996, Neural Networks.

[34]  Anders Krogh,et al.  Introduction to the theory of neural computation , 1994, The advanced book program.

[35]  S. Bhide,et al.  A real-time solution for the traveling salesman problem using a Boolean neural network , 1993, IEEE International Conference on Neural Networks.

[36]  Ravi B. Boppana,et al.  The Average Sensitivity of Bounded-Depth Circuits , 1997, Inf. Process. Lett..

[37]  Pekka Orponen,et al.  General-Purpose Computation with Neural Networks: A Survey of Complexity Theoretic Results , 2003, Neural Computation.

[38]  Sergio A. Cannas,et al.  Non glassy ground-state in a long-range antiferromagnetic frustrated model in the hypercubic cell , 2004 .

[39]  Pekka Orponen,et al.  Computational complexity of neural networks: a survey , 1994 .

[40]  C. Lee Giles,et al.  What Size Neural Network Gives Optimal Generalization? Convergence Properties of Backpropagation , 1998 .

[41]  Nick Chater.  Cognitive science: The logic of human learning , 2000, Nature.

[42]  Claude E. Shannon,et al.  The synthesis of two-terminal switching circuits , 1949, Bell Syst. Tech. J..

[43]  Vladimir Vapnik,et al.  Statistical learning theory , 1998 .

[44]  Chuanyi Ji,et al.  Capacity of Two-Layer Feedforward Neural Networks with Binary Weights , 1998, IEEE Trans. Inf. Theory.

[45]  Leonardo Franco,et al.  Generalization and Selection of Examples in Feedforward Neural Networks , 2000, Neural Computation.

[46]  Noam Nisan,et al.  Constant depth circuits, Fourier transform, and learnability , 1989, 30th Annual Symposium on Foundations of Computer Science.

[47]  Vinay Deolalikar,et al.  Mapping Boolean functions with neural networks having binary weights and zero thresholds , 2001, IEEE Trans. Neural Networks.

[48]  Rich Caruana,et al.  Overfitting in Neural Nets: Backpropagation, Conjugate Gradient, and Early Stopping , 2000, NIPS.

[49]  M. Anthony.  Boolean Functions and Artificial Neural Networks , 2003 .