Learning Binary Perceptrons Perfectly Efficiently

The majority rule algorithm for learning binary weights for a perceptron is analysed under the uniform distribution on inputs. It is shown that even though the algorithm is demonstrably inconsistent on random samples for very small sample sizes, it nevertheless exhibits a curious and abrupt asymptotic transition to consistency at moderate sample sizes. Particular consequences are that the algorithm PAC-learns majority functions in linear time from small samples and that, while the general variant of binary integer programming embodied here is NP-complete, almost all instances of the problem are tractable given a sufficient number of inequalities to be satisfies

[1]  Sompolinsky,et al.  Learning from examples in large neural networks. , 1990, Physical review letters.

[2]  Györgyi,et al.  First-order transition to perfect generalization in a neural network with binary synapses. , 1990, Physical review. A, Atomic, molecular, and optical physics.

[3]  Yuh-Dauh Lyuu,et al.  The Transition to Perfect Generalization in Perceptrons , 1991, Neural Computation.

[4]  William Feller,et al.  An Introduction to Probability Theory and Its Applications , 1967 .

[5]  Santosh S. Venkatesh,et al.  On the average tractability of binary integer programming and the curious transition to perfect generalization in learning majority functions , 1993, COLT '93.

[6]  Wolfgang Kinzel,et al.  Learning algorithm for a neural network with binary synapses , 1990 .

[7]  Feller William,et al.  An Introduction To Probability Theory And Its Applications , 1950 .

[8]  Santosh S. Venkatesh,et al.  On learning binary weights for majority functions , 1991, COLT '91.

[9]  Mario Marchand,et al.  On Learning Perceptrons with Binary Weights , 1993, Neural Computation.

[10]  Leslie G. Valiant,et al.  Computational limitations on learning from examples , 1988, JACM.

[11]  Yuh-Dauh Lyuu,et al.  Tight Bounds on Transition to Perfect Generalization in Perceptrons , 1992, Neural Computation.

[12]  Santosh S. Venkatesh,et al.  Directed Drift: A New Linear Threshold Algorithm for Learning Binary Weights On-Line , 1993, J. Comput. Syst. Sci..

[13]  Shao Chieh Fang.  Capacity and complexity in learning binary perceptrons , 1995 .

[14]  E. Gardner,et al.  Three unfinished works on the optimal storage capacity of networks , 1989 .

[15]  David S. Johnson,et al.  Computers and Intractability: A Guide to the Theory of NP-Completeness , 1978 .