论文信息 - Bidirectional clustering of weights for neural networks with common weights

Bidirectional clustering of weights for neural networks with common weights

This paper proposes a method which succinctly structures neural networks having a few thousand weights. Here structuring means weight sharing where weights in a network are divided into clusters and weights within a cluster have the same value. We newly introduce a weight sharing technique called bidirectional clustering of weights (BCW), together with second-order optimal criteria for both cluster merging and splitting. Our experiments using two artificial data sets showed that the BCW method works well to find a succinct network structure from a neural network having about 2000 weights in both regression and classification problems. © 2007 Wiley Periodicals, Inc. Syst Comp Jpn, 38(10): 46–57, 2007; Published online in Wiley InterScience (www.interscience.wiley.com). DOI 10.1002/scj.20535

Kazumi Saito | Ryohei Nakano | R. Nakano | Kazumi Saito

[1] Richard S. Sutton,et al. Learning Polynomial Functions by Feature Construction , 1991, ML.

[2] Jude W. Shavlik,et al. Extracting refined rules from knowledge-based neural networks , 2004, Machine Learning.

[3] Yann LeCun,et al. Optimal Brain Damage , 1989, NIPS.

[4] Kazumi Saito,et al. Discovering Polynomials to Fit Multivariate Data Having Numeric and Nominal Variables , 2002, Progress in Discovery Science.

[5] Richard O. Duda,et al. Pattern classification and scene analysis , 1974, A Wiley-Interscience publication.

[6] Lawrence D. Jackel,et al. Backpropagation Applied to Handwritten Zip Code Recognition , 1989, Neural Computation.

[7] Gregory J. Wolff,et al. Optimal Brain Surgeon and general network pruning , 1993, IEEE International Conference on Neural Networks.

[8] Geoffrey E. Hinton,et al. Simplifying Neural Networks by Soft Weight-Sharing , 1992, Neural Computation.

[9] Simon Haykin,et al. Neural Networks: A Comprehensive Foundation , 1998 .

[10] Heekuck Oh,et al. Neural Networks for Pattern Recognition , 1993, Adv. Comput..

[11] Geoffrey E. Hinton,et al. SMEM Algorithm for Mixture Models , 1998, Neural Computation.

[12] Kunihiko Fukushima,et al. Neocognitron: A hierarchical neural network capable of visual pattern recognition , 1988, Neural Networks.