Modularity in neural computing

This paper considers neural computing models for information processing in terms of collections of subnetwork modules. Two approaches to generating such networks are studied. The first approach includes networks with functionally independent subnetworks, where each subnetwork is designed to have specific functions, communication, and adaptation characteristics. The second approach is based on algorithms that can actually generate network and subnetwork topologies, connections, and weights to satisfy specific constraints. Associated algorithms to attain these goals include evolutionary computation and self-organizing maps. We argue that this modular approach to neural computing is more in line with the neurophysiology of the vertebrate cerebral cortex, particularly with respect to sensation and perception. We also argue that this approach has the potential to aid in solutions to large-scale network computational problems - an identified weakness of simply defined artificial neural networks.

[1]  Erhardt Barth,et al.  Image Encoding, Labeling, and Reconstruction from Differential Geometry , 1993, CVGIP Graph. Model. Image Process..

[2]  Johannes R. Sveinsson,et al.  Parallel consensual neural networks , 1997, IEEE Trans. Neural Networks.

[3]  D. Hubel.  Eye, brain, and vision , 1988 .

[4]  Ling Guan,et al.  Adaptive regularization in image restoration using evolutionary programming , 1998, 1998 IEEE International Conference on Evolutionary Computation Proceedings. IEEE World Congress on Computational Intelligence (Cat. No.98TH8360).

[5]  Jouko Lampinen,et al.  Distortion tolerant pattern recognition based on self-organizing feature extraction , 1995, IEEE Trans. Neural Networks.

[6]  Ling Guan,et al.  A network of networks processing model for image regularization , 1997, IEEE Trans. Neural Networks.

[7]  T. Sejnowski,et al.  Perspectives on cognitive neuroscience. , 1988, Science.

[8]  Paulo J. G. Lisboa,et al.  Translation, rotation, and scale invariant pattern recognition by high-order neural networks and moment classifiers , 1992, IEEE Trans. Neural Networks.

[9]  Paul Juell,et al.  A hierarchical neural network for human face detection , 1996, Pattern Recognit..

[10]  H B Barlow,et al.  Single units and sensation: a neuron doctrine for perceptual psychology? , 1972, Perception.

[11]  Xin Yao,et al.  A new evolutionary system for evolving artificial neural networks , 1997, IEEE Trans. Neural Networks.

[12]  Ling Guan,et al.  A statistics-based weight assignment in a Hopfield neural network for adaptive image restoration , 1998, 1998 IEEE International Joint Conference on Neural Networks Proceedings. IEEE World Congress on Computational Intelligence (Cat. No.98CH36227).

[13]  C. Sherrington.  Integrative Action of the Nervous System , 1907 .

[14]  H. Nothdurft,et al.  Texture discrimination: Representation of orientation and luminance differences in cells of the cat striate cortex , 1985, Vision Research.

[15]  Teuvo Kohonen,et al.  Self-organization and associative memory: 3rd edition , 1989 .

[16]  Jenq-Neng Hwang,et al.  Neural networks for intelligent multimedia processing , 1998 .

[17]  Lars Kai Hansen,et al.  Massive Weight Sharing: A Cure For Extremely Ill-Posed Problems , 1994 .

[18]  Sun-Yuan Kung,et al.  Decision-based neural networks with signal/image classification applications , 1995, IEEE Trans. Neural Networks.

[19]  S. W. Kuffler.  Neurons in the retina; organization, inhibition and excitation problems. , 1952, Cold Spring Harbor symposia on quantitative biology.

[20]  Stefanos D. Kollias,et al.  Invariant image classification using triple-correlation-based neural networks , 1994, IEEE Trans. Neural Networks.

[21]  J. Szentágothai.  The Ferrier Lecture, 1977 The neuron network of the cerebral cortex: a functional interpretation , 1978, Proceedings of the Royal Society of London. Series B. Biological Sciences.

[22]  K. S. Narendra,et al.  Neural networks for control theory and practice , 1996, Proc. IEEE.

[23]  Kagan Tumer,et al.  Analysis of decision boundaries in linearly combined neural classifiers , 1996, Pattern Recognit..

[24]  Sherif Hashem,et al.  Optimal Linear Combinations of Neural Networks , 1997, Neural Networks.

[25]  Leif H. Finkel,et al.  A multistage neural network for color constancy and color induction , 1995, IEEE Trans. Neural Networks.

[26]  Michael Stiber,et al.  Synaptic coding of spike trains , 1998 .

[27]  Martti Juhola,et al.  Simulation of aphasic naming errors in finnish language with neural networks , 1995, Neural Networks.

[28]  Siwei Lu,et al.  Hierarchical artificial neural networks for edge enhancement , 1993, Pattern Recognit..

[29]  Tony R. Martinez,et al.  Digital Neural Networks , 1988, Proceedings of the 1988 IEEE International Conference on Systems, Man, and Cybernetics.

[30]  Alberto Del Bimbo,et al.  3D object classification using multi-object Kohonen networks , 1996, Pattern Recognit..

[31]  Bruno G. Breitmeyer,et al.  Visual masking : an integrative approach , 1984 .

[32]  Nicholas J. Redding,et al.  Constructive higher-order network that is polynomial time , 1993, Neural Networks.

[33]  Amanda J. C. Sharkey,et al.  On Combining Artificial Neural Nets , 1996, Connect. Sci..

[34]  Sun-Yuan Kung,et al.  A neural network approach to face/palm recognition , 1995, Proceedings of 1995 IEEE Workshop on Neural Networks for Signal Processing.

[35]  Lars Ersland,et al.  Phantom limb imaginary fingertapping causes primary motor cortex activation: an fMRI study , 1996, Neuroreport.

[36]  Jon P. Davis,et al.  Pattern Recognition Properties of Various Feature Spaces for Higher Order Neural Networks , 1993, IEEE Trans. Pattern Anal. Mach. Intell..

[37]  Nasser M. Nasrabadi,et al.  Object recognition using multilayer Hopfield neural network , 1997, IEEE Trans. Image Process..

[38]  Steve R. Waterhouse,et al.  Constructive Algorithms for Hierarchical Mixtures of Experts , 1995, NIPS.

[39]  Keiji Tanaka,et al.  Coding visual images of objects in the inferotemporal cortex of the macaque monkey. , 1991, Journal of neurophysiology.

[40]  Michael Stiber,et al.  Synaptic coding of periodically modulated spike trains , 1993, IEEE International Conference on Neural Networks.

[41]  Erkki Oja,et al.  Engineering applications of the self-organizing map , 1996, Proc. IEEE.

[42]  M. Koppen,et al.  A neural network that uses evolutionary learning , 1997, Proceedings of 1997 IEEE International Conference on Evolutionary Computation (ICEC '97).

[43]  R. Browse,et al.  Asymmetries in visual texture discrimination. , 1989, Spatial vision.

[44]  Steve R. Waterhouse,et al.  Non-linear Prediction of Acoustic Vectors Using Hierarchical Mixtures of Experts , 1994, NIPS.

[45]  Nirmal K. Bose,et al.  Neural network design using Voronoi diagrams , 1993, IEEE Trans. Neural Networks.

[46]  Huan Liu,et al.  Self-generating neural networks , 1992, [Proceedings 1992] IJCNN International Joint Conference on Neural Networks.

[47]  Gail A. Carpenter,et al.  ART-EMAP: A neural network architecture for object recognition by evidence accumulation , 1995, IEEE Trans. Neural Networks.

[48]  R. W. Rodieck.  The vertebrate retina : principles of structure and function , 1973 .

[49]  Athanassios Siapas,et al.  Criticality and Parallelism in Combinatorial Optimization , 1996, Science.

[50]  Pekka Orponen,et al.  On the Computational Complexity of Analyzing Hopfield Nets , 1989, Complex Syst..

[51]  James A. Anderson,et al.  The BSB model: a simple nonlinear autoassociative neural network , 1993 .

[52]  S. Zeki,et al.  Functional specialization and binocular interaction in the visual areas of rhesus monkey prestriate cortex , 1979, Proceedings of the Royal Society of London. Series B. Biological Sciences.

[53]  S. Zeki.  Functional specialisation in the visual cortex of the rhesus monkey , 1978, Nature.

[54]  N. K. Bose,et al.  Neural Network Fundamentals with Graphs, Algorithms and Applications , 1995 .

[55]  A. Lundervold,et al.  Functional magnetic resonance imaging of primary visual processing using a 1.0 Tesla scanner. , 1995, The International journal of neuroscience.

[56]  John H. R. Maunsell,et al.  Hierarchical organization and functional streams in the visual cortex , 1983, Trends in Neurosciences.

[57]  Nirmal K. Bose,et al.  Vector quantization of neural networks , 1998, IEEE Trans. Neural Networks.

[58]  K Tanaka,et al.  Neuronal mechanisms of object recognition. , 1993, Science.

[59]  Sung H. Yoon,et al.  A robust boundary-based object recognition in occlusion environment by hybrid Hopfield neural networks , 1996, Pattern Recognit..

[60]  Rama Chellappa,et al.  Learning Texture Discrimination Rules in a Multiresolution System , 1994, IEEE Trans. Pattern Anal. Mach. Intell..

[61]  Mark E. Oxley,et al.  Neural networks for automatic target recognition , 1995, Neural Networks.

[62]  Leonid I. Perlovsky,et al.  Model-based neural network for target detection in SAR images , 1997, IEEE Trans. Image Process..

[63]  M. M. Moya,et al.  Cueing, feature discovery, and one-class learning for synthetic aperture radar automatic target recognition , 1995, Neural Networks.

[64]  Judith E. Dayhoff,et al.  Trajectory production with the adaptive time-delay neural network , 1995, Neural Networks.

[65]  C. Blakemore,et al.  Innate and environmental factors in the development of the kitten's visual cortex. , 1975, The Journal of physiology.

[66]  Simon Haykin,et al.  Neural networks expand SP's horizons , 1996, IEEE Signal Process. Mag..

[67]  Huan Liu,et al.  Learning a Neural Tree , 1992 .

[68]  Seong-Whan Lee,et al.  Multilayer cluster neural network for totally unconstrained handwritten numeral recognition , 1995, Neural Networks.

[69]  Kunihiko Fukushima,et al.  Neocognitron: A hierarchical neural network capable of visual pattern recognition , 1988, Neural Networks.

[70]  T. Caelli.  Three processing characteristics of visual texture segmentation. , 1985, Spatial vision.

[71]  G. W. PETTIGREW,et al.  The Amino-acid Sequence of Cytochrome c from Euglena gracilis , 1973, Nature.

[72]  Terry Caelli,et al.  Model-based neural networks , 1993, Neural Networks.

[73]  Santanu Chaudhury,et al.  Signature verification using multiple neural classifiers , 1997, Pattern Recognit..

[74]  J. Daugman.  Uncertainty relation for resolution in space, spatial frequency, and orientation optimized by two-dimensional visual cortical filters. , 1985, Journal of the Optical Society of America. A, Optics and image science.

[75]  Chorkin Chan,et al.  Isolated Word Recognition by Neural Network Models with Cross-Correlation Coefficients for Speech Dynamics , 1993, IEEE Trans. Pattern Anal. Mach. Intell..

[76]  Anders Krogh,et al.  A Simple Weight Decay Can Improve Generalization , 1991, NIPS.

[77]  Pat Langley,et al.  Constraints on Tree Structure in Concept Formation , 1991, IJCAI.

[78]  W. J. Freeman.  Why neural networks don't yet fly: inquiry into the neurodynamics of biological intelligence , 1988, IEEE 1988 International Conference on Neural Networks.

[79]  Sankar K. Pal,et al.  PsyCOP-a psychologically motivated connectionist system for object perception , 1995, IEEE Trans. Neural Networks.

[80]  Michael I. Jordan,et al.  Hierarchies of Adaptive Experts , 1991, NIPS.

[81]  Sung-Bae Cho,et al.  Neural-network classifiers for recognizing totally unconstrained handwritten numerals , 1997, IEEE Trans. Neural Networks.

[82]  James A. Anderson,et al.  Radar signal categorization using a neural network , 1990, Proc. IEEE.

[83]  J. Pettigrew,et al.  Binocular neurons which signal change of disparity in area 18 of cat visual cortex. , 1973, Nature: New biology.

[84]  Michael I. Jordan,et al.  A Competitive Modular Connectionist Architecture , 1990, NIPS.

[85]  Robert Laganière,et al.  Gradual perception of structure from motion: a neural approach , 1995, IEEE Trans. Neural Networks.

[86]  S. Y. King.  Parallel architectures for artificial neural nets , 1988, [1988] Proceedings. International Conference on Systolic Arrays.

[87]  Robert I. Damper,et al.  On neural-network implementations of k-nearest neighbor pattern classifiers , 1997 .

[88]  Robert A. Jacobs,et al.  A Bayesian Approach to Model Selection in Hierarchical Mixtures-of-Experts Architectures , 1997, Neural Networks.

[89]  I. W. Habib.  Applications of neurocomputing in traffic management of ATM networks , 1996 .

[90]  Simon Haykin,et al.  Neural network approaches to image compression , 1995, Proc. IEEE.

[91]  Sun-Yuan Kung,et al.  A feature tracking algorithm using neighborhood relaxation with multi-candidate pre-screening , 1996, Proceedings of 3rd IEEE International Conference on Image Processing.

[92]  Athanasios Kehagias,et al.  Modular neural networks for MAP classification of time series and the partition algorithm , 1996, IEEE Trans. Neural Networks.

[93]  Wen-Gou Lin,et al.  A New Neural Model for Invariant Pattern Recognition , 1996, Neural Networks.

[94]  J. P. Sutton,et al.  fMRI evidence of nested networks associated with motor tasks , 1996, NeuroImage.

[95]  Ling Guan,et al.  Adaptive scaled mean square error filtering by neural networks , 1996, J. Electronic Imaging.

[96]  Anil K. Jain,et al.  A self-organizing network for hyperellipsoidal clustering (HEC) , 1994, Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN'94).

[97]  Shuenn-Shyang Wang,et al.  A new self-organizing neural model for invariant pattern recognition , 1996, Pattern Recognit..

[98]  Hiroshi Tsujino,et al.  A Cortical-type Modular Neural Network for Hypothetical Reasoning , 1997, Neural Networks.

[99]  J. P. Sutton,et al.  Neural models I: A hierarchical model of neocortical synaptic organization , 1988 .

[100]  G. Globus.  Toward a Noncomputational Cognitive Neuroscience , 1992, Journal of Cognitive Neuroscience.

[101]  Aggelos K. Katsaggelos,et al.  Image restoration using a modified Hopfield network , 1992, IEEE Trans. Image Process..

[102]  Simon Haykin,et al.  Neural Networks: A Comprehensive Foundation , 1998 .

[103]  William Robertson,et al.  Mapping hierarchical neural networks to VLSI hardware , 1995, Neural Networks.

[104]  Kunihiko Fukushima,et al.  Speed invariant speech recognition using variable velocity delay lines , 1995, Neural Networks.

[105]  Kishan G. Mehrotra,et al.  Efficient classification for multiclass problems using modular neural networks , 1995, IEEE Trans. Neural Networks.

[106]  Heggere S. Ranganath,et al.  Self partitioning neural networks for target recognition , 1995, Neural Networks.

[107]  Marimuthu Palaniswami,et al.  Range image segmentation by dynamic neural network architecture , 1996, Pattern Recognit..

[108]  Ling Guan.  Image restoration by a neural network with hierarchical cluster architecture , 1994, J. Electronic Imaging.

[109]  Terry Caelli,et al.  An adaptive computational model for texture segmentation , 1988, IEEE Trans. Syst. Man Cybern..

[110]  Sun-Yuan Kung,et al.  Face recognition/detection by probabilistic decision-based neural network , 1997, IEEE Trans. Neural Networks.

[111]  Xin Yao,et al.  Evolving modular neural networks which generalise well , 1997, Proceedings of 1997 IEEE International Conference on Evolutionary Computation (ICEC '97).

[112]  D. Fogel,et al.  Discovering patterns in spatial data using evolutionary programming , 1996 .

[113]  Suchendra M. Bhandarkar,et al.  A multilayer self-organizing feature map for range image segmentation , 1995, Neural Networks.

[114]  S. Kung,et al.  A probabilistic DBNN with applications to sensor fusion and object recognition , 1995, Proceedings of 1995 IEEE Workshop on Neural Networks for Signal Processing.

[115]  G. Sperling,et al.  Second-order motion perception: space/time separable mechanisms , 1989, [1989] Proceedings. Workshop on Visual Motion.

[116]  Erol Gelenbe,et al.  Low bit-rate video compression with neural networks and temporal subsampling , 1996, Proc. IEEE.

[117]  Alexander V. Goltsev,et al.  An assembly neural network for texture segmentation , 1996, Neural Networks.

[118]  M. Alexander,et al.  Principles of Neural Science , 1981 .

[119]  Terry Caelli,et al.  Shift, rotation and scale invariant signatures for two-dimensional contours, in a neural network architecture , 1997 .

[120]  Anil K. Jain,et al.  A self-organizing network for hyperellipsoidal clustering (HEC) , 1996, IEEE Trans. Neural Networks.

[121]  J. Pettigrew.  Binocular Neurones which signal Change of Disparity in Area 18 of Cat Visual Cortex , 1973 .

[122]  Ling Guan,et al.  Neural vision system and applications in image processing and analysis , 1998, Proceedings of the 1998 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP '98 (Cat. No.98CH36181).

[123]  Thomas Bäck,et al.  Evolutionary computation: Toward a new philosophy of machine intelligence , 1997, Complex..

[124]  Xinhua Zhuang,et al.  Robust segmentation of noisy images using a neural network model , 1992, Image Vis. Comput..

[125]  Simon Haykin,et al.  A modular neural network for enhancement of cross-polar radar targets , 1996, Neural Networks.

[126]  Terrance L. Huntsberger,et al.  Biologically motivated cross-modality sensory fusion system for automatic target recognition , 1995, Neural Networks.

[127]  Sung-Bae Cho,et al.  Combining modular neural networks developed by evolutionary algorithm , 1997, Proceedings of 1997 IEEE International Conference on Evolutionary Computation (ICEC '97).

[128]  David B. Fogel,et al.  Evolutionary Computation: Towards a New Philosophy of Machine Intelligence , 1995 .

[129]  K Fukushima,et al.  Handwritten alphanumeric character recognition by the neocognitron , 1991, IEEE Trans. Neural Networks.

[130]  Majid Ahmadi,et al.  A hierarchical neural network architecture for handwritten numeral recognition , 1997, Pattern Recognit..

[131]  R. Lippmann,et al.  An introduction to computing with neural nets , 1987, IEEE ASSP Magazine.

[132]  Sebastian Thrun,et al.  The MONK''s Problems-A Performance Comparison of Different Learning Algorithms, CMU-CS-91-197, Sch , 1991 .

[133]  Alan F. Murray,et al.  Asynchronous VLSI neural networks using pulse-stream arithmetic , 1988 .

[134]  Johan A. K. Suykens,et al.  NLq Theory: A Neural Control Framework with Global Asymptotic Stability Criteria , 1997, Neural Networks.

[135]  L. Spillmann,et al.  Visual Perception: The Neurophysiological Foundations , 1989 .

[136]  David J. Spiegelhalter,et al.  Probabilistic Reasoning in Predictive Expert Systems , 1985, UAI.

[137]  Alberto Del Bimbo,et al.  Block-structured recurrent neural networks , 1995, Neural Networks.

[138]  Ennio Mingolla,et al.  Neural dynamics of perceptual grouping: Textures, boundaries, and emergent segmentations , 1985 .

[139]  Athanasios Kehagias,et al.  Predictive Modular Neural Networks for Time Series Classification , 1997, Neural Networks.