Stochastic validation for automated tuning of neural network's hyper-parameters

Abstract In this paper we describe a new method for automated tuning of hyper-parameters of supervised learning systems. It uses memory-based learning principles, follows certain ideas of experimental design and employs an alternative approach to resampling called stochastic validation. The described method allows not only for an efficient search through a decision space, but also for a corcurrent validation of the learning algorithm performance on a given data. Potential usefulness of the proposed approach is illustrated with the Fuzzy-ARTMAP neural network application to learning a qualitative positioning of an indoor mobile robot equipped with ultrasonic range sensors. Automatically selected neural network setpoints reach a comparable performance to those achieved by human experts in two-dimensional parameter optimization cases. Migration of the proposed method to higher-order optimization domains bears a big promise and requires further research.

[1]  Artur Dubrawski,et al.  Artificial neural network for mobile robot topological localization , 1995, Robotics Auton. Syst..

[2]  George E. P. Box,et al.  Empirical Model‐Building and Response Surfaces , 1988 .

[3]  Patrick Reignier,et al.  Learning to categorize perceptual space of a mobile robot using fuzzy-ART neural network , 1994, Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS'94).

[4]  Derek J. Pike,et al.  Empirical Model‐building and Response Surfaces. , 1988 .

[5]  Vasant Honavar,et al.  Evolutionary Design of Neural Architectures -- A Preliminary Taxonomy and Guide to Literature , 1995 .

[6]  Xin Yao,et al.  A review of evolutionary artificial neural networks , 1993, Int. J. Intell. Syst..

[7]  Stephen Grossberg,et al.  Fuzzy ARTMAP: A neural network architecture for incremental supervised learning of analog multidimensional maps , 1992, IEEE Trans. Neural Networks.

[8]  James L. Crowley,et al.  Self-supervised neural system for reactive navigation , 1994, Proceedings of the 1994 IEEE International Conference on Robotics and Automation.

[9]  A. Dubrawski.  Memory-based Stochastic Optimization for Automated Tuning of Neural Network's High Level Parameters ? , 1996 .

[10]  Andrew W. Moore,et al.  Multiresolution Instance-Based Learning , 1995, IJCAI.

[11]  Stephen Grossberg,et al.  Fuzzy ART: Fast stable learning and categorization of analog patterns by an adaptive resonance system , 1991, Neural Networks.

[12]  Christopher G. Atkeson,et al.  Memory-Based Learning Control , 1991, 1991 American Control Conference.

[13]  Sholom M. Weiss,et al.  Computer Systems That Learn , 1990 .

[14]  Andrew W. Moore,et al.  Memory-based Stochastic Optimization , 1995, NIPS.

[15]  Leslie Pack Kaelbling,et al.  Learning in embedded systems , 1993 .