Dimensionality Reduction in Automatic Knowledge Acquisition: A Simple Greedy Search Approach

Knowledge acquisition is the process of collecting domain knowledge, documenting the knowledge, and transforming it into a computerized representation. Due to the difficulties involved in eliciting knowledge from human experts, knowledge acquisition was identified as a bottleneck in the development of knowledge-based system. Over the past decades, a number of automatic knowledge acquisition techniques have been developed. However, the performance of these techniques suffers from the so called curse of dimensionality, i.e., difficulties arise when many irrelevant (or redundant) parameters exist. This paper presents a heuristic approach based on statistics and greedy search for dimensionality reduction to facilitate automatic knowledge acquisition. The approach deals with classification problems. Specifically, Chi-square statistics are used to rank the importance of individual parameters. Then, a backward search procedure is employed to eliminate parameters (less important parameters first) that do not contribute to class separability. The algorithm is very efficient and was found to be effective when applied to a variety of problems with different characteristics.

[1]  George S. Sebestyen,et al.  Decision-making processes in pattern recognition , 1962 .

[2]  George S Sebestyen,et al.  Decision-making processes in pattern recognition (ACM monograph series) , 1962 .

[3]  Aiko M. Hormann,et al.  Programs for Machine Learning. Part I , 1962, Inf. Control..

[4]  Philip M. Lewis,et al.  The characteristic selection problem in recognition systems , 1962, IRE Trans. Inf. Theory.

[5]  Erich L. Lehmann,et al.  Basic Concepts of Probability and Statistics , 1965 .

[6]  Keinosuke Fukunaga,et al.  A Branch and Bound Algorithm for Feature Subset Selection , 1977, IEEE Transactions on Computers.

[7]  Edward A. Feigenbaum,et al.  The Art of Artificial Intelligence: Themes and Case Studies of Knowledge Engineering , 1977, IJCAI.

[8]  J. Kittler,et al.  Feature Set Search Alborithms , 1978 .

[9]  Raymond H. Myers,et al.  Probability and Statistics for Engineers and Scientists. , 1973 .

[10]  Frederick Hayes-Roth,et al.  Building expert systems , 1983, Advanced book program.

[11]  L. N. Kanal,et al.  Handbook of Statistics, Vol. 2. Classification, Pattern Recognition and Reduction of Dimensionality. , 1985 .

[12]  Jack Sklansky,et al.  A note on genetic algorithms for large-scale feature selection , 1989, Pattern Recognit. Lett..

[13]  Maureen Caudill,et al.  Using neural networks: hybrid expert networks , 1990 .

[14]  Yoshihiko Hamamoto,et al.  Evaluation of the branch and bound algorithm for feature selection , 1990, Pattern Recognit. Lett..

[15]  Sebastian Thrun,et al.  The MONK''s Problems-A Performance Comparison of Different Learning Algorithms, CMU-CS-91-197, Sch , 1991 .

[16]  David Martin,et al.  Book review: The Engineering of Knowledge-based Systems Theory and Practice by Avelino J. Gonzales and Douglas D. Dankel (Prentice Hall, 1993) , 1993, SGAR.

[17]  Jack Sklansky,et al.  On Automatic Feature Selection , 1988, Int. J. Pattern Recognit. Artif. Intell..

[18]  David Martin,et al.  Book review: The Engineering of Knowledge-based Systems Theory and Practice by Avelino J. Gonzales and Douglas D. Dankel (Prentice Hall, 1993) , 1993, SGAR.

[19]  Huan Liu,et al.  Chi2: feature selection and discretization of numeric attributes , 1995, Proceedings of 7th IEEE International Conference on Tools with Artificial Intelligence.

[20]  Ron Kohavi,et al.  Wrappers for Feature Subset Selection , 1997, Artif. Intell..

[21]  Anil K. Jain,et al.  Feature Selection: Evaluation, Application, and Small Sample Performance , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[22]  Pat Langley,et al.  Selection of Relevant Features and Examples in Machine Learning , 1997, Artif. Intell..

[23]  Hiroshi Motoda,et al.  Feature Selection for Knowledge Discovery and Data Mining , 1998, The Springer International Series in Engineering and Computer Science.

[24]  Huan Liu,et al.  Some issues on scalable feature selection , 1998 .

[25]  David W. Aha,et al.  A Review and Empirical Evaluation of Feature Weighting Methods for a Class of Lazy Learning Algorithms , 1997, Artificial Intelligence Review.

[26]  Boris G. Mirkin,et al.  Concept Learning and Feature Selection Based on Square-Error Clustering , 1999, Machine Learning.

[27]  Huan Liu,et al.  Incremental Feature Selection , 1998, Applied Intelligence.