Chapter 6 – Feature Generation I: Data Transformation and Dimensionality Reduction

Publisher Summary This chapter discusses the feature generation stage using data transformations and dimensionality reduction. Feature generation is important in any pattern recognition task. Given a set of measurements, the goal is to discover compact and informative representations of the obtained data. The basic approach followed in this chapter is to transform a given set of measurements to a new set of features. If the transform is suitably chosen, transform domain features can exhibit high information packing properties compared with the original input samples. The chapter reviews Karhunen–Loeve transform and the singular value decomposition as dimensionality reduction techniques. The independent component analysis, nonnegative matrix factorization, and nonlinear dimensionality reduction techniques are presented. Then the discrete Fourier transform, discrete cosine transform, discrete sine transform, Hadamard, and Haar transforms are defined. The rest of the chapter focuses on the discrete time wavelet transform. The chapter also shows that the Fourier transform is just one of the tools from a palette of possible transforms.

[1]  Oh-Wook Kwon,et al.  Phoneme recognition using ICA-based feature extraction and transformation , 2004, Signal Process..

[2]  Michael Unser,et al.  Texture classification and segmentation using wavelet frames , 1995, IEEE Trans. Image Process..

[3]  Dennis Gabor,et al.  Theory of communication , 1946 .

[4]  Edward H. Adelson,et al.  The Laplacian Pyramid as a Compact Image Code , 1983, IEEE Trans. Commun..

[5]  Anil K. Jain,et al.  Unsupervised texture segmentation using Gabor filters , 1990, 1990 IEEE International Conference on Systems, Man, and Cybernetics Conference Proceedings.

[6]  J. Daugman.  Uncertainty relation for resolution in space, spatial frequency, and orientation optimized by two-dimensional visual cortical filters. , 1985, Journal of the Optical Society of America. A, Optics and image science.

[7]  Stephen P. Boyd,et al.  The Fastest Mixing Markov Process on a Graph and a Connection to a Maximum Variance Unfolding Problem , 2006, SIAM Rev..

[8]  David J. Field,et al.  What Is the Goal of Sensory Coding? , 1994, Neural Computation.

[9]  M. Narasimha Murty,et al.  Growing subspace pattern recognition methods and their neural-network models , 1997, IEEE Trans. Neural Networks.

[10]  Peter N. Heller,et al.  Theory of regular M-band wavelet bases , 1993, IEEE Trans. Signal Process..

[11]  Martin Vetterli,et al.  Wavelets and filter banks: theory and design , 1992, IEEE Trans. Signal Process..

[12]  Susan T. Dumais,et al.  Using Linear Algebra for Intelligent Information Retrieval , 1995, SIAM Rev..

[13]  H. B. Barlow,et al.  Unsupervised Learning , 1989, Neural Computation.

[14]  J. Nachmias,et al.  Discrimination of simple and complex gratings , 1975, Vision Research.

[15]  Christian Jutten,et al.  Blind separation of sources, part I: An adaptive algorithm based on neuromimetic architecture , 1991, Signal Process..

[16]  M. R. Turner,et al.  Texture discrimination by Gabor functions , 1986, Biological Cybernetics.

[17]  Michael Unser,et al.  Multiresolution Feature Extraction and Selection for Texture Segmentation , 1989, IEEE Trans. Pattern Anal. Mach. Intell..

[18]  Jian Fan,et al.  Frame representations for texture segmentation , 1996, IEEE Trans. Image Process..

[19]  C.-C. Jay Kuo,et al.  Wavelet descriptor of planar curves: theory and applications , 1996, IEEE Trans. Image Process..

[20]  Harold H. Szu,et al.  Neural network adaptive wavelets for signal representation and classification , 1992 .

[21]  M. Unser.  Local linear transforms for texture measurements , 1986 .

[22]  Constantine Kotropoulos,et al.  Applying Supervised Classifiers Based on Non-negative Matrix Factorization to Musical Instrument Classification , 2006, 2006 IEEE International Conference on Multimedia and Expo.

[23]  Gil-Jin Jang,et al.  Feature vector transformation using independent component analysis and its application to speaker identification , 1999, EUROSPEECH.

[24]  Christopher J. C. Burges,et al.  Geometric Methods for Feature Extraction and Dimensional Reduction - A Guided Tour , 2005, Data Mining and Knowledge Discovery Handbook.

[25]  Xiaofei He,et al.  Locality Preserving Projections , 2003, NIPS.

[26]  S T Roweis,et al.  Nonlinear dimensionality reduction by locally linear embedding. , 2000, Science.

[27]  Thomas R. Crimmins.  A Complete Set of Fourier Descriptors for Two-Dimensional Shapes , 1982, IEEE Transactions on Systems, Man, and Cybernetics.

[28]  Anil K. Jain.  Fundamentals of Digital Image Processing , 2018, Control of Color Imaging Systems.

[29]  Danny Coomans,et al.  Classification Using Adaptive Wavelets for Feature Extraction , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[30]  Manos Papadakis,et al.  Character recognition using a biorthogonal discrete wavelet transform , 1996, Optics & Photonics.

[31]  Aleksandra Mojsilovic,et al.  On the Selection of an Optimal Wavelet Basis for Texture Characterization , 1998, ICIP.

[32]  D. Esteban,et al.  Application of quadrature mirror filters to split band voice coding schemes , 1977 .

[33]  P. Paatero,et al.  Positive matrix factorization: A non-negative factor model with optimal utilization of error estimates of data values† , 1994 .

[34]  Sergios Theodoridis,et al.  An optical music recognition system for the notation of the Orthodox Hellenic Byzantine Music , 2000, Proceedings 15th International Conference on Pattern Recognition. ICPR-2000.

[35]  Victoria Stodden,et al.  When Does Non-Negative Matrix Factorization Give a Correct Decomposition into Parts? , 2003, NIPS.

[36]  Bedrich J. Hosticka,et al.  A comparison of texture feature extraction using adaptive gabor filtering, pyramidal and tree structured wavelet transforms , 1996, Pattern Recognit..

[37]  Simon Haykin,et al.  Neural Networks: A Comprehensive Foundation , 1998 .

[38]  Robin Sibson,et al.  What is projection pursuit , 1987 .

[39]  F. Campbell,et al.  Orientational selectivity of the human visual system , 1966, The Journal of physiology.

[40]  Phil Brodatz,et al.  Textures: A Photographic Album for Artists and Designers , 1966 .

[41]  Shuicheng Yan,et al.  Graph Embedding and Extensions: A General Framework for Dimensionality Reduction , 2007 .

[42]  Tommi S. Jaakkola,et al.  Weighted Low-Rank Approximations , 2003, ICML.

[43]  Andrew F. Laine,et al.  Wavelet descriptors for multiresolution recognition of handprinted characters , 1995, Pattern Recognit..

[44]  Athanasios Papoulis,et al.  Probability, Random Variables and Stochastic Processes , 1965 .

[45]  Kilian Q. Weinberger,et al.  Unsupervised Learning of Image Manifolds by Semidefinite Programming , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[46]  C.-C. Jay Kuo,et al.  Texture analysis and classification with tree-structured wavelet transform , 1993, IEEE Trans. Image Process..

[47]  Jian Fan,et al.  Texture Classification by Wavelet Packet Signatures , 1993, MVA.

[48]  Pasi Aalto,et al.  Matrix factorization methods for analysing diffusion battery data , 1991 .

[49]  William E. Higgins,et al.  Efficient Gabor filter design for texture segmentation , 1996, Pattern Recognit..

[50]  Bernhard Schölkopf,et al.  Nonlinear Component Analysis as a Kernel Eigenvalue Problem , 1998, Neural Computation.

[51]  Ronald R. Coifman,et al.  Wavelet analysis and signal processing , 1990 .

[52]  Pablo Tamayo,et al.  Metagenes and molecular pattern discovery using matrix factorization , 2004, Proceedings of the National Academy of Sciences of the United States of America.

[53]  Alexander Thomasian,et al.  CSVD: Clustering and Singular Value Decomposition for Approximate Similarity Search in High-Dimensional Spaces , 2003, IEEE Trans. Knowl. Data Eng..

[54]  B. S. Manjunath,et al.  Rotation-invariant texture classification using modified Gabor filters , 1995, Proceedings., International Conference on Image Processing.

[55]  Sergios Theodoridis,et al.  Optical character recognition of the Orthodox Hellenic Byzantine Music notation , 2002, Pattern Recognit..

[56]  K. Karhunen.  Zur Spektraltheorie stochastischer prozesse , 1946 .

[57]  Pierre Comon,et al.  Independent component analysis, A new concept? , 1994, Signal Process..

[58]  Nicolai Petkov,et al.  Comparison of texture features based on Gabor filters , 2002, IEEE Trans. Image Process..

[59]  Edwin R. Hancock,et al.  Clustering and Embedding Using Commute Times , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[60]  Joshua B. Tenenbaum,et al.  Global Versus Local Methods in Nonlinear Dimensionality Reduction , 2002, NIPS.

[61]  Juha Karhunen,et al.  Representation and separation of signals using nonlinear PCA type learning , 1994, Neural Networks.

[62]  H. Sebastian Seung,et al.  Learning the parts of objects by non-negative matrix factorization , 1999, Nature.

[63]  Santosh S. Vempala,et al.  On clusterings: Good, bad and spectral , 2004, JACM.

[64]  Stéphane Mallat,et al.  Multifrequency channel decompositions of images and wavelet models , 1989, IEEE Trans. Acoust. Speech Signal Process..

[65]  Wesley E. Snyder,et al.  Application of Affine-Invariant Fourier Descriptors to Recognition of 3-D Objects , 1990, IEEE Trans. Pattern Anal. Mach. Intell..

[66]  Gustavo Deco,et al.  Linear redundancy reduction learning , 1995, Neural Networks.

[67]  Lijuan Cao,et al.  A comparison of PCA, KPCA and ICA for dimensionality reduction in support vector machine , 2003, Neurocomputing.

[68]  John Langford,et al.  Cover trees for nearest neighbor , 2006, ICML.

[69]  Terrence J. Sejnowski,et al.  The “independent components” of natural scenes are edge filters , 1997, Vision Research.

[70]  Mikhail Belkin,et al.  Laplacian Eigenmaps for Dimensionality Reduction and Data Representation , 2003, Neural Computation.

[71]  Joseph J. Atick.  Entropy Minimization: a Design Principle for Sensory Perception? , 1992, Int. J. Neural Syst..

[72]  Francesco Camastra,et al.  Data dimensionality estimation methods: a survey , 2003, Pattern Recognit..

[73]  Anil K. Jain,et al.  Incremental nonlinear dimensionality reduction by manifold learning , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[74]  Lawrence K. Saul,et al.  Analysis and extension of spectral methods for nonlinear dimensionality reduction , 2005, ICML.

[75]  T. Landauer,et al.  Indexing by Latent Semantic Analysis , 1990 .

[76]  P. Smaragdis,et al.  Non-negative matrix factorization for polyphonic music transcription , 2003, 2003 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (IEEE Cat. No.03TH8684).

[77]  Nicolas Le Roux,et al.  Out-of-Sample Extensions for LLE, Isomap, MDS, Eigenmaps, and Spectral Clustering , 2003, NIPS.

[78]  J. Robson,et al.  Application of fourier analysis to the visibility of gratings , 1968, The Journal of physiology.

[79]  Marian Stewart Bartlett,et al.  Face recognition by independent component analysis , 2002, IEEE Trans. Neural Networks.

[80]  Jan Larsen,et al.  Clustering via kernel decomposition , 2006, IEEE Transactions on Neural Networks.

[81]  D. Boulaud.  Use of granular beds in the inertial impaction regime for aerosol size distribution measurement , 1991 .

[82]  Alan C. Bovik,et al.  Analysis of multichannel narrow-band filters for image texture segmentation , 1991, IEEE Trans. Signal Process..

[83]  L. Saul,et al.  An Introduction to Locally Linear Embedding , 2001 .

[84]  Anastasios Tefas,et al.  Exploiting discriminant information in nonnegative matrix factorization with application to frontal face verification , 2006, IEEE Transactions on Neural Networks.

[85]  Wilson S. Geisler,et al.  Multichannel Texture Analysis Using Localized Spatial Filters , 1990, IEEE Trans. Pattern Anal. Mach. Intell..

[86]  Ann B. Lee,et al.  Diffusion maps and coarse-graining: a unified framework for dimensionality reduction, graph partitioning, and data set parameterization , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[87]  H. Hotelling.  Analysis of a complex of statistical variables into principal components. , 1933 .

[88]  Yousef Saad,et al.  Orthogonal Neighborhood Preserving Projections: A Projection-Based Dimensionality Reduction Technique , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[89]  B. S. Manjunath,et al.  Rotation-invariant texture classification using a complete space-frequency model , 1999, IEEE Trans. Image Process..

[90]  Bernhard Schölkopf,et al.  A kernel view of the dimensionality reduction of manifolds , 2004, ICML.

[91]  Chi-Man Pun,et al.  Log-Polar Wavelet Energy Signatures for Rotation and Scale Invariant Texture Classification , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[92]  Hongbin Zha,et al.  Riemannian Manifold Learning , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.