A study of the generalized eigenvalue decomposition in discriminant analysis

The well-known Linear Discriminant Analysis (LDA) approac h to feature extraction in classification problems is typically formulated using a gen eralized eigenvalue decomposition, S1V = S2VΛ, whereS1 andS2 are two symmetric, positive-semidefinite matrices defining the measure to be maximized and that to be minimized. Most of the LDA algorithms developed to date are based on tuning one of these two m atrices to solve a specific problem. However, the search for a set of metrics that can be a pplied to a large number of problems has met difficulty. In this thesis, we take the view t hat most of these problem are caused by the use of the generalized eigenvalue decompositi on equation described above. Further, we argue that many of these problems can be solved by stud ing and modifying this basic equation. At the core of this thesis lays a new fact orization ofS−1 2 S1 that can be used to resolve several of the problems of LDA. Three novel algorithms are derived, each based on our propos ed factorization. In the first algorithm, we define a criterion to prune noisy bases in L DA. This is possible thanks to the flexibility of our factorization, which allows the sup pression of a set of vectors of any metric. The second algorithm is called Subclass Discrimina nt Analysis (SDA). SDA can be applied to a large variety of distribution types because i t approximates the underlying distribution of each class with a mixture of Gaussians. The m ost convenient number of Gaussians can be readily selected thanks to our proposed fac toriz tion. The third algorithm is aimed to address the over-fitting issue in LDA. A direct app lication of this algorithm is