Principal component learning networks and applications

In this thesis we propose various extensions of Principal Component Analysis in relation to neural network learning theory. We first propose a new network model (APEX), for multiple principal component extraction which utilizes the normalized Hebbian learning rule for training all the synaptic weights. The network features a novel structure of lateral connections among the output units for the orthogonalization of the weights. The exponential convergence of the model is formally proved while there is significant performance improvement over previous methods. Also the network is able to grow or shrink without need for retraining the old units. Furthermore, APEX is proved to be applicable to the Constrained PCA problem where the signal variance is maximized under external orthogonality constraints. Another PCA extension introduced in this thesis is Oriented PCA where the variance ratio between two signals is maximized. We propose two learning models for OPC extraction, (justified both mathematically and by simulation) based again on the Hebbian rule. The least squares criterion for a supervised two-layer linear network with reduced hidden units is related to a Generalized SVD problem which we call Asymmetric PCA (since PCA is a special case when input and teacher are the same). We show that back-propagation extracts the APCA component subspace but we also propose a new model using the Lateral Orthogonalization Net for extracting the exact components. Another PCA extension is Cross-correlation APCA where the cross-correlation between two signals is maximized. A new model featuring a LON is proposed and proved to work for this task. We then study some of the applications of the new models which require adaptive solution methods, in particular signal detection, system identification and image compression applications. Finally, we address the problem of motion compensation for image coding. Standard block-matching techniques introduce block artifacts in the predicted picture which are not easily removed at very low bit rates. We propose a new motion compensation technique using a histogram-based segmentation scheme that reduces the artifacts at no cost to the decoder. The technique achieves reduced bit-rate for the same error compared to the standard algorithm.