Shared Weights Neural Networks in Image Analysis

This thesis is concerned with the use of shared weights neural networks in image analysis. This type of neural network has been extensively described in literature since 1989. It is believed that networks incorporating shared weights are capable of local, shift-invariant feature extraction due to the restrictions placed on their architecture. The rst experiments were focused mainly on the neural network architectures as suggested by, amongst others, Le Cun et al. LBD + 89, LBD + 90, LJB + 89] and Viennet Vie93]. These archi-tectures basically are back-propagation neural networks. However, they restrain the number of free parameters and introduce the notion of receptive elds, combining local information into more abstract patterns at a higher level. Three of these networks were tested on the problem of handwritten digit recognition and the results were compared to those of methods based on other feature extraction or classiication techniques. As an intermezzo, a second order minimization method used in neural network training will be discussed, the pseudo-Newton method. There has been some debate on the advantages and disadvantages of this method. Experiments have been performed on a simulated error surface and on real neural network problems to investigate this method's behaviour. The goal of the last part of the thesis is to place the speciic example of handwritten digit recognition using shared weights neural networks in a more general context. The feature extraction capabilities of these networks were investigated on some problems and the learned templates and lters were compared to classic image processing approaches. Some qualitative conclusions can be drawn from these experiments concerning the innuence of network and data-set complexity, training algorithms and their parameters and weight initialization on training results. v vi Acknowledgements This thesis was written as a report of the research I performed for my graduation in the Know-like to thank them both for their guidance, help and support, and for showing me new ways to look at artiicial intelligence and pattern recognition. It seems to have become a tradition for students of the Pattern Recogniton Group to thank just about every person they have met in Delft in their M.Sc. theses. Following that tradition, I would simply like to thank everyone who would like to be thanked. Special thanks go to ir. Michael van Ginkel for (too) frequently letting me pick his brain on some image processing problems. vii viii Contents Abstract v Acknowledgements vii 1 Introduction …