The detection of 2D image features using local energy

Accurate detection and localization of two-dimensional (2D) image features (or `keypoints') is important for vision tasks such as structure from motion, stereo matching, and line labeling. 2D image features are ideal for these vision tasks because 2D image features are high in information and yet they occur sparsely in typical images. Several methods for the detection of 2D image features have already been developed. However, it is di cult to assess the performance of these methods because no one has produced an adequate de nition of corners that encompasses all types of 2D luminance variations that make up 2D image features. The fact that there does not exist a consensus on the de nition of 2D image features is not surprising given the confusion surrounding the de nition of 1D image features. The general perception of 1D image features has been that they correspond to `edges' in an image and so are points where the intensity gradient in some direction is a local maximum. The Sobel [68], Canny [7] and Marr-Hildreth [37] operators all use this model of 1D features, either implicitly or explicitly. However, other pro les in an image also make up valid 1D features, such as spike and roof pro les, as well as combinations of all these feature types. Spikes and roof pro les can also be found by looking for points where the rate of change of the intensity gradient is locally maximal, as Canny did in de ning a `roof-detector' in much the same manner that he developed his `edge-detector'. While this allows the detection of a wider variety of 1D features pro les, it comes no closer to the goal of unifying these di erent feature types to an encompassing de nition of 1D features. The introduction of the local energy model of image features by Morrone and Owens [45] in 1987 provided a uni ed de nition of 1D image features for the rst iii time. They postulated that image features correspond to points in an image where there is maximal phase congruency in the frequency domain representation of the image. That is, image features correspond to points of maximal order in the phase domain of the image signal. These points of maximal phase congruency correspond to step-edge, roof, and ramp intensity pro les, and combinations thereof. They also correspond to the Mach bands perceived by humans in trapezoidal feature pro les. This thesis extends the notion of phase congruency to 2D image features. As 1D image features correspond to points of maximal 1D order in the phase domain of the image signal, this thesis contends that 2D image features correspond to maximal 2D order in this domain. These points of maximal 2D phase congruency include all the di erent types of 2D image features, including grey-level corners, line terminations, blobs, and a variety of junctions. Early attempts at 2D feature detection were simple `corner detectors' based on a model of a grey-level corner in much the same way that early 1D feature detectors were based on a model of step-edges. Some recent attempts have included more complex models of 2D features, although this is basically a more complex a priori judgement of the types of luminance pro les that are to be labeled as 2D features. This thesis develops the 2D local energy feature detector based on a new, uni ed de nition of 2D image features that marks points of locally maximum 2D order in the phase domain representation of the image as 2D image features. The performance of an implementation of 2D local energy is assessed, and compared to several existing methods of 2D feature detection. This thesis also shows that in contrast to most other methods of 2D feature detection, 2D local energy is an idempotent operator. The extension of phase congruency to 2D image features also uni es the detection of image features. 1D and 2D image features correspond to 1D and 2D order in the phase domain representation of the image respectively. This de nition imposes a hierarchy of image features, with 2D image features being a subset of 1D image features. This ordering of image features has been implied ever since 1D features were used as candidate points for 2D feature detection by Kitchen [28] and others. Local energy enables the extraction of both 1D and 2D image features in a consistent manner; 2D image features are extracted from the 1D image features using the same iv operations that are used to extract 1D image features from the input image. The consistent approach to the detection of image features presented in this thesis allows the hierarchy of primitive image features to be naturally extended to higher order image features. These higher order image features can then also be extracted from higher order image data using the same hierarchical approach. This thesis shows how local energy can be naturally extended to the detection of 1D (surface) and higher order image features in 3D data sets. Results are presented for the detection of 1D image features in 3D confocal microscope images, showing superior performance to the 3D extension of the Sobel operator [74]. v Preface Some of the work in this thesis has already been published. Most of the work in Chapters 4 to 6 appears in a technical report in the Department of Computer Science [58], and a more concise version of this work is to appear in Image and Vision Computing [59]. I am the principal contributing author for both these papers. With the exception of the 3D surface detector described in Section 7.1, all of the work presented in this thesis|including algorithms and implementations|is my own. The surface detection work has been principally performed by Chris Pudney, with my contribution being to the general methodology of implementation and the optimizations with regard to performing the FFTs and applying the energy lters. Various forms of this work have been published in the Proceedings of ANZIIS'95 [53] and Proceedings of the International Computer Science Conference [54] with another paper on this work to appear in the Journal of Assisted Confocal Microscopy. vi Acknowledgements First of all I would like to thank my supervisor Robyn Owens for her constant support and guidance throughout my candidature. She ensured that I got o on the right foot and always knew where I was, and where I was going even when I didn't, particularly at the beginning of my studies. It was always reassuring to know that she would quickly understand any problems I was grappling with and o er helpful suggestions, although the speed she did this and its apparent ease to her were a little disconcerting. Robyn also strongly encouraged me to apply to travel to Switzerland to study at ETH Z urich which was both very bene cial to my studies and a fantastic time. I would like to thank UWA and ETH Z urich for supporting my stay in Switzerland. Thanks go to everyone in the Image Sciences (BIWI) group at ETH for making my stay educational and enjoyable. Olaf, Vreni, Martin, Markus and the lunch crew, Gaudenz, Friedrich, Wolfram, Tuomo, Olof, and Marjan made for an entertaining time. Extra special thanks go to the K ublers for making me feel like part of the family during my stay in Switzerland; Olaf and Guni for showing me the beautiful Swiss countryside while unsuccessfully trying to give me wanderschaden, Dani for risking the meat with an Aussie at his birthday party BBQ, and Flo for being such a great mate, for losing the Kiwi accent (eh?), and for nally getting his revenge for numerous dumpings at Triggs Beach by taking me snowboarding on a hard-as-rock glacier. I would also like to thank the non-resident member of the K ubler household, Gaudie, for his friendship and for introducing me to the outdoor cinema, and Roman for all the basketball. I have bene ted from many conversations with Mike Robbins, Peter Kovesi and vii Chris Pudney regarding local energy. Bruce Backman was a good sounding board for ideas and always provided a di erent perspective. Friedrich Heitger was very helpful with e-mail messages when I was trying to implement his work, and while I was at ETH taught me much of what I know about lter design. Olof Henricsson and I also had many interesting discussions regarding the use of low-level features after our weekly hit of tennis. Lachlan Partington and Chris Pudney provided valuable feedback through their careful reading of a late draft of this thesis. I also thank Chris for supplying both the output for Figure 4 and numerous tips on LATEX. I would like to thank the Department of Computer Science at The University of Western Australia and the Department of Education, Employment and Training for their support during this candidature. Thanks go to my family for their support of my studies, especially my parents who put me through school and then put up with me at home for most of my tertiary studies. I am forever endebted to mywife Karenza for her love and support throughout my studies and to her I give special thanks. viii