Spacetime Texture Representation and Recognition Based on a Spatiotemporal Orientation Analysis

This paper is concerned with the representation and recognition of the observed dynamics (i.e., excluding purely spatial appearance cues) of spacetime texture based on a spatiotemporal orientation analysis. The term “spacetime texture” is taken to refer to patterns in visual spacetime, (x,y,t), that primarily are characterized by the aggregate dynamic properties of elements or local measurements accumulated over a region of spatiotemporal support, rather than in terms of the dynamics of individual constituents. Examples include image sequences of natural processes that exhibit stochastic dynamics (e.g., fire, water, and windblown vegetation) as well as images of simpler dynamics when analyzed in terms of aggregate region properties (e.g., uniform motion of elements in imagery, such as pedestrians and vehicular traffic). Spacetime texture representation and recognition is important as it provides an early means of capturing the structure of an ensuing image stream in a meaningful fashion. Toward such ends, a novel approach to spacetime texture representation and an associated recognition method are described based on distributions (histograms) of spacetime orientation structure. Empirical evaluation on both standard and original image data sets shows the promise of the approach, including significant improvement over alternative state-of-the-art approaches in recognizing the same pattern from different viewpoints.

[1]  David G. Stork,et al.  Pattern Classification , 1973 .

[2]  Shigeo Abe DrEng.  Pattern Classification , 2001, Springer London.

[3]  Suzanne Beauchemin,et al.  The Frequency Structure of 1D Occluding Image Signals , 2000 .

[4]  W. Marsden.  I and J , 2012 .

[5]  E. Rosch,et al.  Family resemblances: Studies in the internal structure of categories , 1975, Cognitive Psychology.

[6]  Song-Chun Zhu,et al.  Modeling textured motion : particle, wave and sketch , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[7]  D J Heeger,et al.  Model for the extraction of image flow. , 1987, Journal of the Optical Society of America. A, Optics and image science.

[8]  Kristin J. Dana,et al.  3D Texture Recognition Using Bidirectional Feature Histograms , 2004, International Journal of Computer Vision.

[9]  Martin Szummer,et al.  Temporal texture modeling , 1996, Proceedings of 3rd IEEE International Conference on Image Processing.

[10]  Michael S. Langer,et al.  Optical Snow , 2003, International Journal of Computer Vision.

[11]  David J. Fleet.  Measurement of image velocity , 1992 .

[12]  Richard P. Wildes,et al.  The Structure of Multiplicative Motions in Natural Imagery , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[13]  Konstantinos G. Derpanis,et al.  Three-dimensional nth derivative of Gaussian separable steerable filters , 2005, IEEE International Conference on Image Processing 2005.

[14]  Steven S. Beauchemin,et al.  The Frequency Structure of One-Dimensional Occluding Image Signals , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[15]  Weichuan Yu,et al.  Detection and characterization of multiple motion points , 1999, Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149).

[16]  Richard P. Wildes,et al.  Efficient action spotting based on a spacetime oriented structure representation , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[17]  David J. Heeger,et al.  Seeing structure through chaos , 1986 .

[18]  Richard P. Wildes,et al.  Spatiotemporal stereo via spatiotemporal quadric element (stequel) matching , 2009, CVPR.

[19]  Richard P. Wildes,et al.  Detecting Spatiotemporal Structure Boundaries: Beyond Motion Discontinuities , 2009, ACCV.

[20]  Weixin Xie,et al.  Dynamic Texture Recognition by Spatio-Temporal Multiresolution Histograms , 2005, 2005 Seventh IEEE Workshops on Applications of Computer Vision (WACV/MOTION'05) - Volume 1.

[21]  Takeo Kanade,et al.  Analysis of Rain and Snow in Frequency Space , 2008, International Journal of Computer Vision.

[22]  Dmitry Chetverikov,et al.  A Brief Survey of Dynamic Texture Description and Recognition , 2005, CORES.

[23]  Shree K. Nayar,et al.  Vision and Rain , 2007, International Journal of Computer Vision.

[24]  Edward H. Adelson,et al.  The Design and Use of Steerable Filters , 1991, IEEE Trans. Pattern Anal. Mach. Intell..

[25]  Richard P. Wildes,et al.  Early spatiotemporal grouping with a distributed oriented energy representation , 2009, CVPR.

[26]  Serge J. Belongie,et al.  Behavior recognition via sparse spatio-temporal features , 2005, 2005 IEEE International Workshop on Visual Surveillance and Performance Evaluation of Tracking and Surveillance.

[27]  Andrew B. Watson,et al.  A look at motion in the frequency domain , 1983 .

[28]  Richard P. Wildes,et al.  Visual Tracking Using a Pixelwise Spatiotemporal Oriented Energy Representation , 2010, ECCV.

[29]  Hans Knutsson,et al.  Signal processing for computer vision , 1994 .

[30]  Nuno Vasconcelos,et al.  Probabilistic kernels for the classification of auto-regressive visual processes , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[31]  Robert Sekuler,et al.  Coherent global motion percepts from stochastic local motions , 1984, Vision Research.

[32]  Stefano Soatto,et al.  Dynamic Textures , 2003, International Journal of Computer Vision.

[33]  C. H. Chen,et al.  Handbook of Pattern Recognition and Computer Vision , 1993 .

[34]  Patrick Bouthemy,et al.  Motion characterization from temporal cooccurrences of local motion-based measures for video indexing , 1998, Proceedings. Fourteenth International Conference on Pattern Recognition (Cat. No.98EX170).

[35]  Matti Pietikäinen,et al.  Dynamic Texture Recognition Using Local Binary Patterns with an Application to Facial Expressions , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[36]  A. Fitzgibbon.  Stochastic rigidity: image registration for nowhere-static scenes , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[37]  T. Poggio,et al.  Visual hyperacuity: spatiotemporal interpolation in human vision , 1981, Proceedings of the Royal Society of London. Series B. Biological Sciences.

[38]  Richard P. Wildes,et al.  Qualitative Spatiotemporal Analysis Using an Oriented Energy Representation , 2000, ECCV.

[39]  René Vidal,et al.  View-invariant dynamic texture recognition using a bag of dynamical systems , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[40]  Stefan Treue,et al.  Seeing multiple directions of motion—physiology and psychophysics , 2000, Nature Neuroscience.

[41]  Nuno Vasconcelos,et al.  Classifying Video with Kernel Dynamic Textures , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[42]  Andrei Zaharescu,et al.  Anomalous Behaviour Detection Using Spatiotemporal Oriented Energies, Subset Inclusion Histogram Comparison and Event-Driven Processing , 2010, ECCV.

[43]  Tony Lindeberg,et al.  Linear Spatio-Temporal Scale-Space , 1997, Scale-Space.

[44]  Anil K. Jain,et al.  Texture Analysis , 2018, Handbook of Image Processing and Computer Vision.

[45]  Robert Sekuler,et al.  Using metamers to explore motion perception , 1991, Vision Research.

[46]  Leonidas J. Guibas,et al.  The Earth Mover's Distance as a Metric for Image Retrieval , 2000, International Journal of Computer Vision.

[47]  Richard P. Wildes,et al.  Classification of traffic video based on a spatiotemporal orientation analysis , 2011, 2011 IEEE Workshop on Applications of Computer Vision (WACV).

[48]  Andrew W. Fitzgibbon,et al.  Shift-Invariant Dynamic Texture Recognition , 2006, ECCV.

[49]  Andrew Zisserman,et al.  A Statistical Approach to Texture Classification from Single Images , 2004, International Journal of Computer Vision.

[50]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[51]  Randal C. Nelson,et al.  Qualitative recognition of motion using temporal texture , 1992, CVGIP Image Underst..

[52]  Richard P. Wildes,et al.  Dynamic texture recognition based on distributions of spacetime oriented structure , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[53]  Dani Lischinski,et al.  Texture Mixing and Texture Movie Synthesis Using Statistical Learning , 2001, IEEE Trans. Vis. Comput. Graph..

[54]  Jitendra Malik,et al.  Representing and Recognizing the Visual Appearance of Materials using Three-dimensional Textons , 2001, International Journal of Computer Vision.

[55]  K. W. Cattermole.  The Fourier Transform and its Applications , 1965 .

[56]  James L. Crowley,et al.  Probabilistic recognition of activity using local appearance , 1999, Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149).

[57]  Ramprasad Polana,et al.  Temporal texture and activity recognition , 1994 .

[58]  E H Adelson,et al.  Spatiotemporal energy models for the perception of motion. , 1985, Journal of the Optical Society of America. A, Optics and image science.

[59]  Payam Saisan,et al.  Dynamic texture recognition , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.