Data-driven approaches for texture and motion

Access to vast amounts of visual information and the increase in computing power are facilitating the emergence of a new class of simple data-driven approaches for image analysis and synthesis. Instead of building an explicit parametric model of a phenomenon, the data-driven techniques rely on the underlying data to serve as its own representation. This dissertation presents work in two different domains, visual texture and human motion, where data-driven approaches have been particularly successful. Part I describes two algorithms for texture synthesis, the problem of synthesizing visual texture (e.g. grass, bricks, pebbles) from an example image. The goal is to produce novel samples of a given texture, which, while not identical, will be perceived by humans as the same texture. The proposed texture synthesis process grows a new image outward from an initial seed, one pixel/patch at a time. A Markov random field model is assumed, and the conditional distribution of a pixel/patch given all its neighbors synthesized so far is estimated by querying the sample image and finding all similar neighborhoods. The degree of randomness is controlled by a single perceptually intuitive parameter. The method aims at preserving as much local structure as possible and produces good results for a wide variety of synthetic and real-world textures. One discussed application of the method is texture transfer, a novel technique that allows texture from one object to be “painted” onto a different object. Part II presents an algorithm for the analysis and synthesis of human motion from video. Instead of reconstructing a 3D model of the human figure (which is hard), the idea is to “explain” a novel motion sequence with bits and pieces from a large collection of stored video data. This simple method can be used for both action recognition as well as motion transfer—synthesizing a novel person imitating the actions of another person (“Do as I Do” synthesis) or performing actions according to the specified action labels (“Do as I Say” synthesis).

[1]  R. Hetherington.  The Perception of the Visual World , 1952 .

[2]  Béla Julesz,et al.  Visual Pattern Discrimination , 1962, IRE Trans. Inf. Theory.

[3]  P. O. Bishop,et al.  Spatial vision. , 1971, Annual review of psychology.

[4]  David Donovan Garber,et al.  Computational models for texture analysis and texture synthesis , 1981 .

[5]  Takeo Kanade,et al.  An Iterative Image Registration Technique with an Application to Stereo Vision , 1981, IJCAI.

[6]  A. Mackay.  "Textures" , 1987 .

[7]  E. Adelson,et al.  Early vision and texture perception , 1988, Nature.

[8]  P Perona,et al.  Preattentive texture discrimination with early vision mechanisms. , 1990, Journal of the Optical Society of America. A, Optics and image science.

[9]  Kris Popat,et al.  Novel cluster-based probability model for texture synthesis, classification, and compression , 1993, Other Conferences.

[10]  James R. Bergen,et al.  Pyramid-based texture analysis/synthesis , 1995, Proceedings., International Conference on Image Processing.

[11]  Jonas Gårding,et al.  Surface orientation and curvature from differential texture distortion , 1995, Proceedings of IEEE International Conference on Computer Vision.

[12]  Douglas R. Hofstadter,et al.  Fluid Concepts and Creative Analogies , 1995 .

[13]  Jitendra Malik,et al.  Modeling and Rendering Architecture from Photographs: A hybrid geometry- and image-based approach , 1996, SIGGRAPH.

[14]  Kazuo Kyuma,et al.  Computer vision for computer games , 1996, Proceedings of the Second International Conference on Automatic Face and Gesture Recognition.

[15]  Jeremy S. De Bonet,et al.  Multiresolution sampling procedure for analysis and synthesis of texture images , 1997, SIGGRAPH.

[16]  Joshua B. Tenenbaum,et al.  Learning bilinear models for two-factor problems in vision , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[17]  Christoph Bregler,et al.  Video Rewrite: Driving Visual Speech with Audio , 1997, SIGGRAPH.

[18]  David Salesin,et al.  Computer-generated watercolor , 1997, SIGGRAPH.

[19]  David Salesin,et al.  Orientable textures for image-based pen-and-ink illustration , 1997, SIGGRAPH.

[20]  Mubarak Shah,et al.  Motion-Based Recognition , 1997, Computational Imaging and Vision.

[21]  James Davis,et al.  Mosaics of scenes with moving objects , 1998, Proceedings. 1998 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No.98CB36231).

[22]  Roger D. Hersch,et al.  Multi-color and artistic dithering , 1999, SIGGRAPH.

[23]  Dariu Gavrila,et al.  The Visual Analysis of Human Movement: A Survey , 1999, Comput. Vis. Image Underst..

[24]  Alexei A. Efros,et al.  Texture synthesis by non-parametric sampling , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[25]  Michael J. Black,et al.  Parameterized Modeling and Recognition of Activities , 1999, Comput. Vis. Image Underst..

[26]  Adam Finkelstein,et al.  Lapped textures , 2000, SIGGRAPH.

[27]  Takeo Kanade,et al.  A statistical method for 3D object detection applied to faces and cars , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[28]  Xavier Binefa,et al.  Robust Real-Time Periodic Motion Detection, Analysis, and Applications , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[29]  Camillo J. Taylor,et al.  Reconstruction of Articulated Objects from Point Correspondences in a Single Uncalibrated Image , 2000, Comput. Vis. Image Underst..

[30]  Pierre Poulin,et al.  Extraction and Synthesis of Bump Maps from Photographs , 2000 .

[31]  Richard Szeliski,et al.  Video textures , 2000, SIGGRAPH.

[32]  Baining Guo,et al.  Chaos Mosaic: Fast and Memory Efficient Texture Synthesis , 2000 .

[33]  Marc Levoy,et al.  Fast texture synthesis using tree-structured vector quantization , 2000, SIGGRAPH.

[34]  Mubarak Shah,et al.  View-invariance in action recognition , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[35]  Marc Levoy,et al.  Texture synthesis over arbitrary manifold surfaces , 2001, SIGGRAPH.

[36]  Greg Turk,et al.  Texture synthesis on surfaces , 2001, SIGGRAPH.

[37]  Henning Biermann,et al.  Texture and Shape Synthesis on Surfaces , 2001, Rendering Techniques.

[38]  M. Irani,et al.  Event-Based Video Analysis, , 2001 .

[39]  James W. Davis,et al.  The Recognition of Human Movement Using Temporal Templates , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[40]  Jitendra Malik,et al.  Geometric blur for template matching , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[41]  Ramakant Nevatia,et al.  Segmentation and tracking of multiple humans in complex situations , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[42]  Alexei A. Efros,et al.  Image quilting for texture synthesis and transfer , 2001, SIGGRAPH.

[43]  Baining Guo,et al.  Real-time texture synthesis by patch-based sampling , 2001, TOGS.

[44]  Paul Harrison,et al.  A Non-Hierarchical Procedure for Re-Synthesis of Complex Textures , 2001, WSCG.

[45]  Michael Ashikhmin,et al.  Synthesizing natural textures , 2001, I3D '01.

[46]  Segmentation by Example , 2002 .

[47]  Ramakant Nevatia,et al.  3D tracking of human locomotion: a tracking as recognition approach , 2002, Object recognition supported by user interaction for service robots.

[48]  Baining Guo,et al.  Synthesis of bidirectional texture functions on arbitrary surfaces , 2002, SIGGRAPH.

[49]  Shimon Ullman,et al.  Class-Specific, Top-Down Segmentation , 2002, ECCV.

[50]  Robert T. Collins,et al.  Silhouette-based human identification from body shape and gait , 2002, Proceedings of Fifth IEEE International Conference on Automatic Face Gesture Recognition.

[51]  Nuria Oliver,et al.  Curve Analogies , 2002, Rendering Techniques.

[52]  Jitendra Malik,et al.  Estimating Human Body Configurations Using Shape Context Matching , 2002, ECCV.

[53]  Marie-Paule Cani,et al.  Hierarchical pattern mapping , 2002, ACM Trans. Graph..

[54]  Patrick Pérez,et al.  Object removal by exemplar-based inpainting , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[55]  Yang Song,et al.  Unsupervised Learning of Human Motion , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[56]  Irfan A. Essa,et al.  Graphcut textures: image and video synthesis using graph cuts , 2003, ACM Trans. Graph..

[57]  David A. Forsyth,et al.  Automatic Annotation of Everyday Movements , 2003, NIPS.

[58]  Kun Zhou,et al.  Synthesis of progressively-variant textures on arbitrary surfaces , 2003, ACM Trans. Graph..

[59]  Eero P. Simoncelli,et al.  A Parametric Texture Model Based on Joint Statistics of Complex Wavelet Coefficients , 2000, International Journal of Computer Vision.

[60]  Jitendra Malik,et al.  Computing Local Surface Orientation and Shape from Texture for Curved Surfaces , 1997, International Journal of Computer Vision.

[61]  Randal C. Nelson,et al.  Detection and Recognition of Periodic, Nonrigid Motion , 1997, International Journal of Computer Vision.

[62]  Song-Chun Zhu,et al.  Filters, Random Fields and Maximum Entropy (FRAME): Towards a Unified Theory for Texture Modeling , 1998, International Journal of Computer Vision.

[63]  Steven M. Seitz,et al.  View-Invariant Analysis of Cyclic Motion , 1997, International Journal of Computer Vision.