On the computational rationale for generative models

Generative and discriminative models are best defined by the structure of their graphical representation. This paper introduces such a definition and uses it to argue that, in some practical cases, generative models need to be formulated in order to be implemented within generate-and-test algorithms. This argument is inspired mainly by the ideas of the late Donald MacKay and by considerations of computational complexity.

[1]  A. Pugh.  The art of electronics. 2nd edn: By Paul Horowitz and Winfield Hill. Pp. 1125. Cambridge University Presss. 1989. £29.95, US$49.50 , 1990 .

[2]  Andrew Zisserman,et al.  Texture classification: are filter banks necessary? , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[3]  Giorgio Gambosi,et al.  Complexity and Approximation , 1999, Springer Berlin Heidelberg.

[4]  Shai Avidan,et al.  Support Vector Tracking , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[5]  Ilkay Ulusoy,et al.  Generative versus discriminative methods for object recognition , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[6]  Giorgio Gambosi,et al.  Complexity and approximation: combinatorial optimization problems and their approximability properties , 1999 .

[7]  Ahmed M. Elgammal,et al.  Nonlinear manifold learning for dynamic shape and dynamic appearance , 2007, Comput. Vis. Image Underst..

[8]  Frank Dellaert,et al.  EM, MCMC, and Chain Flipping for Structure from Motion with Unknown Correspondence , 2004, Machine Learning.

[9]  Arthur E. C. Pece,et al.  Contour tracking based on marginalized likelihood ratios , 2006, Image Vis. Comput..

[10]  Luc Van Gool,et al.  Smart particle filtering for high-dimensional tracking , 2007, Comput. Vis. Image Underst..

[11]  Judea Pearl,et al.  Fusion, Propagation, and Structuring in Belief Networks , 1986, Artif. Intell..

[12]  Arthur E. C. Pece.  Generative model based vision , 2003, Image Vis. Comput..

[13]  Brendan J. Frey,et al.  A comparison of algorithms for inference and learning in probabilistic graphical models , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[14]  P. Tseng,et al.  Block Coordinate Relaxation Methods for Nonparametric Wavelet Denoising , 2000 .

[15]  S. Mallat,et al.  Adaptive greedy approximations , 1997 .

[16]  Donald Ervin Knuth,et al.  The Art of Computer Programming , 1968 .

[17]  D. Mumford.  Pattern theory: a unifying perspective , 1996 .

[18]  Terry Caelli,et al.  Bayesian Stereo Matching , 2004, 2004 Conference on Computer Vision and Pattern Recognition Workshop.

[19]  Pietro Perona,et al.  Learning Generative Visual Models from Few Training Examples: An Incremental Bayesian Approach Tested on 101 Object Categories , 2004, 2004 Conference on Computer Vision and Pattern Recognition Workshop.

[20]  Michael C. Nechyba,et al.  Interpretation of complex scenes using dynamic tree-structure Bayesian networks , 2007, Comput. Vis. Image Underst..

[21]  P. C. Murphy,et al.  Corticofugal feedback influences the generation of length tuning in the visual pathway , 1987, Nature.

[22]  Arthur E. C. Pece,et al.  Generative-model-based tracking by cluster analysis of image differences , 2002, Robotics Auton. Syst..

[23]  Nando de Freitas,et al.  Sequential Monte Carlo Methods in Practice , 2001, Statistics for Engineering and Information Science.

[24]  S. Thorpe,et al.  A Limit to the Speed of Processing in Ultra-Rapid Visual Categorization of Novel Natural Scenes , 2001, Journal of Cognitive Neuroscience.

[25]  W. Richards,et al.  Perception as Bayesian Inference , 2008 .

[26]  Timothy F. Cootes,et al.  Texture enhanced appearance models , 2007, Comput. Vis. Image Underst..

[27]  Brendan J. Frey,et al.  Learning flexible sprites in video layers , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[28]  S. Thorpe,et al.  Speed of processing in the human visual system , 1996, Nature.

[29]  David Mumford,et al.  Statistics of natural images and models , 1999, Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149).

[30]  James M. Coughlan,et al.  Dynamic quantization for belief propagation in sparse spaces , 2007, Comput. Vis. Image Underst..

[31]  Charles A. Bouman,et al.  Fast search for best representations in multitree dictionaries , 2006, IEEE Transactions on Image Processing.

[32]  William H. Press,et al.  Numerical recipes in C , 2002 .

[33]  D. Du,et al.  Theory of Computational Complexity , 2000 .

[34]  Luc Van Gool,et al.  A Probabilistic Approach to Optical Flow based Super-Resolution , 2004, 2004 Conference on Computer Vision and Pattern Recognition Workshop.

[35]  Paul A. Viola,et al.  Rapid object detection using a boosted cascade of simple features , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[36]  J. M. Hupé,et al.  Cortical feedback improves discrimination between figure and background by V1, V2 and V3 neurons , 1998, Nature.

[37]  D. Mackay.  The Epistemological Problem for Automata , 1956 .

[38]  Hans-Hellmut Nagel,et al.  3D Pose Estimation by Directly Matching Polyhedral Models to Gray Value Gradients , 1997, International Journal of Computer Vision.

[39]  A. Tarantola.  Popper, Bayes and the inverse problem , 2006 .

[40]  Song-Chun Zhu,et al.  Statistical Modeling and Conceptualization of Visual Patterns , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[41]  Heekuck Oh,et al.  Neural Networks for Pattern Recognition , 1993, Adv. Comput..

[42]  Kenji Kawano,et al.  Global and fine information coded by single neurons in the temporal visual cortex , 1999, Nature.

[43]  Manuele Bicego,et al.  Unsupervised scene analysis: a hidden Markov model approach , 2006 .

[44]  Anthony D. Worrall,et al.  A comparison between feature-based and EM-based contour tracking , 2005, Image Vis. Comput..

[45]  Ronald R. Coifman,et al.  Entropy-based algorithms for best basis selection , 1992, IEEE Trans. Inf. Theory.

[46]  John E. Dennis,et al.  Numerical methods for unconstrained optimization and nonlinear equations , 1983, Prentice Hall series in computational mathematics.

[47]  Song-Chun Zhu,et al.  Primal sketch: Integrating structure and texture , 2007, Comput. Vis. Image Underst..

[48]  Victor A. F. Lamme,et al.  Feedforward, horizontal, and feedback processing in the visual cortex , 1998, Current Opinion in Neurobiology.

[49]  Andrew Blake,et al.  Sparse Bayesian learning for efficient visual tracking , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[50]  Yoav Freund,et al.  Experiments with a New Boosting Algorithm , 1996, ICML.

[51]  Michael A. Saunders,et al.  Atomic Decomposition by Basis Pursuit , 1998, SIAM J. Sci. Comput..

[52]  John Nolte,et al.  The Human Brain An Introduction to Its Functional Anatomy , 2013 .

[53]  T. Minka.  Discriminative models, not discriminative training , 2005 .

[54]  Geoffrey E. Hinton,et al.  Learning representations by back-propagating errors , 1986, Nature.

[55]  Paul Horowitz,et al.  The Art of Electronics , 1980 .

[56]  J. Rauschecker,et al.  Hierarchical Organization of the Human Auditory Cortex Revealed by Functional Magnetic Resonance Imaging , 2001, Journal of Cognitive Neuroscience.

[57]  Arthur E. C. Pece,et al.  The Problem of Sparse Image Coding , 2002, Journal of Mathematical Imaging and Vision.

[58]  Donald E. Knuth,et al.  The art of computer programming, volume 3: (2nd ed.) sorting and searching , 1998 .

[59]  Bo Markussen,et al.  Large deformation diffeomorphisms with application to optic flow , 2007, Comput. Vis. Image Underst..

[60]  D. Ruderman.  The statistics of natural images , 1994 .