The multiscale structure of non-differentiable image manifolds

In this paper, we study families of images generated by varying a parameter that controls the appearance of the object/scene in each image. Each image is viewed as a point in high-dimensional space; the family of images forms a low-dimensional submanifold that we call an image appearance manifold (IAM). We conduct a detailed study of some representative IAMs generated by translations/rotations of simple objects in the plane and by rotations of objects in 3-D space. Our central, somewhat surprising, finding is that IAMs generated by images with sharp edges are nowhere differentiable. Moreover, IAMs have an inherent multiscale structure in that approximate tangent planes fitted to ε-neighborhoods continually twist off into new dimensions as the scale parameter ε varies. We explore and explain this phenomenon. An additional, more exotic kind of local non-differentiability happens at some exceptional parameter points where occlusions cause image edges to disappear. These non-differentiabilities help to understand some key phenomena in image processing. They imply that Newton's method will not work in general for image registration, but that a multiscale Newton's method will work. Such a multiscale Newton's method is similar to existing coarse-to-fine differential estimation algorithms for image registration; the manifold perspective offers a well-founded theoretical motivation for the multiscale approach and allows quantitative study of convergence and approximation. The manifold viewpoint is also generalizable to other image understanding problems.

[1]  W. Kahan,et al.  The Rotation of Eigenvectors by a Perturbation. III , 1970 .

[2]  Eero P. Simoncelli Coarse-to-fine estimation of visual motion , 1993 .

[3]  William T. Freeman,et al.  Exploiting the generic viewpoint assumption , 1996, International Journal of Computer Vision.

[4]  J. Tenenbaum,et al.  A global geometric framework for nonlinear dimensionality reduction. , 2000, Science.

[5]  Peter Schröder,et al.  Multiscale Representations for Manifold-Valued Data , 2005, Multiscale Model. Simul..

[6]  S T Roweis,et al.  Nonlinear dimensionality reduction by locally linear embedding. , 2000, Science.

[7]  David L. Donoho,et al.  Image Manifolds which are Isometric to Euclidean Space , 2005, Journal of Mathematical Imaging and Vision.

[8]  D. Donoho Wedgelets: nearly minimax estimation of edges , 1999 .

[9]  L. Quam Hierarchical warp stereo , 1987 .

[10]  Justin K. Romberg,et al.  Wavelet-domain approximation and compression of piecewise smooth images , 2006, IEEE Transactions on Image Processing.

[11]  Wilfried Enkelmann,et al.  Investigations of multigrid algorithms for the estimation of optical flow fields in image sequences , 1988, Comput. Vis. Graph. Image Process..

[12]  V. N. Bogaevski,et al.  Matrix Perturbation Theory , 1991 .

[13]  Leonidas J. Guibas,et al.  Persistence barcodes for shapes , 2004, SGP '04.

[14]  Takeo Kanade,et al.  An Iterative Image Registration Technique with an Application to Stereo Vision , 1981, IJCAI.

[15]  D. Donoho,et al.  Hessian eigenmaps: Locally linear embedding techniques for high-dimensional data , 2003, Proceedings of the National Academy of Sciences of the United States of America.

[16]  C. Grimes New methods in nonlinear dimensionality reduction , 2003 .

[17]  Douglas M. Bates,et al.  Nonlinear Regression Analysis and Its Applications , 1988 .

[18]  M. Turk,et al.  Eigenfaces for Recognition , 1991, Journal of Cognitive Neuroscience.

[19]  Amir Averbuch,et al.  Fast motion estimation using bidirectional gradient methods , 2004, IEEE Transactions on Image Processing.

[20]  Eero P. Simoncelli Bayesian multi-scale differential optical flow , 1999 .

[21]  Michal Irani,et al.  Improving resolution by image registration , 1991, CVGIP Graph. Model. Image Process..