Image Morphing in Deep Feature Spaces: Theory and Applications

This paper combines image metamorphosis with deep features. To this end, images are considered as maps into a high-dimensional feature space and a structure-sensitive, anisotropic flow regularization is incorporated in the metamorphosis model proposed by Miller and Younes (Int J Comput Vis 41(1):61–84, 2001) and Trouvé and Younes (Found Comput Math 5(2):173–198, 2005). For this model, a variational time discretization of the Riemannian path energy is presented and the existence of discrete geodesic paths minimizing this energy is demonstrated. Furthermore, convergence of discrete geodesic paths to geodesic paths in the time continuous model is investigated. The spatial discretization is based on a finite difference approximation in image space and a stable spline approximation in deformation space; the fully discrete model is optimized using the iPALM algorithm. Numerical experiments indicate that the incorporation of semantic deep features is superior to intensity-based approaches.

[1]  Alain Trouvé,et al.  Metamorphoses Through Lie Group Action , 2005, Found. Comput. Math..

[2]  Thomas Pock,et al.  Inertial Proximal Alternating Linearized Minimization (iPALM) for Nonconvex and Nonsmooth Problems , 2016, SIAM J. Imaging Sci..

[3]  Alain Trouvé,et al.  Computing Large Deformation Metric Mappings via Geodesic Flows of Diffeomorphisms , 2005, International Journal of Computer Vision.

[4]  M. Rumpf,et al.  Variational time discretization of geodesic calculus , 2012, 1210.2097.

[5]  Alain Trouvé,et al.  Metamorphoses of Functional Shapes in Sobolev Spaces , 2016, Foundations of Computational Mathematics.

[6]  Benjamin Berkels,et al.  Time Discrete Geodesic Paths in the Space of Images , 2015, SIAM J. Imaging Sci..

[7]  F. Santambrogio,et al.  Extension to BV functions of the large deformation diffeomorphisms matching approach , 2009 .

[8]  L. Younes,et al.  Computing metamorphoses between discrete measures , 2013 .

[9]  Jitendra Malik,et al.  Scale-Space and Edge Detection Using Anisotropic Diffusion , 1990, IEEE Trans. Pattern Anal. Mach. Intell..

[10]  L. Younes,et al.  On the metrics and euler-lagrange equations of computational anatomy. , 2002, Annual review of biomedical engineering.

[11]  Michael I. Miller,et al.  Group Actions, Homeomorphisms, and Matching: A General Framework , 2004, International Journal of Computer Vision.

[12]  Michael I. Miller,et al.  Landmark matching via large deformation diffeomorphisms , 2000, IEEE Trans. Image Process..

[13]  Laurent Younes,et al.  Metamorphosis of images in reproducing kernel Hilbert spaces , 2014, Adv. Comput. Math..

[14]  Martin Rumpf,et al.  Time Discrete Geodesics in Deep Feature Spaces for Image Morphing , 2019, SSVM.

[15]  Martin Rumpf,et al.  A Variational Approach to Nonrigid Morphological Image Registration , 2004, SIAM J. Appl. Math..

[16]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[17]  Paul Dupuis,et al.  Variational problems on ows of di eomorphisms for image matching , 1998 .

[18]  G. D. Maso,et al.  An Introduction to-convergence , 1993 .

[19]  J. Ball.  Global invertibility of Sobolev functions and the interpenetration of matter , 1981, Proceedings of the Royal Society of Edinburgh: Section A Mathematics.

[20]  J. Necas,et al.  Multipolar viscous fluids , 1991 .

[21]  U. Mosco.  Convergence of convex sets and of solutions of variational inequalities , 1969 .

[22]  Alain Trouvé,et al.  Hamiltonian Systems and Optimal Control in Computational Anatomy: 100 Years Since D'Arcy Thompson. , 2015, Annual review of biomedical engineering.

[23]  V. Arnold.  Sur la géométrie différentielle des groupes de Lie de dimension infinie et ses applications à l'hydrodynamique des fluides parfaits , 1966 .

[24]  Gabriele Steidl,et al.  Morphing of Manifold-Valued Images Inspired by Discrete Geodesics in Image Spaces , 2017, SIAM J. Imaging Sci..

[25]  P. G. Ciarlet,et al.  Three-dimensional elasticity , 1988 .

[26]  Martin Burger,et al.  A Hyperelastic Regularization Energy for Image Registration , 2013, SIAM J. Sci. Comput..

[27]  Alain Trouvé,et al.  Local Geometry of Deformable Templates , 2005, SIAM J. Math. Anal..

[28]  Louis Nirenberg,et al.  An extended interpolation inequality , 1966 .

[29]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[30]  L. Younes.  Shapes and Diffeomorphisms , 2010 .

[31]  Alexander Effland.  Discrete Riemannian Calculus and A Posteriori Error Control on Shape Spaces , 2017 .

[32]  Martin Rumpf,et al.  Convergence of the Time Discrete Metamorphosis Model on Hadamard Manifolds , 2019, SIAM J. Imaging Sci..

[33]  S. Gillen.  In the public domain. , 2012, Nursing standard (Royal College of Nursing (Great Britain) : 1987).

[34]  Daniel Rueckert,et al.  Diffeomorphic 3D Image Registration via Geodesic Shooting Using an Efficient Adjoint Calculation , 2011, International Journal of Computer Vision.