Variational method for super-resolution optical flow

The motion fields in an image sequence observed by a car-mounted imaging system depend on the positions in the imaging plane. Since the motion displacements in the regions close to the camera centre are small, for accurate optical flow computation in this region, we are required to use super-resolution of optical flow fields. We develop an algorithm for super-resolution optical flow computation. Super-resolution of images is a technique for recovering a high-resolution image from a low-resolution image and/or image sequence. Optical flow is the appearance motion of points on the image. Therefore, super-resolution optical flow computation yields the appearance motion of each point on the high-resolution image from a sequence of low-resolution images. We combine variational super-resolution and variational optical flow computation in super-resolution optical flow computation. Our method directly computes the gradient and spatial difference of high-resolution images from those of low-resolution images, without computing any high-resolution images used as intermediate data for the computation of optical flow vectors of the high-resolution image.

[1]  Pascal Frossard,et al.  L1 regularized super-resolution from unregistered omnidirectional images , 2009, 2009 IEEE International Conference on Acoustics, Speech and Signal Processing.

[2]  Robert L. Stevenson,et al.  Linear models for multiframe super-resolution restoration under nonaffine registration and spatially varying PSF , 2004, IS&T/SPIE Electronic Imaging.

[3]  Brendt Wohlberg,et al.  An Iteratively Reweighted Norm Algorithm for Minimization of Total Variation Functionals , 2007, IEEE Signal Processing Letters.

[4]  Truong Q. Nguyen,et al.  Wavelets and filter banks , 1996 .

[5]  Thomas Brox,et al.  Universität Des Saarlandes Fachrichtung 6.1 – Mathematik Highly Accurate Optic Flow Computation with Theoretically Justified Warping Highly Accurate Optic Flow Computation with Theoretically Justified Warping , 2022 .

[6]  Steven S. Beauchemin,et al.  The computation of optical flow , 1995, CSUR.

[7]  A. Hyvärinen,et al.  A multi-layer sparse coding network learns contour coding from natural images , 2002, Vision Research.

[8]  Atsushi Imiya,et al.  An efficient statistical method for subpixel optical flow detection , 2002 .

[9]  H. Köstler,et al.  Robust and efficient multigrid techniques for the optical flow problem using different regularizers , 2005 .

[10]  Nahum Kiryati,et al.  Coarse to over-fine optical flow estimation , 2007, Pattern Recognit..

[11]  A. Bruckstein,et al.  K-SVD : An Algorithm for Designing of Overcomplete Dictionaries for Sparse Representation , 2005 .

[12]  Xin Li,et al.  Super-Resolution for Synthetic Zooming , 2006, EURASIP J. Adv. Signal Process..

[13]  Ron Kimmel,et al.  Demosaicing: Image Reconstruction from Color CCD Samples , 1998, ECCV.

[14]  Michael Elad,et al.  Example-Based Regularization Deployed to Super-Resolution Reconstruction of a Single Image , 2009, Comput. J..

[15]  Subhasis Chaudhuri,et al.  Super-resolution imaging: use of zoom as a cue , 2004, Image Vis. Comput..

[16]  P. Dragotti,et al.  Image super-resolution with B-Spline kernels , 2006 .

[17]  Deepu Rajan,et al.  Generalized interpolation and its application in super-resolution imaging , 2001, Image Vis. Comput..

[18]  Michael Unser,et al.  Splines: a perfect fit for signal and image processing , 1999, IEEE Signal Process. Mag..

[19]  D.G. Tzikas,et al.  The variational approximation for Bayesian inference , 2008, IEEE Signal Processing Magazine.

[20]  Henry Stark,et al.  Image recovery: Theory and application , 1987 .

[21]  Joachim Weickert,et al.  Variational Optic Flow Computation with a Spatio-Temporal Smoothness Constraint , 2001, Journal of Mathematical Imaging and Vision.

[22]  Jing Xu,et al.  Convergence of fixed point iteration for deblurring and denoising problem , 2007, Appl. Math. Comput..

[23]  A. Chambolle.  Practical, Unified, Motion and Missing Data Treatment in Degraded Video , 2004, Journal of Mathematical Imaging and Vision.

[24]  John F. Canny,et al.  A Computational Approach to Edge Detection , 1986, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[25]  Michael Spann,et al.  Robust Optical Flow Computation Based on Least-Median-of-Squares Regression , 1999, International Journal of Computer Vision.

[26]  Michael W. Marcellin,et al.  Iterative multiframe super-resolution algorithms for atmospheric turbulence-degraded imagery , 1998, Proceedings of the 1998 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP '98 (Cat. No.98CH36181).

[27]  Edward H. Adelson,et al.  The Laplacian Pyramid as a Compact Image Code , 1983, IEEE Trans. Commun..

[28]  Michael J. Black,et al.  The Robust Estimation of Multiple Motions: Parametric and Piecewise-Smooth Flow Fields , 1996, Comput. Vis. Image Underst..

[29]  Jian-Feng Cai,et al.  Blind motion deblurring from a single image using sparse approximation , 2009, CVPR.

[30]  Michael J. Black,et al.  On the Spatial Statistics of Optical Flow , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[31]  Mrinal K. Mandal,et al.  Efficient Hodge-Helmholtz decomposition of motion fields , 2005, Pattern Recognit. Lett..

[32]  Denis Hamad,et al.  Optic flow estimation by support vector regression , 2006, Eng. Appl. Artif. Intell..

[33]  Sebastian Thrun,et al.  Reverse Optical Flow for Self-Supervised Adaptive Autonomous Robot Navigation , 2007, International Journal of Computer Vision.

[34]  Mark W. Schmidt,et al.  Fast Optimization Methods for L1 Regularization: A Comparative Study and Two New Approaches , 2007, ECML.

[35]  Masatoshi Okutomi,et al.  Direct super-resolution and registration using raw CFA images , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[36]  Horst Bischof,et al.  A Duality Based Approach for Realtime TV-L1 Optical Flow , 2007, DAGM-Symposium.

[37]  Zhaozhong Wang,et al.  Analysis of multiframe super-resolution reconstruction for image anti-aliasing and deblurring , 2005, Image Vis. Comput..

[38]  Atsushi Imiya,et al.  Two Step Variational Method for Subpixel Optical Flow Computation , 2009, ISVC.

[39]  T. Brox,et al.  Variational Optic Flow Computation: From Continuous Models to Algorithms , 2003 .

[40]  ANTONIN CHAMBOLLE,et al.  An Algorithm for Total Variation Minimization and Applications , 2004, Journal of Mathematical Imaging and Vision.

[41]  Stephen J. Roberts,et al.  Bayesian Methods for Image Super-Resolution , 2009, Comput. J..

[42]  J. D. van Ouwerkerk,et al.  Image super-resolution survey , 2006, Image Vis. Comput..

[43]  Jean-François Aujol,et al.  Some First-Order Algorithms for Total Variation Based Image Restoration , 2009, Journal of Mathematical Imaging and Vision.

[44]  Sang Uk Lee,et al.  A hierarchical optical flow estimation algorithm based on the interlevel motion smoothness constraint , 1993, Pattern Recognit..

[45]  L. Rudin,et al.  Nonlinear total variation based noise removal algorithms , 1992 .

[46]  A. Kirsch.  An Introduction to the Mathematical Theory of Inverse Problems , 1996, Applied Mathematical Sciences.

[47]  G. Wahba,et al.  Some New Mathematical Methods for Variational Objective Analysis Using Splines and Cross Validation , 1980 .

[48]  Brendt Wohlberg,et al.  Efficient Minimization Method for a Generalized Total Variation Functional , 2009, IEEE Transactions on Image Processing.

[49]  Timo Kohlberger,et al.  Variational optical flow estimation for particle image velocimetry , 2005 .

[50]  Luc Van Gool,et al.  A Probabilistic Approach to Optical Flow based Super-Resolution , 2004, 2004 Conference on Computer Vision and Pattern Recognition Workshop.

[51]  Andrea Giachetti,et al.  The use of optical flow for road navigation , 1998, IEEE Trans. Robotics Autom..

[52]  D. Donoho.  For most large underdetermined systems of equations, the minimal 𝓁1‐norm near‐solution approximates the sparsest near‐solution , 2006 .

[53]  Vasileios Argyriou,et al.  A Study of Sub-pixel Motion Estimation using Phase Correlation , 2006, BMVC.

[54]  Tapio Saramäki,et al.  Edge-preserving image resizing using modified B-splines , 2001, 2001 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings (Cat. No.01CH37221).

[55]  Jing Yuan,et al.  Simultaneous Higher-Order Optical Flow Estimation and Decomposition , 2007, SIAM J. Sci. Comput..

[56]  Luis Ángel Ruiz Fernández,et al.  Non-linear fourth-order image interpolation for subpixel edge detection and localization , 2008, Image Vis. Comput..

[57]  Youji Iiguni,et al.  Image restoration from a downsampled image by using the DCT , 2007, Signal Process..

[58]  Fermín S. Viloche Bazán,et al.  Determining the regularization parameters for super-resolution problems , 2008, Signal Process..

[59]  Berthold K. P. Horn,et al.  Determining Optical Flow , 1981, Other Conferences.

[60]  Gilbert Strang,et al.  Computational Science and Engineering , 2007 .

[61]  Guillermo Sapiro,et al.  A Variational Framework for Simultaneous Motion Estimation and Restoration of Motion-Blurred Video , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[62]  David Suter,et al.  Robust Optic Flow Computation , 1998, International Journal of Computer Vision.

[63]  Gang Liu,et al.  Cardiac video analysis using Hodge-Helmholtz field decomposition , 2006, Comput. Biol. Medicine.

[64]  Rutuparna Panda,et al.  Least squares generalized B-spline signal and image processing , 2001, Signal Process..

[65]  Michael Unser,et al.  Image interpolation and resampling , 2000 .

[66]  Weiqi Jin,et al.  A subpixel motion estimation algorithm based on digital correlation for illumination variant and noise image sequences , 2009 .

[67]  Atsushi Imiya,et al.  Decomposition and Construction of Neighbourhood Operations Using Linear Algebra , 2008, DGCI.

[68]  Jan Modersitzki,et al.  Numerical Methods for Image Registration , 2004 .

[69]  Martin Vetterli,et al.  Subspace-based methods for image registration and super-resolution , 2008, 2008 15th IEEE International Conference on Image Processing.

[70]  Albert Tarantola,et al.  Inverse problem theory - and methods for model parameter estimation , 2004 .

[71]  J. Modersitzki,et al.  A unified approach to fast image registration and a new curvature based registration technique , 2004 .

[72]  Anup Basu,et al.  Applications of the Discrete Hodge Helmholtz Decomposition to Image and Video Processing , 2005, PReMI.

[73]  H. Enomoto,et al.  Application of Structure Lines to Surface Construction and 3-Dimensional Analysis , 1982 .

[74]  Laurent Najman,et al.  Watershed of a continuous function , 1994, Signal Process..

[75]  Michael W. Marcellin,et al.  Iterative multiframe superresolution algorithms for atmospheric-turbulence-degraded imagery , 1998 .

[76]  Huei-Yung Lin,et al.  Motion blur removal and its application to vehicle speed detection , 2004, 2004 International Conference on Image Processing, 2004. ICIP '04..

[77]  David J. Fleet,et al.  Performance of optical flow techniques , 1994, International Journal of Computer Vision.

[78]  M. Elad,et al.  $rm K$-SVD: An Algorithm for Designing Overcomplete Dictionaries for Sparse Representation , 2006, IEEE Transactions on Signal Processing.

[79]  Stanley Osher,et al.  Image Super-Resolution by TV-Regularization and Bregman Iteration , 2008, J. Sci. Comput..

[80]  Takahiro Saito,et al.  Interframe Motion Deblurring using Spatio-Temporal Regularization , 2007, 2007 IEEE International Conference on Image Processing.

[81]  Jian-Feng Cai,et al.  Convergence of the linearized Bregman iteration for ℓ1-norm minimization , 2009, Math. Comput..

[82]  William T. Freeman,et al.  Example-Based Super-Resolution , 2002, IEEE Computer Graphics and Applications.

[83]  Atsushi Imiya,et al.  Formal Polynomials for Image Processing , 1997 .

[84]  François Malgouyres,et al.  Edge Direction Preserving Image Zooming: A Mathematical and Numerical Analysis , 2001, SIAM J. Numer. Anal..

[85]  Mila Nikolova,et al.  Efficient Minimization Methods of Mixed l2-l1 and l1-l1 Norms for Image Restoration , 2005, SIAM J. Sci. Comput..

[86]  Paulo Jorge S. G. Ferreira.  Interpolation and the discrete Papoulis-Gerchberg algorithm , 1994, IEEE Trans. Signal Process..

[87]  Warren M. Kruegger,et al.  The Geometry of Differential Operators with Application to Image Processing , 1989, IEEE Trans. Pattern Anal. Mach. Intell..

[88]  J. Weickert.  Applications of nonlinear diffusion in image processing and computer vision , 2000 .

[89]  B. Chalmond.  Modeling and inverse problems in image analysis , 2003 .

[90]  James Demmel,et al.  Applied Numerical Linear Algebra , 1997 .

[91]  Michael K. Ng,et al.  Super‐resolution image reconstruction using multisensors , 2005, Numer. Linear Algebra Appl..

[92]  Horst Bischof,et al.  A Duality Based Algorithm for TV- L 1-Optical-Flow Image Registration , 2007, MICCAI.

[93]  Dante C. Youla,et al.  Generalized Image Restoration by the Method of Alternating Orthogonal Projections , 1978 .

[94]  Yihong Gong,et al.  Resolution enhancement based on learning the sparse association of image patches , 2010, Pattern Recognit. Lett..