Predictive Online Optimisation with Applications to Optical Flow

Online optimisation revolves around new data being introduced into a problem while it is still being solved; think of deep learning as more training samples become available. We adapt the idea to dynamic inverse problems such as video processing with optical flow. We introduce a corresponding predictive online primal-dual proximal splitting method. The video frames now exactly correspond to the algorithm iterations . A user-prescribed predictor describes the evolution of the primal variable. To prove convergence we need a predictor for the dual variable based on (proximal) gradient flow. This affects the model that the method asymptotically minimises. We show that for inverse problems the effect is, essentially, to construct a new dynamic regulariser based on infimal convolution of the static regularisers with the temporal coupling. We finish by demonstrating excellent real-time performance of our method in computational image stabilisation and convergence in terms of regularisation theory.

[1]  Antonin Chambolle,et al.  A First-Order Primal-Dual Algorithm for Convex Problems with Applications to Imaging , 2011, Journal of Mathematical Imaging and Vision.

[2]  Marius Tico.  Digital Image Stabilization , 2009 .

[3]  ANTONIN CHAMBOLLE,et al.  An Algorithm for Total Variation Minimization and Applications , 2004, Journal of Mathematical Imaging and Vision.

[4]  Joachim Weickert,et al.  Variational Optic Flow Computation with a Spatio-Temporal Smoothness Constraint , 2001, Journal of Mathematical Imaging and Vision.

[5]  Elad Hazan,et al.  Introduction to Online Convex Optimization , 2016, Found. Trends Optim..

[6]  T. Valkonen.  Predictive Online Optimisation with Applications to Optical Flow , 2020, Journal of Mathematical Imaging and Vision.

[7]  Marc Teboulle,et al.  A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems , 2009, SIAM J. Imaging Sci..

[8]  Bernd Neumann,et al.  Optical flow , 1986, Workshop on Motion.

[9]  Luca Sanguinetti,et al.  Online convex optimization and no-regret learning: Algorithms, guarantees and applications , 2018, ArXiv.

[10]  Léon Bottou,et al.  The Tradeoffs of Large Scale Learning , 2007, NIPS.

[11]  Barbara Kaltenbacher,et al.  Minimization Based Formulations of Inverse Problems and Their Regularization , 2018, SIAM J. Optim..

[12]  Aaas News,et al.  Book Reviews , 1893, Buffalo Medical and Surgical Journal.

[13]  Rajiv Mehrotra,et al.  A trajectory-based computational model for optical flow estimation , 1995, IEEE Trans. Robotics Autom..

[14]  Tuomo Valkonen,et al.  Testing and Non-linear Preconditioning of the Proximal Point Method , 2017, Applied Mathematics & Optimization.

[15]  Bingsheng He,et al.  Convergence Analysis of Primal-Dual Algorithms for a Saddle-Point Problem: From Contraction Perspective , 2012, SIAM J. Imaging Sci..

[16]  M. Nikolova.  An Algorithm for Total Variation Minimization and Applications , 2004 .

[17]  Alan Edelman,et al.  Julia: A Fresh Approach to Numerical Computing , 2014, SIAM Rev..

[18]  H. Engl,et al.  Regularization of Inverse Problems , 1996 .

[20]  Ruggero Carli,et al.  Prediction-Correction Splittings for Time-Varying Optimization With Intermittent Observations , 2020, IEEE Control Systems Letters.

[21]  Clemens Kirisits,et al.  Convective regularization for optical flow , 2015, ArXiv.

[22]  Berthold K. P. Horn,et al.  Determining Optical Flow , 1981, Other Conferences.

[23]  T. Valkonen.  Transport equation and image interpolation with SBD velocity fields , 2011 .

[24]  Martin Grötschel,et al.  Online optimization of large scale systems , 2001 .

[25]  Agustín Salgado,et al.  Temporal constraints in large optical flow estimation , 2007 .

[26]  Dirk A. Lorenz,et al.  Image Sequence Interpolation Based on Optical Flow, Segmentation, and Optimal Control , 2012, IEEE Transactions on Image Processing.

[27]  Henning Zimmer,et al.  Modeling temporal coherence for optical flow , 2011, 2011 International Conference on Computer Vision.

[28]  Francesco Orabona.  A Modern Introduction to Online Learning , 2019, ArXiv.

[29]  Hans-Hellmut Nagel,et al.  Extending the 'Oriented Smoothness Constraint' into the Temporal Domain and the Estimation of Derivatives of Optical Flow , 1990, ECCV.

[30]  Tuomo Valkonen,et al.  Preconditioned proximal point methods and notions of partial subregularity , 2017, 1711.05123.

[31]  Heinz H. Bauschke,et al.  Convex Analysis and Monotone Operator Theory in Hilbert Spaces , 2011, CMS Books in Mathematics.

[32]  Andrea Simonetto.  Time-Varying Convex Optimization via Time-Varying Averaged Operators , 2017, 1704.07338.

[33]  Hans-Hellmut Nagel,et al.  An Investigation of Smoothness Constraints for the Estimation of Displacement Vector Fields from Image Sequences , 1983, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[34]  M. Heinkenschloss,et al.  Real-Time PDE-Constrained Optimization , 2007 .

[35]  Rebecca Willett,et al.  Dynamical Models and tracking regret in online convex programming , 2013, ICML.