Advanced Real-Time Manipulation of Video Streams

Diminished Reality is a new fascinating technology that removes real-world content from live video streams. This sensational live video manipulation actually removes real objects and generates a coherent video stream in real-time. Viewers cannot detect modified content. Existing approaches are restricted to moving objects and static or almost static cameras and do not allow real-time manipulation of video content. Jan Herling presents a new and innovative approach for real-time object removal with arbitrary camera movements.

[1]  V. Lepetit,et al.  An intuitive tool for outlining objetcs in video sequences: applications to augmented and diminished reality , 2001 .

[2]  Patrick Pérez,et al.  Region filling and object removal by exemplar-based image inpainting , 2004, IEEE Transactions on Image Processing.

[3]  Allen Gersho,et al.  Vector quantization and signal compression , 1991, The Kluwer international series in engineering and computer science.

[4]  Jenn-Jier James Lien,et al.  Rapid Image Completion System Using Multiresolution Patch-Based Directional and Nondirectional Approaches , 2009, IEEE Transactions on Image Processing.

[5]  Eli Shechtman,et al.  Image melding , 2012, ACM Trans. Graph..

[6]  William H. Press,et al.  Numerical Recipes 3rd Edition: The Art of Scientific Computing , 2007 .

[7]  Ronald Azuma,et al.  A Survey of Augmented Reality , 1997, Presence: Teleoperators & Virtual Environments.

[8]  Wolfgang Broll,et al.  Advanced self-contained object removal for realizing real-time Diminished Reality in unconstrained environments , 2010, 2010 IEEE International Symposium on Mixed and Augmented Reality.

[9]  David W. Murray,et al.  Parallel Tracking and Mapping on a camera phone , 2009, 2009 8th IEEE International Symposium on Mixed and Augmented Reality.

[10]  Hideo Saito,et al.  Diminished reality using plane-sweep algorithm with weakly-calibrated cameras , 2010 .

[11]  Ming Liu,et al.  Video completion via motion guided spatial-temporal global optimization , 2009, ACM Multimedia.

[12]  Demetri Terzopoulos,et al.  Snakes: Active contour models , 2004, International Journal of Computer Vision.

[13]  P. Anandan,et al.  Hierarchical Model-Based Motion Estimation , 1992, ECCV.

[14]  Matthijs C. Dorst.  Distinctive Image Features from Scale-Invariant Keypoints , 2011 .

[15]  Denis Simakov,et al.  Summarizing visual data using bidirectional similarity , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[16]  Alexei A. Efros,et al.  Image quilting for texture synthesis and transfer , 2001, SIGGRAPH.

[17]  Wolfgang Broll,et al.  Interactive Bits: Prototyping of Mixed Reality Applications and Interaction Techniques through Visual Programming , 2008, 2008 IEEE Symposium on 3D User Interfaces.

[18]  Dani Lischinski,et al.  Gradient Domain High Dynamic Range Compression , 2023 .

[19]  Baining Guo,et al.  Chaos Mosaic: Fast and Memory Efficient Texture Synthesis , 2000 .

[20]  Guillermo Sapiro,et al.  A Comprehensive Framework for Image Inpainting , 2010, IEEE Transactions on Image Processing.

[21]  Nassir Navab,et al.  Multiview paraperspective projection model for diminished reality , 2003, The Second IEEE and ACM International Symposium on Mixed and Augmented Reality, 2003. Proceedings..

[22]  Li-Yi Wei,et al.  Deterministic texture analysis and synthesis using tree structure vector quantization , 1999, XII Brazilian Symposium on Computer Graphics and Image Processing (Cat. No.PR00481).

[23]  Zongben Xu,et al.  Image Inpainting by Patch Propagation Using Patch Sparsity , 2010, IEEE Transactions on Image Processing.

[24]  Xiaochun Cao,et al.  Video Completion for Perspective Camera Under Constrained Motion , 2006, 18th International Conference on Pattern Recognition (ICPR'06).

[25]  Tom Drummond,et al.  Fusing points and lines for high performance tracking , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[26]  Tai-Pang Wu,et al.  Video repairing under variable illumination using cyclic motions , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[27]  Eli Shechtman,et al.  Space-Time Completion of Video , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[28]  Takashi Shibata,et al.  Image Inpainting Based on Probabilistic Structure Estimation , 2010, ACCV.

[29]  Dieter Schmalstieg,et al.  Pose tracking from natural features on mobile phones , 2008, 2008 7th IEEE/ACM International Symposium on Mixed and Augmented Reality.

[30]  Jack Bresenham,et al.  Algorithm for computer control of a digital plotter , 1965, IBM Syst. J..

[31]  Daniel Cohen-Or,et al.  Fragment-based image completion , 2003, ACM Trans. Graph..

[32]  Ryuuki Sakamoto,et al.  Diminished Reality for Landscape Video Sequences with Homographies , 2010, KES.

[33]  Steve Mann,et al.  Mediated Reality , 1999 .

[34]  Robert C. Bolles,et al.  Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography , 1981, CACM.

[35]  Ivan Poupyrev,et al.  The MagicBook: a transitional AR interface , 2001, Comput. Graph..

[36]  Guillermo Sapiro,et al.  Video Inpainting Under Constrained Camera Motion , 2007, IEEE Transactions on Image Processing.

[37]  Charlie C. L. Wang,et al.  Gradient based image completion by solving the Poisson equation , 2007, Comput. Graph..

[38]  Dana H. Ballard,et al.  Generalizing the Hough transform to detect arbitrary shapes , 1981, Pattern Recognit..

[39]  Guillermo Sapiro,et al.  Image inpainting , 2000, SIGGRAPH.

[40]  Wolfgang Broll,et al.  Markerless Tracking for Augmented Reality , 2011, Handbook of Augmented Reality.

[41]  Charless C. Fowlkes,et al.  Contour Detection and Hierarchical Image Segmentation , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[42]  Andrew Blake,et al.  "GrabCut" , 2004, ACM Trans. Graph..

[43]  Wolfgang Broll.  Interacting in distributed collaborative virtual environments , 1995, Proceedings Virtual Reality Annual International Symposium '95.

[44]  Luc Van Gool,et al.  Speeded-Up Robust Features (SURF) , 2008, Comput. Vis. Image Underst..

[45]  Charlie C. L. Wang,et al.  Fast Query for Exemplar-Based Image Completion , 2010, IEEE Transactions on Image Processing.

[46]  Harry Shum,et al.  Image completion with structure propagation , 2005, ACM Trans. Graph..

[47]  Alexei A. Efros,et al.  Scene completion using millions of photographs , 2008, Commun. ACM.

[48]  Alexander Wong,et al.  A nonlocal-means approach to exemplar-based inpainting , 2008, 2008 15th IEEE International Conference on Image Processing.

[49]  Yun Zhang,et al.  Video Brush: A Novel Interface for Efficient Video Cutout , 2011, Comput. Graph. Forum.

[50]  Michael Ashikhmin,et al.  Synthesizing natural textures , 2001, I3D '01.

[51]  Wolfgang Broll,et al.  An adaptive training-free feature tracker for mobile phones , 2010, VRST '10.

[52]  Sung Yong Shin,et al.  On pixel-based texture synthesis by non-parametric sampling , 2006, Comput. Graph..

[53]  Jian Zhao,et al.  Efficient Object-Based Video Inpainting , 2006, 2006 International Conference on Image Processing.

[54]  Eli Shechtman,et al.  PatchMatch: a randomized correspondence algorithm for structural image editing , 2009, ACM Trans. Graph..

[55]  Assaf Zomet,et al.  Learning how to inpaint from global image statistics , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[56]  Louis Laborelli,et al.  Missing data correction in still images and image sequences , 2002, MULTIMEDIA '02.

[57]  Mubarak Shah,et al.  Region Completion in a Single Image , 2004, Eurographics.

[58]  Adam Finkelstein,et al.  The Generalized PatchMatch Correspondence Algorithm , 2010, ECCV.

[59]  Yael Pritch,et al.  Shift-map image editing , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[60]  T. Chan,et al.  Image inpainting by correspondence maps: A deterministic approach , 2003 .

[61]  Nezam Mahdavi-Amiri,et al.  Structure and texture image inpainting , 2010, 2010 International Conference on Signal and Image Processing.

[62]  Mark Horowitz,et al.  CPU DB: Recording Microprocessor History , 2012, ACM Queue.

[63]  Wonjun Lee,et al.  Video inpainting and implant via diversified temporal continuations , 2006, MM '06.

[64]  Thomas Wiegand,et al.  Automatic Structure-Aware Inpainting for Complex Image Content , 2008, ISVC.

[65]  J. Laurie Snell,et al.  Markov Random Fields and Their Applications , 1980 .

[66]  Tai-Pang Wu,et al.  Video repairing: inference of foreground and background under severe occlusion , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[67]  Jian Sun,et al.  Video object cut and paste , 2005, SIGGRAPH 2005.

[68]  Nikos Komodakis,et al.  Image Completion Using Efficient Belief Propagation Via Priority Scheduling and Dynamic Pruning , 2007, IEEE Transactions on Image Processing.

[69]  Jan Borgosz,et al.  Maximum Disparity Threshold Estimation for Stereo Imaging Systems via Variogram Analysis , 2003, International Conference on Computational Science.

[70]  Guillermo Sapiro,et al.  Video SnapCut: robust video object cutout using localized classifiers , 2009, ACM Trans. Graph..

[71]  H. Saito,et al.  Diminished Reality using Multiple Handheld Cameras , 2007 .

[72]  G. Klein,et al.  Parallel Tracking and Mapping for Small AR Workspaces , 2007, 2007 6th IEEE and ACM International Symposium on Mixed and Augmented Reality.

[73]  Patrick Pérez,et al.  Poisson image editing , 2003, ACM Trans. Graph..

[74]  Jian Zhao,et al.  Efficient Object-Based Video Inpainting , 2006, ICIP.

[75]  Marc Levoy,et al.  Fast texture synthesis using tree-structured vector quantization , 2000, SIGGRAPH.

[76]  Hanno Scharr,et al.  Optimal operators in digital image processing , 2000 .

[77]  Christopher G. Harris,et al.  A Combined Corner and Edge Detector , 1988, Alvey Vision Conference.

[78]  Jenq-Neng Hwang,et al.  Exemplar-Based Video Inpainting Without Ghost Shadow Artifacts by Maintaining Temporal Continuity , 2009, IEEE Transactions on Circuits and Systems for Video Technology.

[79]  Miika Aittala,et al.  Light-weight marker hiding for augmented reality , 2010, 2010 IEEE International Symposium on Mixed and Augmented Reality.

[80]  Naokazu Yokoya,et al.  Image Inpainting Considering Brightness Change and Spatial Locality of Textures , 2009, VISAPP.

[81]  Henry Been-Lirn Duh,et al.  Trends in augmented reality tracking, interaction and display: A review of ten years of ISMAR , 2008, 2008 7th IEEE/ACM International Symposium on Mixed and Augmented Reality.

[82]  Kenneth Levenberg.  A METHOD FOR THE SOLUTION OF CERTAIN NON – LINEAR PROBLEMS IN LEAST SQUARES , 1944 .

[83]  Fumio Kishino,et al.  Augmented reality: a class of displays on the reality-virtuality continuum , 1995, Other Conferences.

[84]  Stormy Attaway.  Matlab, Second Edition: A Practical Introduction to Programming and Problem Solving , 2011 .

[85]  Patrick Pérez,et al.  Object removal by exemplar-based inpainting , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[86]  Dani Lischinski,et al.  A Closed-Form Solution to Natural Image Matting , 2008 .

[87]  Wolfgang Broll,et al.  Supporting Reusability of VR and AR Interface Elements and Interaction Techniques , 2009, HCI.

[88]  Wolfgang Broll,et al.  PixMix: A real-time approach to high-quality Diminished Reality , 2012, 2012 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[89]  Sanni Siltanen,et al.  Texture generation over the marker area , 2006, 2006 IEEE/ACM International Symposium on Mixed and Augmented Reality.

[90]  Bernd Jähne,et al.  Digital image processing (3rd ed.): concepts, algorithms, and scientific applications , 1995 .

[91]  Hirokazu Kato,et al.  Marker tracking and HMD calibration for a video-based augmented reality conferencing system , 1999, Proceedings 2nd IEEE and ACM International Workshop on Augmented Reality (IWAR'99).

[92]  Wolfgang Broll,et al.  High-Quality Real-Time Video Inpaintingwith PixMix , 2014, IEEE Transactions on Visualization and Computer Graphics.

[93]  Chun Chen,et al.  Video completion based on improved belief propagation , 2006 .

[94]  S. Avidan,et al.  Seam carving for content-aware image resizing , 2007, SIGGRAPH 2007.

[95]  Wolfgang Broll,et al.  Random model variation for universal feature tracking , 2012, VRST '12.

[96]  Chi-Keung Tang,et al.  Image repairing: robust image synthesis by adaptive ND tensor voting , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..