Time-of-Flight Sensors in Computer Graphics

A growing number of applications depend on accurate and fast 3D scene analysis. Examples are model and lightfield acquisition, collision prevention, mixed reality, and gesture recognition. The estimation of a range map by image analysis or laser scan techniques is still a time-consuming and expensive part of such systems. A lower-priced, fast and robust alternative for distance measurements are Time-of-Flight (ToF) cameras. Recently, significant advances have been made in producing low-cost and compact ToF-devices, which have the potential to revolutionize many fields of research, including Computer Graphics, Computer Vision and Human Machine Interaction (HMI). These technologies are starting to have an impact on research and commercial applications. The upcoming generation of ToF sensors, however, will be even more powerful and will have the potential to become “ubiquitous real-time geometry devices” for gaming, web-conferencing, and numerous other applications. This STAR gives an account of recent developments in ToF-technology and discusses the current state of the integration of this technology into various graphics-related applications.

[1]  B. Huhle,et al.  Self-Localization in Scanned 3DTV Sets , 2007, 2007 3DTV Conference.

[2]  D. Falie,et al.  Measurements with ToF Cameras and Their Necessary Corrections , 2007, 2007 International Symposium on Signals, Circuits and Systems.

[3]  Ryan Crabb,et al.  Real-time foreground segmentation via range and color imaging , 2008, 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[4]  Thomas B. Moeslund,et al.  View invariant gesture recognition using the CSEM SwissRanger SR-2 camera , 2008, Int. J. Intell. Syst. Technol. Appl..

[5]  Andreas Kolb,et al.  New insights into the calibration of ToF-sensors , 2008, 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[6]  D. Falie,et al.  Respiratory motion visualization and the sleep apnea diagnosis with the time of flight (ToF) camera , 2008 .

[7]  Sebastian Thrun,et al.  An Application of Markov Random Fields to Range Sensing , 2005, NIPS.

[8]  K. Hartmann,et al.  Data-Fusion of PMD-Based Distance-Information and High-Resolution RGB-Images , 2007, 2007 International Symposium on Signals, Circuits and Systems.

[9]  Reinhard Koch,et al.  Calibration of focal length and 3D pose based on the reflectance and depth image of a planar object , 2008, Int. J. Intell. Syst. Technol. Appl..

[10]  Timo Schairer,et al.  Robust non-local denoising of colored depth data , 2008, 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[11]  Rasmus Larsen,et al.  Cluster tracking with Time-of-Flight cameras , 2008, 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[12]  Sebastian Thrun,et al.  High-quality scanning using time-of-flight depth superresolution , 2008, 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[13]  D. Falie,et al.  Noise Characteristics of 3D Time-of-Flight Cameras , 2007, 2007 International Symposium on Signals, Circuits and Systems.

[14]  Holger H. Rapp,et al.  Experimental and Theoretical Investigation of Correlating TOF-Camera Systems , 2007 .

[15]  Svenska Sällskapet för Automatiserad Bildanalys.  Proceedings of the ... Scandinavian Conference on Image Analysis , 1980 .

[16]  Thomas Martinetz,et al.  Deictic Gestures with a Time-of-Flight Camera , 2009, Gesture Workshop.

[17]  Thomas Martinetz,et al.  Guiding Eye Movements for Better Communication and Augmented Vision , 2006, PIT.

[18]  Stefan May,et al.  Calibration and registration for precise surface reconstruction with Time-Of-Flight cameras , 2008, Int. J. Intell. Syst. Technol. Appl..

[19]  V. Peters,et al.  A Simulation Framework for Time-Of-Flight Sensors , 2007, 2007 International Symposium on Signals, Circuits and Systems.

[20]  Andreas Kolb,et al.  GPU‐Based Spherical Light Field Rendering with Per‐Fragment Depth Correction , 2008, Comput. Graph. Forum.

[21]  Fabio Remondino,et al.  Range imaging technology: new developments and applications for people identification and tracking , 2007, Electronic Imaging.

[22]  Eyal Ofek,et al.  Depth keying , 2003, IS&T/SPIE Electronic Imaging.

[23]  A. Kolb,et al.  Raycasting of Light Field Galleries from Volumetric Data , 2008, Comput. Graph. Forum.

[24]  Gerd Hirzinger,et al.  Extrinsic and depth calibration of ToF-cameras , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[25]  R. Koch,et al.  CALIBRATION OF A PMD-CAMERA USING A PLANAR CALIBRATION PATTERN TOGETHER WITH A MULTI-CAMERA SETUP , 2008 .

[26]  Andreas Kolb,et al.  Calibration of the intensity-related distance error of the PMD TOF-camera , 2007, SPIE Optics East.

[27]  Young Min Kim,et al.  Design and calibration of a multi-view TOF sensor fusion system , 2008, 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[28]  Wolfgang Straßer,et al.  On-the-fly scene acquisition with a handy multi-sensor system , 2008, Int. J. Intell. Syst. Technol. Appl..

[29]  Reinhard Koch,et al.  Robust GPU-assisted camera tracking using free-form surface models , 2007, Journal of Real-Time Image Processing.

[30]  Thomas Martinetz,et al.  Scale-invariant range features for time-of-flight camera applications , 2008, 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[31]  Richard Szeliski,et al.  The lumigraph , 1996, SIGGRAPH.

[32]  Reinhard Koch,et al.  Pose Estimation and Map Building with a PMD-Camera for Robot Navigation , 2007 .

[33]  Andreas Kolb,et al.  Lateral and Depth Calibration of PMD-Distance Sensors , 2006, ISVC.

[34]  Harry Shum,et al.  Plenoptic sampling , 2000, SIGGRAPH.

[35]  Ruigang Yang,et al.  Fusion of time-of-flight depth and stereo for high accuracy depth maps , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[36]  Joachim Hornegger,et al.  Robust real-time 3D time-of-flight based gesture navigation , 2008, 2008 8th IEEE International Conference on Automatic Face & Gesture Recognition.

[37]  Joachim Hornegger,et al.  3-D gesture-based scene navigation in medical imaging applications using Time-of-Flight cameras , 2008, 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[38]  G. Yahav,et al.  3D Imaging Camera for Gaming Application , 2007, 2007 Digest of Technical Papers International Conference on Consumer Electronics.

[39]  Behzad Dariush,et al.  Controlled human pose estimation from depth image streams , 2008, 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[40]  Thomas Martinetz,et al.  A facial feature tracker for human-computer interaction based on 3D TOF cameras , 2009 .

[41]  Andreas Kolb,et al.  Real-time simulation of time-of-flight sensors , 2009, Simul. Model. Pract. Theory.

[42]  D.W. Hansen,et al.  Improving Face Detection with TOF Cameras , 2007, 2007 International Symposium on Signals, Circuits and Systems.

[43]  Reinhard Koch,et al.  Extraction of 3D freeform surfaces as visual landmarks for real-time tracking , 2007, Journal of Real-Time Image Processing.

[44]  Jürgen Valldorf,et al.  Fast Fusion of Range and Video Sensor Data , 2007 .

[45]  Rasmus Larsen,et al.  TOF imaging in Smart room environments towards improved people tracking , 2008, 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[46]  Timo Kahlmann,et al.  CALIBRATION FOR INCREASED ACCURACY OF THE RANGE IMAGING CAMERA SWISSRANGER , 2006 .

[47]  Reinhard Koch,et al.  Increasing Realism and Supporting Content Planning for Dynamic Scenes in a Mixed Reality System incorporating a Time-of-Flight Camera , 2010, J. Virtual Real. Broadcast..

[48]  H. Aanaes,et al.  Environmental Effects on Measurement Uncertainties of Time-of-Flight Cameras , 2007, 2007 International Symposium on Signals, Circuits and Systems.

[49]  David J. Ward,et al.  Artificial intelligence: Fast hands-free writing by gaze direction , 2002, Nature.

[50]  G. A. Thomas,et al.  A versatile camera position measurement system for virtual reality TV production , 1997 .

[51]  Til Aach,et al.  Analysis of Superimposed Oriented Patterns , 2006, IEEE Transactions on Image Processing.

[52]  Liu Wei.  A Survey of Image-Based Rendering , 2009 .

[53]  Jens T. Thielemann,et al.  Pipeline landmark detection for autonomous robot navigation using time-of-flight imagery , 2008, 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[54]  D. Falie,et al.  Distance errors correction for the Time of Flight (ToF) cameras , 2008, 2008 IEEE International Workshop on Imaging Systems and Techniques.

[55]  Jens T. Thielemann,et al.  Modelling and compensating measurement errors caused by scattering in time-of-flight cameras , 2008, Optical Engineering + Applications.

[56]  Joachim Hornegger,et al.  Robust real-time 3D respiratory motion detection using time-of-flight cameras , 2008, International Journal of Computer Assisted Radiology and Surgery.

[57]  Reinhard Koch,et al.  Supporting Structure from Motion with a 3D-Range-Camera , 2007, SCIA.

[58]  Li Guan,et al.  3D Object Reconstruction with Heterogeneous Sensor Data , 2008 .

[59]  Tsuhan Chen,et al.  A survey on image-based rendering - representation, sampling and compression , 2004, Signal Process. Image Commun..

[60]  Ruigang Yang,et al.  Spatial-Depth Super Resolution for Range Images , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[61]  Andreas Kolb,et al.  Spherical light field rendering in application for analysis by synthesis , 2008, Int. J. Intell. Syst. Technol. Appl..

[62]  Agnes Swadzba,et al.  Tracking objects in 6D for reconstructing static scenes , 2008, 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[63]  Thomas B. Moeslund,et al.  Fusion of range and intensity information for view invariant gesture recognition , 2008, 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[64]  Heinz Hügli,et al.  Optimized scattering compensation for time-of-flight camera , 2007, SPIE Optics East.

[65]  Klaus-Dieter Kuhnert,et al.  Fusion ofStereo-Camera andPMD-Camera DataforReal- TimeSuited Precise 3DEnvironment Reconstruction , 2006 .

[66]  Lance Williams,et al.  Casting curved shadows on curved surfaces , 1978, SIGGRAPH.

[67]  David J. Ward,et al.  Fast Hands-free Writing by Gaze Direction , 2002, ArXiv.

[68]  Reinhard Koch,et al.  Integration of a Time-of-Flight Camera into a Mixed Reality System for Handling Dynamic Scenes, Moving Viewpoints and Occlusions in Real-Time , 2008 .

[69]  Reinhard Koch,et al.  A Combined Approach for Estimating Patchlets from PMD Depth Images and Stereo Intensity Images , 2007, DAGM-Symposium.

[70]  Joachim Hornegger,et al.  Time-of-flight sensor for respiratory motion gating. , 2008, Medical physics.

[71]  D. Falie.  3D image correction for time of flight (ToF) cameras , 2009, International Conference on Optical Instruments and Technology.

[72]  Rasmus Larsen,et al.  Fusion of stereo vision and Time-Of-Flight imaging for improved 3D estimation , 2008, Int. J. Intell. Syst. Technol. Appl..

[73]  Andreas Kolb,et al.  Sub-pixel data fusion and edge-enhanced distance refinement for 2D/3D images , 2008, Int. J. Intell. Syst. Technol. Appl..

[74]  E. Barth,et al.  Geometric Invariants for Facial Feature Tracking with 3D TOF Cameras , 2007, 2007 International Symposium on Signals, Circuits and Systems.

[75]  Reinhard Koch,et al.  ToF-sensors: New dimensions for realism and interactivity , 2008, 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[76]  Roberto Manduchi,et al.  Robust curb and ramp detection for safe parking using the Canesta TOF camera , 2008, 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[77]  Thomas Martinetz,et al.  Shading constraint improves accuracy of time-of-flight measurements , 2010, Comput. Vis. Image Underst..

[78]  Otmar Loffeld,et al.  Real Time Hand Based Robot Control Using 2D/3D Images , 2008, ISVC.

[79]  Bernd Jähne,et al.  Range Flow Estimation , 2002, Comput. Vis. Image Underst..

[80]  Robert Lange,et al.  3D time-of-flight distance measurement with custom solid-state image sensors in CMOS/CCD-technology , 2006 .

[81]  Jörg Krüger,et al.  Incident light related distance error study and calibration of the PMD-range imaging camera , 2008, 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.