Augmented reality applications for cultural heritage using Kinect

This paper explores the use of data from the Kinect sensor for performing augmented reality, with emphasis on cultural heritage applications. It is shown that the combination of depth and image correspondences from the Kinect can yield a reliable estimate of the location and pose of the camera, though noise from the depth sensor introduces an unpleasant jittering of the rendered view. Kalman filtering of the camera position was found to yield a much more stable view. Results show that the system is accurate enough for in situ augmented reality applications. Skeleton tracking using Kinect data allows the appearance of participants to be augmented, and together these facilitate the development of cultural heritage applications.

[1]  Didier Stricker,et al.  Real-time and markerless vision-based tracking for outdoor augmented reality applications , 2001, Proceedings IEEE and ACM International Symposium on Augmented Reality.

[2]  Takeo Kanade,et al.  3D Research Challenges in Cultural Heritage , 2014, Lecture Notes in Computer Science.

[3]  Fabio Remondino,et al.  Heritage Recording and 3D Modeling with Photogrammetry and 3D Scanning , 2011, Remote. Sens..

[4]  Fabio Remondino,et al.  Kinect and 3D GIS in archaeology , 2012, 2012 18th International Conference on Virtual Systems and Multimedia.

[5]  A. F. Clark,et al.  Extracting planar features from Kinect sensor , 2012, 2012 4th Computer Science and Electronic Engineering Conference (CEEC).

[6]  Erkan Bostanci,et al.  Kinect-Derived Augmentation of the Real World for Cultural Heritage , 2013, 2013 UKSim 15th International Conference on Computer Modelling and Simulation.

[7]  Erkan Bostanci,et al.  Vision-based user tracking for outdoor augmented reality , 2012, 2012 IEEE Symposium on Computers and Communications (ISCC).

[8]  Boris Jutzi,et al.  Semi-automatic image-based fusion of range imaging data with different characteristics , 2011 .

[9]  Vittorio Scarano,et al.  Real Positioning in Virtual Environments Using Game Engines , 2007, Eurographics Italian Chapter Conference.

[10]  A. F. Clark,et al.  Feature coverage for better homography estimation: An application to image stitching , 2012, 2012 19th International Conference on Systems, Signals and Image Processing (IWSSIP).

[11]  Alex Zelinsky,et al.  Learning OpenCV---Computer Vision with the OpenCV Library (Bradski, G.R. et al.; 2008)[On the Shelf] , 2009, IEEE Robotics & Automation Magazine.

[12]  Adrian David Cheok,et al.  Online 6 DOF augmented reality registration from natural features , 2002, Proceedings. International Symposium on Mixed and Augmented Reality.

[13]  John F. Canny,et al.  A Computational Approach to Edge Detection , 1986, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[14]  Richard Szeliski,et al.  Computer Vision - Algorithms and Applications , 2011, Texts in Computer Science.

[15]  Luc Van Gool,et al.  SURF: Speeded Up Robust Features , 2006, ECCV.

[16]  Sebastian Thrun,et al.  Probabilistic robotics , 2002, CACM.

[17]  Zhigeng Pan,et al.  A Review on Augmented Reality for Virtual Heritage System , 2009, Edutainment.

[18]  Vincent Lepetit,et al.  Accurate Non-Iterative O(n) Solution to the PnP Problem , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[19]  Juho Kannala,et al.  Accurate and Practical Calibration of a Depth and Color Camera Pair , 2011, CAIP.

[20]  Tom Drummond,et al.  Faster and Better: A Machine Learning Approach to Corner Detection , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[21]  Kourosh Khoshelham,et al.  Accuracy analysis of kinect depth data , 2012 .

[22]  Tom Drummond,et al.  Machine Learning for High-Speed Corner Detection , 2006, ECCV.

[23]  Michael Isard,et al.  CONDENSATION—Conditional Density Propagation for Visual Tracking , 1998, International Journal of Computer Vision.

[24]  David H. Douglas,et al.  ALGORITHMS FOR THE REDUCTION OF THE NUMBER OF POINTS REQUIRED TO REPRESENT A DIGITIZED LINE OR ITS CARICATURE , 1973 .

[25]  Mark Fiala,et al.  A Real Time Augmented Reality System Using GPU Acceleration , 2012, 2012 Ninth Conference on Computer and Robot Vision.

[26]  T. Başar,et al.  A New Approach to Linear Filtering and Prediction Problems , 2001 .

[27]  Vincent Lepetit,et al.  BRIEF: Binary Robust Independent Elementary Features , 2010, ECCV.

[28]  Bruce H. Thomas,et al.  Augmented reality working planes: a foundation for action and construction at a distance , 2004, Third IEEE and ACM International Symposium on Mixed and Augmented Reality.

[29]  Bernhard P. Wrobel,et al.  Multiple View Geometry in Computer Vision , 2001 .

[30]  Jan T. Fischer,et al.  Rendering methods for augmented reality , 2008 .

[31]  Erkan Bostanci,et al.  User Tracking Methods for Augmented Reality , 2013 .

[32]  Erkan Bostanci,et al.  Living the Past in the Future , 2011, Intelligent Environments.

[33]  Harald Ganster,et al.  Hybrid Tracking for Outdoor Augmented Reality Applications , 2002, IEEE Computer Graphics and Applications.

[34]  Vincent Lepetit,et al.  Monocular Model-Based 3D Tracking of Rigid Objects: A Survey , 2005, Found. Trends Comput. Graph. Vis..

[35]  R. Hawkey.  Learning with Digital Technologies in Museums, Science Centres and Galleries , 2004 .

[36]  Robert C. Bolles,et al.  Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography , 1981, CACM.

[37]  Gary R. Bradski,et al.  Learning OpenCV 3: Computer Vision in C++ with the OpenCV Library , 2016 .

[38]  George Papagiannakis,et al.  A survey of mobile and wireless technologies for augmented reality systems , 2008, Comput. Animat. Virtual Worlds.

[39]  Eva Pietroni,et al.  The Etruscanning project: Gesture-based interaction and user experience in the virtual reconstruction of the Regolini-Galassi tomb , 2013, 2013 Digital Heritage International Congress (DigitalHeritage).

[40]  Marco Gaiani,et al.  An Integrated 3D Geodatabase for Palladio's Work , 2010 .

[41]  Andy C. Downton,et al.  Real-time positioning for augmented reality on a custom parallel machine , 2005, Image Vis. Comput..

[42]  V. Lepetit,et al.  EPnP: An Accurate O(n) Solution to the PnP Problem , 2009, International Journal of Computer Vision.

[43]  Suya You,et al.  A robust hybrid tracking system for outdoor augmented reality , 2004, IEEE Virtual Reality 2004.

[44]  Ronald Azuma,et al.  A Survey of Augmented Reality , 1997, Presence: Teleoperators & Virtual Environments.

[45]  S. Shankar Sastry,et al.  An Invitation to 3-D Vision , 2004 .

[46]  Robert M. Haralick,et al.  Review and analysis of solutions of the three point perspective pose estimation problem , 1994, International Journal of Computer Vision.

[47]  John N. Karigiannis,et al.  Archeoguide: system architecture of a mobile outdoor augmented reality system , 2002, Proceedings. International Symposium on Mixed and Augmented Reality.