Position stabilisation and lag reduction with Gaussian processes in sensor fusion system for user performance improvement

In this paper we present a novel Gaussian Process (GP) prior model-based sensor fusion approach to dealing with position uncertainty and lag in a system composed of an external position sensing device (Kinect) and inertial sensors embedded in a mobile device for user performance improvement. To test the approach, we conducted two experiments: (1) GPs sensor fusion simulation. Experimental results show that the novel GP sensor fusion helps improve the accuracy of position estimation, and reduce the lag (0.11 s). (2) User study on a trajectory-based target acquisition task in a spatially aware display application. We implemented the real-time sensor fusion system by augmenting the Kinect with a Nokia N9. In the trajectory-based interaction experiment, each user performed target selection tasks following a trajectory in (a) the Kinect system and (b) the sensor fusion system. In comparison with the Kinect time-delay system, our system enables the user to perform the task easier and faster. The MSE of target selection was reduced by 38.3 % and the average task completion time was reduced by 26.7 %.

[1]  Dou Long,et al.  Fusion of detection probabilities and comparison of multisensor systems , 1990, IEEE Trans. Syst. Man Cybern..

[2]  David J. Fleet,et al.  This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE Gaussian Process Dynamical Model , 2007 .

[3]  Jonathan W. Decker,et al.  Performance measurements for the Microsoft Kinect skeleton , 2012, 2012 IEEE Virtual Reality Workshops (VRW).

[4]  James Llinas,et al.  Handbook of Multisensor Data Fusion : Theory and Practice, Second Edition , 2008 .

[5]  Greg Welch,et al.  SCAAT: incremental tracking with incomplete information , 1997, SIGGRAPH.

[6]  Carl Gutwin,et al.  Assessing target acquisition and tracking performance for complex moving targets in the presence of latency and jitter , 2012, Graphics Interface.

[7]  Dieter Fox,et al.  GP-UKF: Unscented kalman filters with Gaussian process prediction and observation models , 2007, 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[8]  Fakhri Karray,et al.  Multisensor data fusion: A review of the state-of-the-art , 2013, Inf. Fusion.

[9]  James Llinas,et al.  An introduction to multisensor data fusion , 1997, Proc. IEEE.

[10]  Sebastian Thrun,et al.  Probabilistic robotics , 2002, CACM.

[11]  Nicolas Roussel,et al.  1 € filter: a simple speed-based low-pass filter for noisy input in interactive systems , 2012, CHI.

[12]  Steven Strachan,et al.  Bearing-based selection in mobile spatial interaction , 2009, Personal and Ubiquitous Computing.

[13]  Robin R. Murphy,et al.  Biological and cognitive foundations of intelligent sensor fusion , 1996, IEEE Trans. Syst. Man Cybern. Part A.

[14]  Ronald Azuma,et al.  A Survey of Augmented Reality , 1997, Presence: Teleoperators & Virtual Environments.

[15]  Zhipei Huang,et al.  Adaptive Information Fusion for Human Upper Limb Movement Estimation , 2012, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[16]  Carl E. Rasmussen,et al.  Derivative Observations in Gaussian Process Models of Dynamic Systems , 2002, NIPS.

[17]  Loring Holden,et al.  Providing a low latency user experience in a high latency application , 1997, SI3D.

[18]  Dieter Fox,et al.  GP-BayesFilters: Bayesian filtering using Gaussian process prediction and observation models , 2008, IROS.

[19]  Alexander I. J. Forrester,et al.  Multi-fidelity optimization via surrogate modelling , 2007, Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences.

[20]  John Weston,et al.  Basic principles of strapdown inertial navigation systems , 2004 .

[21]  I. Scott MacKenzie,et al.  Lag as a determinant of human performance in interactive systems , 1993, INTERCHI.

[22]  Zoubin Ghahramani,et al.  A Unifying Review of Linear Gaussian Models , 1999, Neural Computation.

[23]  Danilo De Rossi,et al.  Designing and Evaluating a Social Gaze-Control System for a Humanoid Robot , 2014, IEEE Transactions on Human-Machine Systems.

[24]  Guy H. Walker,et al.  Human Factors Methods: A Practical Guide for Engineering and Design , 2012 .

[25]  Shrihari Vasudevan,et al.  Data fusion with Gaussian processes , 2012, Robotics Auton. Syst..

[26]  Basilio Sierra,et al.  Fusing multiple image transformations and a thermal sensor with kinect to improve person detection ability , 2013, Eng. Appl. Artif. Intell..

[27]  Ka-Ping Yee,et al.  Peephole displays: pen interaction on spatially aware handheld computers , 2003, CHI '03.

[28]  S. Hart,et al.  Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research , 1988 .

[29]  Andy J. Keane,et al.  Engineering Design via Surrogate Modelling - A Practical Guide , 2008 .

[30]  Barak A. Pearlmutter,et al.  Transformations of Gaussian Process Priors , 2004, Deterministic and Statistical Methods in Machine Learning.

[31]  Peter Corke,et al.  An Introduction to Inertial and Visual Sensing , 2007, Int. J. Robotics Res..

[32]  Asha,et al.  A Hand Gesture Recognition Framework and Wearable Gesture Based Interaction Prototype for Mobile Devices , 2015 .

[33]  Chun C. Lai,et al.  Multisensor Fusion and Integration: Theories, Applications, and its Perspectives , 2011, IEEE Sensors Journal.

[34]  D. Mackay,et al.  Introduction to Gaussian processes , 1998 .

[35]  John Williamson,et al.  Continuous uncertain interaction , 2006 .

[36]  Stefan Görtz,et al.  Improving variable-fidelity surrogate modeling via gradient-enhanced kriging and a generalized hybrid bridge function , 2013 .

[37]  Roderick Murray-Smith,et al.  Shoogle: excitatory multimodal interaction on mobile devices , 2007, CHI.

[38]  Albrecht Schmidt,et al.  Implicit human computer interaction through context , 2000, Personal Technologies.

[39]  Marion Koelle,et al.  Putting books back on the shelf: situated interactions with digital book collections on smartphones , 2013, MUM.

[40]  James Llinas,et al.  Handbook of Multisensor Data Fusion , 2001 .

[41]  Sameer Singh,et al.  Approaches to Multisensor Data Fusion in Target Tracking: A Survey , 2006, IEEE Transactions on Knowledge and Data Engineering.

[42]  Michael Rohs,et al.  Target acquisition with camera phones when used as magic lenses , 2008, CHI.

[43]  John Weston,et al.  Strapdown Inertial Navigation Technology , 1997 .

[44]  P. Goovaerts.  Ordinary Cokriging Revisited , 1998 .

[45]  D.J. Leith,et al.  Gaussian process prior models for electrical load forecasting , 2004, 2004 International Conference on Probabilistic Methods Applied to Power Systems.

[46]  Peter D. Lawrence,et al.  Fixation Precision in High-Speed Noncontact Eye-Gaze Tracking , 2008, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[47]  Ravin Balakrishnan,et al.  Reaching for objects in VR displays: lag and frame rate , 1994, TCHI.

[48]  George W. Fitzmaurice,et al.  Situated information spaces and spatially aware palmtop computers , 1993, CACM.

[49]  Christopher K. I. Williams,et al.  Gaussian Processes for Machine Learning (Adaptive Computation and Machine Learning) , 2005 .

[50]  Roderick Murray-Smith,et al.  Fusing Kinect sensor and inertial sensors with multi-rate Kalman filter , 2014 .

[51]  Thomas B. Schön,et al.  Robust real-time tracking by fusing measurements from inertial and vision sensors , 2007, Journal of Real-Time Image Processing.

[52]  Andriy Pavlovych,et al.  The tradeoff between spatial jitter and latency in pointing tasks , 2009, EICS '09.

[53]  C. Rasmussen,et al.  Gaussian Process Priors with Uncertain Inputs - Application to Multiple-Step Ahead Time Series Forecasting , 2002, NIPS.

[54]  Philippe Poignet,et al.  Joint angle estimation in rehabilitation with inertial sensors and its integration with Kinect , 2011, 2011 Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[55]  Ben Shneiderman,et al.  Designing The User Interface , 2013 .

[56]  R. D. Turner.  Gaussian processes for state space models and change point detection , 2012 .

[57]  Roderick Murray-Smith,et al.  Transformations of Gaussian Process priors for user matching , 2016, Int. J. Hum. Comput. Stud..

[58]  Soo Jeon,et al.  Kinematic Kalman Filter (KKF) for Robot End-Effector Sensing , 2009 .