GestureAnalyzer: visual analytics for pattern analysis of mid-air hand gestures

Understanding the intent behind human gestures is a critical problem in the design of gestural interactions. A common method to observe and understand how users express gestures is to use elicitation studies. However, these studies require time-consuming analysis of user data to identify gesture patterns. Also, the analysis by humans cannot describe gestures in as detail as in data-based representations of motion features. In this paper, we present GestureAnalyzer, a system that supports exploratory analysis of gesture patterns by applying interactive clustering and visualization techniques to motion tracking data. GestureAnalyzer enables rapid categorization of similar gestures, and visual investigation of various geometric and kinematic properties of user gestures. We describe the system components, and then demonstrate its utility through a case study on mid-air hand gestures obtained from elicitation studies.

[1]  A. Savitzky,et al.  Smoothing and Differentiation of Data by Simplified Least Squares Procedures. , 1964 .

[2]  Konrad Tollmar,et al.  Gesture + play: full-body interaction for virtual environments , 2003, CHI Extended Abstracts.

[3]  S. Mitra,et al.  Gesture Recognition: A Survey , 2007, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[4]  Mark Gahegan,et al.  Opening the black box: interactive hierarchical clustering for multivariate spatial patterns , 2002, GIS '02.

[5]  Lise Amy Hansen.  Full-body movement as material for interaction design , 2011, Digit. Creativity.

[6]  Brad A. Myers,et al.  Maximizing the guessability of symbolic input , 2005, CHI Extended Abstracts.

[7]  Yi Yang,et al.  Learning a 3D Human Pose Distance Metric from Geometric Pose Descriptor , 2011, IEEE Transactions on Visualization and Computer Graphics.

[8]  Kay Nieselt,et al.  iHAT: interactive Hierarchical Aggregation Table for Genetic Association Data , 2012, BMC Bioinformatics.

[9]  Manfred Tscheligi,et al.  Exploring the Possibilities of Body Motion Data for Human Computer Interaction Research , 2010, USAB.

[10]  Scott R. Klemmer,et al.  Authoring sensor-based interactions by demonstration with direct manipulation and pattern recognition , 2007, CHI.

[11]  Clement T. Yu,et al.  An interactive clustering-based approach to integrating source query interfaces on the deep Web , 2004, SIGMOD '04.

[12]  Lucas Kovar,et al.  Motion graphs , 2002, SIGGRAPH Classes.

[13]  Ben Shneiderman,et al.  Interactively Exploring Hierarchical Clustering Results , 2002, Computer.

[14]  Marcelo Worsley,et al.  Towards the development of multimodal action based assessment , 2013, LAK '13.

[15]  T. Warren Liao,et al.  Clustering of time series data - a survey , 2005, Pattern Recognit..

[16]  Andy Cockburn,et al.  FingARtips: gesture based direct manipulation in Augmented Reality , 2004, GRAPHITE '04.

[17]  S. B. Needleman,et al.  A general method applicable to the search for similarities in the amino acid sequence of two proteins. , 1970, Journal of molecular biology.

[18]  Radu-Daniel Vatavu,et al.  User-defined gestures for free-hand TV control , 2012, EuroITV.

[19]  Hans-Peter Kriegel,et al.  OPTICS: ordering points to identify the clustering structure , 1999, SIGMOD '99.

[20]  Yang Li,et al.  User-defined motion gestures for mobile interaction , 2011, CHI.

[21]  Andy Cockburn,et al.  User-defined gestures for augmented reality , 2013, INTERACT.

[22]  Tony DeRose,et al.  Proton++: a customizable declarative multitouch framework , 2012, UIST.

[23]  Ruzena Bajcsy,et al.  Sequence of the Most Informative Joints (SMIJ): A new representation for human skeletal action recognition , 2012, 2012 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[24]  Ben Shneiderman,et al.  Interactively Exploring Hierarchical Clustering Results , 2003 .

[25]  Daniel A. Keim,et al.  Visual Analytics: Definition, Process, and Challenges , 2008, Information Visualization.

[26]  Michela Bertolotto,et al.  SPATIO-TEMPORAL CLUSTERING OF MOVEMENT DATA: AN APPLICATION TO TRAJECTORIES GENERATED BY HUMAN-COMPUTER INTERACTION , 2012 .

[27]  Meredith Ringel Morris,et al.  User-defined gestures for surface computing , 2009, CHI.

[28]  Arne Jönsson,et al.  Wizard of Oz studies: why and how , 1993, IUI '93.

[29]  Regan L. Mandryk,et al.  Full-body motion-based game interaction for older adults , 2012, CHI.

[30]  Dieter W. Fellner,et al.  Trajectory-based visual analysis of large financial time series data , 2007, SKDD.

[31]  Perttu Hämäläinen,et al.  Wizard of Oz prototyping of computer vision based action games for children , 2004, IDC '04.

[32]  Philippe De Maeyer,et al.  Analysing the spatial dimension of eye movement data using a visual analytic approach , 2012, Expert Syst. Appl..

[33]  Patrick Baudisch,et al.  Stitching: pen gestures that span multiple displays , 2004, AVI.

[34]  Hao-Chuan Wang,et al.  Kinect-taped communication: using motion sensing to study gesture use and similarity in face-to-face and computer-mediated brainstorming , 2014, CHI.

[35]  Thad Starner,et al.  MAGIC: a motion gesture design tool , 2010, CHI.

[36]  John T. Stasko,et al.  Effectiveness of Animation in Trend Visualization , 2008, IEEE Transactions on Visualization and Computer Graphics.

[37]  Tek-Jin Nam,et al.  EventHurdle: supporting designers' exploratory interaction prototyping with gesture-based sensors , 2013, CHI.

[38]  Yang Li,et al.  Gesture coder: a tool for programming multi-touch gestures by demonstration , 2012, CHI.

[39]  Jun Rekimoto,et al.  Z-touch: an infrastructure for 3d gesture interaction in the proximity of tabletop surfaces , 2010, ITS '10.

[40]  Tobias Schreck,et al.  MotionExplorer: Exploratory Search in Human Motion Capture Data Based on Hierarchical Aggregation , 2013, IEEE Transactions on Visualization and Computer Graphics.

[41]  Bongshin Lee,et al.  Reducing legacy bias in gesture elicitation studies , 2014, INTR.

[42]  Dino Pedreschi,et al.  Visually driven analysis of movement data by progressive clustering , 2008, Inf. Vis..

[43]  Sander Oude Elberink,et al.  Accuracy and Resolution of Kinect Depth Data for Indoor Mapping Applications , 2012, Sensors.