To move or to remove?: a human-centric approach to understanding gesture interpretation

This paper explores how na?ve observers recognize and interpret transitive actions (actions involving manipulation of objects) without accompanying speech, in order to derive guidelines for the design of gesture interpretation systems. Semi-structured interviews with 11 observers, interpreting 106 video clips of transitive actions elicited unstaged from 16 participants, reveal that people are generally able to interpret the transitive action as well as characteristics of the object manipulated despite individual variations in how people naturally gesture. In particular, people focus primarily on hand movement and hand shape to correctly interpret object characteristics, and on manner of movement of arms and/or final location of hands to interpret the goal of the transitive action (e.g., arrange objects vs. clear objects). These findings provide insights on aspects of gestures one can focus on to inform and guide the design of gesture interpretation models for interfaces that allow for individual variations in natural gesture production.