SurfaceLink: using inertial and acoustic sensing to enable multi-device interaction on a surface

We present SurfaceLink, a system where users can make natural surface gestures to control association and information transfer among a set of devices that are placed on a mutually shared surface (e.g., a table). SurfaceLink uses a combination of on-device accelerometers, vibration motors, speakers and microphones (and, optionally, an off-device contact microphone for greater sensitivity) to sense gestures performed on the shared surface. In a controlled evaluation with 10 participants, SurfaceLink detected the presence of devices on the same surface with 97.7% accuracy, their relative arrangement with 89.4% accuracy, and various single- and multi-touch surface gestures with an average accuracy of 90.3%. A usability analysis showed that SurfaceLink has advantages over current multi-device interaction techniques in a number of situations.

[1]  Paul Lukowicz,et al.  Sound-based proximity detection with mobile phones , 2012, PhoneSense '12.

[2]  Dominik Schmidt.  PhoneTouch: a technique for direct phone interaction on surfaces , 2010, ITS '10.

[3]  Robert Xiao,et al.  Acoustic barcodes: passive, durable and inexpensive notched identification tags , 2012, UIST.

[4]  Sandra G. Hart,et al.  Nasa-Task Load Index (NASA-TLX); 20 Years Later , 2006 .

[5]  Andrés Lucero,et al.  EasyGroups: binding mobile devices for collaborative interactions , 2012, CHI EA '12.

[6]  Tom Cuypers.  Smartphone Localization on Interactive Surfaces Using the Built-in Camera , 2009 .

[7]  Chris Harrison,et al.  Lightweight material detection for placement-aware mobile computing , 2008, UIST '08.

[8]  Shwetak N. Patel,et al.  GripSense: using built-in sensors to detect hand posture and pressure on commodity mobile phones , 2012, UIST.

[9]  Jun Rekimoto,et al.  Augmented surfaces: a spatially continuous work space for hybrid computing environments , 1999, CHI '99.

[10]  Pedro Lopes,et al.  Augmenting touch interaction through acoustic sensing , 2011, ITS '11.

[11]  Chris Harrison,et al.  TapSense: enhancing finger interaction on touch surfaces , 2011, UIST.

[12]  René Mayrhofer,et al.  Shake Well Before Use: Authentication Based on Accelerometer Data , 2007, Pervasive.

[13]  Ken Hinckley,et al.  Synchronous Gestures in Multi-Display Environments , 2009, Hum. Comput. Interact..

[14]  Dorothea Blostein,et al.  Pen Acoustic Emissions for Text and Gesture Recognition , 2009, 2009 10th International Conference on Document Analysis and Recognition.

[15]  Joseph A. Paradiso,et al.  Passive acoustic sensing for tracking knocks atop large interactive displays , 2002, Proceedings of IEEE Sensors.

[16]  Pietro Polotti,et al.  Tangible Acoustic Interfaces and their Applications for the Design of New Musical Instruments , 2005, NIME.

[17]  Ken Hinckley,et al.  Cooperative Stitching: Spontaneous Wireless Connections for Small Co-Located Groups. , 2016 .

[18]  Khai N. Truong,et al.  Determining the orientation of proximate mobile devices using their back facing camera , 2012, CHI.

[19]  Jun Rekimoto,et al.  HoloWall: designing a finger, hand, body, and object sensitive wall , 1997, UIST '97.

[20]  Diana K. Smetters,et al.  Talking to Strangers: Authentication in Ad-Hoc Wireless Networks , 2002, NDSS.

[21]  Shahram Izadi,et al.  SideSight: multi-"touch" interaction around small devices , 2008, UIST '08.

[22]  Paul Lukowicz,et al.  Symbolic Object Localization Through Active Sampling of Acceleration and Sound Signatures , 2007, UbiComp.

[23]  Patrick Baudisch,et al.  Stitching: pen gestures that span multiple displays , 2004, AVI.

[24]  Roderick Murray-Smith,et al.  Stane: synthesized surfaces for tactile input , 2008, CHI.

[25]  Andrew D. Wilson,et al.  BlueTable: connecting wireless mobile devices on interactive surfaces using vision-based handshaking , 2007, GI '07.

[26]  Peng Song,et al.  Distinguishing multiple smart-phone interactions on a multi-touch wall display using tilt correlation , 2011, CHI.

[27]  Chris Harrison,et al.  Scratch input: creating large, inexpensive, unpowered and mobile finger input surfaces , 2008, UIST '08.

[28]  Jacob O. Wobbrock,et al.  Bonfire: a nomadic system for hybrid laptop-tabletop interaction , 2009, UIST '09.

[29]  Gerd Kortuem,et al.  Sensing and visualizing spatial relations of mobile devices , 2005, UIST.

[30]  Andrés Lucero,et al.  Pass-them-around: collaborative use of mobile phones for photo sharing , 2011, CHI.

[31]  Bernt Schiele,et al.  Smart-Its Friends: A Technique for Users to Easily Establish Connections between Smart Artefacts , 2001, UbiComp.

[32]  Will Archer Arentz,et al.  Near ultrasonic directional data transfer for modern smartphones , 2011, UbiComp '11.