Device recognition for intuitive interaction with the web of things

Supporting human users when interacting with smart devices is important to drive the successful adoption of the Internet of Things in people's homes and at their workplaces. In this poster contribution, we present a system that helps users control Web-enabled smart things in their environment. Our approach involves a handheld interaction device that recognizes smart things in its view using state-of-the-art visual object recognition techniques. It then augments the camera feed with appropriate interaction primitives such as knobs or buttons for control, and can also display measured values, for instance, when recognizing a sensor. The interaction primitives are generated from user interface descriptions that are embedded in the Web representations of the smart things. Our prototype implementation achieves frame rates that allow for interactive use of the system by human users, and indeed proved to facilitate the interaction with smart things in a demonstration testbed in our research group.