CyPhy-UI: Cyber-Physical User Interaction Paradigm to Control Networked Appliances with Augmented Reality

Many kinds of networked home appliances, which are connected by standardized control functions, have recently appeared and continue to increase. Because a general infrared remote control is for single-way communication from a remote control to a specific appliance, but not to receive signals from an appliance to a remote control, it is impossible to gather an appliance's information with a remote control. Meanwhile, unlike a general infrared remote control, it is difficult to control a specific appliance because users can simultaneously operate all of their appliances with a WiFi controller. In this paper, to control networked appliances with a smart phone or tablet computer as a WiFi controller, we propose a new interface paradigm called a cyber-physical user interaction that creates virtual (cyber) space, sends commands, and receives responses from networked (physical) appliances through space with augmented-reality (AR) technology. With the paradigm, which enables interconnectivity among appliances from various vendors, it is possible users with uniform and intuitive operation of home appliances. In addition, we implement and evaluate an Embodied Visualization with Augmented-Reality for Networked Systems (EVANS) that controls a system of home appliances and sensor devices through a cyber-physical user interaction (CyPhy-UI) paradigm by a web camera to retrieve information from real world environments and touch-screen display to show AR visualization and user interaction components to retrieve user input.