Grasping, communicating, understanding: Connecting reality and virtuality

Several simulation projects in the area of production and logistics indicated that, although we have sophisticated input and output devices for computer supported modelling, physical models still play an important role for cognition and communication. We therefore introduce the concept of a Graspable User Interface that aims at combining two model worlds, the one inside the computer and a corresponding physical one in the outside world. Sensored user hands will couple physical objects of the real world with virtual objects, thus allowing fairly unrestricted manipulation and expression. In this way modelling with real physical objects can create an abstract virtual model. Some applications of this concept are presented. A further perspective for a new action oriented communication and learning with artifacts is envisioned.

[1]  Hiroshi Ishii,et al.  Bricks: laying the foundations for graspable user interfaces , 1995, CHI '95.

[2]  Friedrich Wilhelm Bruns,et al.  Joint Project: Simulating Work and Technology — Development, Implementation and Application of Computer-Based Systems for Planning, Simulation and Animation in Manufacturing , 1995 .

[3]  Katsushi Ikeuchi,et al.  Grasp recognition and manipulative motion characterization from human hand motion sequences , 1994, Proceedings of the 1994 IEEE International Conference on Robotics and Automation.

[4]  Fritz Böhle,et al.  Vom Handrad zum Bildschirm : eine Untersuchung zur sinnlichen Erfahrung im Arbeitsprozeß , 1988 .

[5]  Dieter Müller.  Application framework für die Entwicklung interaktiver Anwendungen mit multisensorischen Benutzungsschnittstellen: Ansätze einer prototypischen Realisierung von Rechnersystemen für die Produktion , 1993 .

[6]  Tamotsu Murakami,et al.  Direct and intuitive input device for 3-D shape deformation , 1994, CHI '94.

[7]  Kees de Koning,et al.  Qualitative reasoning: Modeling and simulation with incomplete knowledge , 1996 .

[8]  Richard Fikes,et al.  Causal functional representation language with behavior-based semantics , 1995, Appl. Artif. Intell..

[9]  Mitchel Resnick,et al.  Behavior construction kits , 1993, CACM.

[10]  Wendy E. Mackay,et al.  Computer-Augmented Environments: Back to the Real World - Introduction to the Special Issue. , 1993 .

[11]  Wendy E. Mackay,et al.  Back to the real world , 1993, CACM.

[12]  Jean Piaget,et al.  La formation du symbole chez l'enfant : imitation, jeu et rêve, image et représentation , 1976 .

[13]  J. Piaget.  La formation du symbole chez l’enfant , 2007 .

[14]  W. Bion,et al.  Learning from experience , 2007 .