An Interactive Computer Vision System DyPERS: Dynamic Personal Enhanced Reality System

DyPERS, 'Dynamic Personal Enhanced Reality System', uses augmented reality and computer vision to autonomously retrieve 'media memories' based on associations with real objects the user encounters. These are evoked as audio and video clips relevant for the user and overlayed on top of real objects the user encounters. The system utilizes an adaptive, audio-visual learning system on a tetherless wearable computer. The user's visual and auditory scene is stored in real-time by the system (upon request) and is then associated (by user input) with a snap shot of a visual object. The object acts as a key such that when the real-time vision system detects its presence in the scene again, DyPERS plays back the appropriate audio-visual sequence. The vision system is a probabilistic algorithm which is capable of discriminating between hundreds of everyday objects under varying viewing conditions (view changes, lighting, etc.). Once an audio-visual clip is stored, the vision system automatically recalls it and plays it back when it detects the object that the user wished to use to remind him of the sequence. The DyPERS interface augments the user without encumbering him and effectively mimics a form of audio-visual memory. First results on performance and usability are shown.

[1]  Chris Schmandt.  Chatter: A Conversational Learning Speech Interface , 1994 .

[2]  J. Davenport.  Editor , 1960 .

[3]  Katashi Nagao,et al.  Agent Augmented Reality: A Software Agent Meets the Real World , 2001 .

[4]  M. Lamming,et al.  "Forget-me-not" Intimate Computing in Support of Human Memory , 1994 .

[5]  Benjamin B. Bederson,et al.  Audio augmented reality: a prototype automated tour guide , 1995, CHI 95 Conference Companion.

[6]  Yiannis Aloimonos,et al.  Purposive and qualitative active vision , 1990, [1990] Proceedings. 10th International Conference on Pattern Recognition.

[7]  Jennifer Healey,et al.  Augmented Reality through Wearable Computing , 1997, Presence: Teleoperators & Virtual Environments.

[8]  Gerd Kortuem,et al.  Software organization for dynamic and adaptable wearable systems , 1997, Digest of Papers. First International Symposium on Wearable Computers.

[9]  Mark Weiser.  The computer for the 21st century , 1991 .

[10]  Thad Starner,et al.  Remembrance Agent: A Continuously Running Automated Information Retrieval System , 1996, PAAM.

[11]  Blair MacIntyre,et al.  Annotating the real world with knowledge-based graphics on a see-through head-mounted display , 1992 .

[12]  Alex Pentland,et al.  Stochasticks: augmenting the billiards experience with probabilistic vision and wearable computers , 1997, Digest of Papers. First International Symposium on Wearable Computers.

[13]  Vania Conan,et al.  Virtually documented environments , 1997, Digest of Papers. First International Symposium on Wearable Computers.

[14]  Jeffrey M. Levin.  Real-time target and pose recognition for 3-D graphical overlay , 1997 .

[15]  Steve Mann,et al.  Wearable Computing: A First Step Toward Personal Imaging , 1997, Computer.

[16]  Thad Starner,et al.  The locust swarm: an environmentally-powered, networkless location and messaging system , 1997, Digest of Papers. First International Symposium on Wearable Computers.

[17]  Ron Frederick,et al.  Audio aura: light-weight audio augmented reality , 1997, UIST '97.

[18]  Bernt Schiele,et al.  Probabilistic object recognition using multidimensional receptive field histograms , 1996, Proceedings of 13th International Conference on Pattern Recognition.

[19]  Refractor.  Vision , 2000, The Lancet.

[20]  Matthew Turk,et al.  Perceptual user interfaces , 2000 .

[21]  Dana H. Ballard,et al.  Animate Vision , 1991, Artif. Intell..

[22]  John J. Leggett,et al.  Interaction styles and input/output devices , 1993, Behav. Inf. Technol..

[23]  R. Bajcsy.  Active perception , 1988 .

[24]  Bernt Schiele,et al.  Object Recognition Using Multidimensional Receptive Field Histograms , 1996, ECCV.

[25]  Ernst D. Dickmanns,et al.  Vehicles Capable of Dynamic Vision , 1997, IJCAI.

[26]  Katashi Nagao,et al.  The world through the computer: computer augmented interaction with real world environments , 1995, UIST '95.