Hard and soft information fusion in sonification for assistive mobile device technology

Current research integrating sonification with hard and soft information fusion is poised to enable a new class of assistive technology that is driven by sensor technology and informed by semantic understanding of the userÕs capabilities and current goals as well as events unfolding in the world around them. At the time of this writing, Penn State University is in the fourth year of an Army Research Office (ARO) funded MURI grant to develop networked hard and soft information fusion techniques with the main goal of using Òsoft dataÓ (including textual and spoken reports from military personnel; and weather, news, and nearby hazard reports from the web and publicly available social media feeds) along with Òhard dataÓ from a variety of physical sensors (including video, acoustic, LIDAR, and rangefinder) with the goal of detecting and preventing the use of improvised explosive devices (IEDs) against deployed troops. Over the course of this research, we have identified several areas in which the sonification-based utilization of hard and soft information can offer significant improvements to existing assistive technologies and tools. This paper presents the key elements of this research, describes areas of applicability in assistive technology, and presents suggestions for future research.

[1]  David L. Hall,et al.  Human cognitive and perceptual factors in JDL level 4 hard/soft data fusion , 2012, Defense + Commercial Sensing.

[2]  Deborah L. McGuinness,et al.  An Intelligent Personal Assistant for Task and Time Management , 2007, AI Mag..

[3]  Bruce N. Walker,et al.  Navigation Performance With a Virtual Auditory Display: Effects of Beacon Sound, Capture Radius, and Practice , 2006, Hum. Factors.

[4]  Susanne Boll,et al.  Interactive 3D sonification for the exploration of city maps , 2006, NordiCHI '06.

[5]  Frank Dellaert,et al.  SWAN: System for Wearable Audio Navigation , 2007, 2007 11th IEEE International Symposium on Wearable Computers.

[6]  Bruce N Walker,et al.  Using Virtual Environments to Prototype Auditory Navigation Displays , 2005, Assistive technology : the official journal of RESNA.

[7]  Michael D. McNeese,et al.  Conserving analyst attention units: use of multi-agent software and CEP methods to assist information analysis , 2013, Defense, Security, and Sensing.

[8]  Alberto de Campo,et al.  Toward a data sonification design space map , 2007 .

[9]  Frank Dellaert,et al.  Map-based priors for localization , 2004, 2004 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (IEEE Cat. No.04CH37566).

[10]  Richard L. Tutwiler,et al.  Using full motion 3D Flash LIDAR video for target detection, segmentation, and tracking , 2010, 2010 IEEE Southwest Symposium on Image Analysis & Interpretation (SSIAI).

[11]  Mark Ballora,et al.  Beyond visualization of big data: a multi-stage data exploration approach using visualization, sonification, and storification , 2013, Defense, Security, and Sensing.

[12]  Pawel Strumillo,et al.  Naviton—A Prototype Mobility Aid for Auditory Presentation of Three-Dimensional Scenes to the Visually Impaired , 2012 .

[13]  T. Shioyama,et al.  Detection of pedestrian crossing and measurement of crossing length - an image-based navigational aid for blind people , 2005, Proceedings. 2005 IEEE Intelligent Transportation Systems, 2005..

[14]  Jeffrey C. Rimland.  Service-Oriented Architecture for Human-Centric Information Fusion , 2012 .

[15]  Fang Zhigang,et al.  Audification-based Electronic Travel Aid system , 2010, 2010 International Conference On Computer Design and Applications.

[16]  Alessandro Margara,et al.  Processing flows of information: From data stream to complex event processing , 2012, CSUR.

[17]  Jake Graham,et al.  JDL level 0 and 1 algorithms for processing and fusion of hard sensor data , 2011, Defense + Commercial Sensing.

[18]  Fatos Xhafa,et al.  A next generation emerging technologies roadmap for enabling collective computational intelligence in disaster management , 2011, Int. J. Space Based Situated Comput..

[19]  Pattie Maes,et al.  Agents that reduce work and information overload , 1994, CACM.