Sensor data fusion for context-aware computing using dempster-shafer theory

Towards having computers understand human users' “context” information, this dissertation proposes a systematic context-sensing implementation methodology that can easily combine sensor outputs with subjective judgments. The feasibility of this idea is demonstrated via a meeting-participant's focus-of-attention analysis case study with several simulated sensors using prerecorded experimental data and artificially generated sensor outputs distributed over a LAN network. The methodology advocates a top-down approach: (1) For a given application, a context information structure is defined; all lower-level sensor fusion is done locally. (2) Using the context information architecture as a guide, a context sensing system with layered and modularized structure is developed using the Georgia Tech Context Toolkit system, enhanced with sensor fusion modules, as its building-blocks. (3) Higher-level context outputs are combined through “sensor fusion mediator” widgets, and the results populate the context database. The key contribution of this thesis is introducing the Dempster-Shafer theory of evidence as a generalizable sensor fusion solution to overcome the typical context-sensing difficulties, wherein some of the available information items are subjective, sensor observations' probability (objective chance) distribution is not known accurately, and the sensor set is dynamic in content and configuration. In the sensor fusion implementation, this method is further extended in two directions: (1) weight factors are introduced to adjust each sensor's voting influence, thus providing an “objective” sensor performance justification; and (2) when the ground truth becomes available, it is used to dynamically adjust the sensors' voting weights. The effectiveness of the improved Dempster-Shafer method is demonstrated with both the prerecorded experimental data and the simulated data.

[1]  Vincent M. Stanford,et al.  Using Pervasive Computing to Deliver Elder Care , 2002, IEEE Pervasive Comput..

[2]  Ted Faison,et al.  Interaction Patterns for Communicating Processes , 1998 .

[3]  Ronald P. S. Mahler,et al.  The modified Dempster-Shafer approach to classification , 1997, IEEE Trans. Syst. Man Cybern. Part A.

[4]  Marcela D. Rodríguez,et al.  Context-Aware Mobile Communication in Hospitals , 2003, Computer.

[5]  Keith Cheverst,et al.  Using and Determining Location in a Context-Sensitive Tour Guide , 2001, Computer.

[6]  Stephen M. Jameson,et al.  Architectures for Distributed Information Fusion To Support Situation Awareness on the Digital Battlefield , 2002 .

[7]  Irfan Essa,et al.  Towards reliable multimodal sensing in aware environments , 2001, PUI '01.

[8]  Marija Mikic-Rakic,et al.  Software Architectural Support for Handheld Computing , 2003, Computer.

[9]  James W. Hall,et al.  Uncertain inference using interval probability theory , 1998, Int. J. Approx. Reason..

[10]  Steve Shafer.  Ten Dimensions of Ubiquitous Computing , 2000 .

[11]  Building Plug-and-Play Networked Smart Transducers The Proposed IEEE P 1451 . 2 Standard Enables Transducer Manufacturers to Support Multiple Fieldbuses , 1999 .

[12]  Nenad Medvidovic,et al.  Modeling software architectures in the Unified Modeling Language , 2002, TSEM.

[13]  Terry Winograd,et al.  Architectures for Context , 2001, Hum. Comput. Interact..

[14]  Stan P. Woods.  The IEEE-P1451.2 Draft Standard For Smart Transducer Interface Modules , 1997 .

[15]  William G. Griswold,et al.  A component architecture for an extensible, highly integrated context-aware computing infrastructure , 2003, 25th International Conference on Software Engineering, 2003. Proceedings..

[16]  David Frankel,et al.  Model Driven Architecture: Applying MDA to Enterprise Computing , 2003 .

[17]  Kuo-Chu Chang,et al.  Modular neural net architecture for automatic target recognition , 1996, Defense, Security, and Sensing.

[18]  Albrecht Schmidt,et al.  Implicit human computer interaction through context , 2000, Personal Technologies.

[19]  Mary Shaw,et al.  A field guide to boxology: preliminary classification of architectural styles for software systems , 1997, Proceedings Twenty-First Annual International Computer Software and Applications Conference (COMPSAC'97).

[20]  Andy Hopper,et al.  Implementing a Sentient Computing System , 2001, Computer.

[21]  Lawrence A. Klein,et al.  Sensor and Data Fusion Concepts and Applications , 1993 .

[22]  David M. Kroenke,et al.  Database Processing: Fundamentals, Design, Implementation , 1991 .

[23]  Albrecht Schmidt,et al.  Multi-Sensor Context-Awareness in Mobile Devices and Smart Artifacts , 2002, Mob. Networks Appl..

[24]  Brad A. Myers,et al.  Handheld computing , 2003 .

[25]  Context-Aware Computing,et al.  Reconfigurable Context- Sensitive Middleware for Pervasive Computing , 2002 .

[26]  George J. Klir.  Uncertainty and Information Measures for Imprecise Probabilities: An Overview , 1999, ISIPTA.

[27]  Gregory D. Abowd,et al.  The Aware Home: A Living Laboratory for Ubiquitous Computing Research , 1999, CoBuild.

[28]  J. W. McManus.  Design and analysis tools for concurrent blackboard systems , 1991, IEEE/AIAA 10th Digital Avionics Systems Conference.

[29]  George Coulouris,et al.  Location Information Management , 2001, UbiComp.

[30]  Kari Sentz,et al.  Combination of Evidence in Dempster-Shafer Theory , 2002 .

[31]  S. James Press,et al.  Subjective and objective Bayesian statistics : principles, models, and applications , 2003 .

[32]  Philip A. Bernstein,et al.  Middleware: a model for distributed system services , 1996, CACM.

[33]  David Garlan,et al.  Software architecture: a roadmap , 2000, ICSE '00.

[34]  Kristof Van Laerhoven.  On-line Adaptive Context Awareness starting from low-level sensors , 1999 .

[35]  Gregory D. Abowd,et al.  The Location Service : A framework for handling multiple location sensing technologies , 2002 .

[36]  Charles Dean Haynie.  Development of a Novel Zero-Turn-Radius Autonomous Vehicle , 1998 .

[37]  Gregory D. Abowd,et al.  Rapid prototyping of mobile context-aware applications: the Cyberguide case study , 1996, MobiCom '96.

[38]  Craig Stoneking,et al.  Army Aviation Fusion of Sensor-Pushed and Agent-Pulled Information , 2001 .

[39]  Arthur C. Graesser,et al.  Is it an Agent, or Just a Program?: A Taxonomy for Autonomous Agents , 1996, ATAL.

[40]  I. Douven.  Inference to the Best Explanation is Coherent , 1998 .

[41]  Dov Dori.  CONCEPTUAL MODELING AND SYSTEM ARCHITECTING , 2003 .

[42]  D. Corkill.  Blackboard Systems , 1991 .

[43]  Erik Blasch,et al.  JDL level 5 fusion model: user refinement issues and applications in group tracking , 2002, SPIE Defense + Commercial Sensing.

[44]  Philippe Smets.  No Dutch Book can be built against the TBM even though update is not obtained by Bayes rule of conditioning , 1999 .

[45]  Ronald P. S. Mahler.  Combining ambiguous evidence with respect to ambiguous a priori knowledge. I. Boolean logic , 1996, IEEE Trans. Syst. Man Cybern. Part A.

[46]  Douglas K. Barry,et al.  Web Services and Service-Oriented Architecture: The Savvy Manager's Guide , 2003 .

[47]  H. Qi,et al.  Multi-Resolution Data Integration Using Mobile Agents in Distributed Sensor Networks , 2001 .

[48]  Chris J. Harris,et al.  Multisensor data fusion for obstacle tracking using neuro-fuzzy estimation algorithms , 1994, Defense, Security, and Sensing.

[49]  Gregory D. Abowd,et al.  Formalizing style to understand descriptions of software architecture , 1995, TSEM.

[50]  E. L. Waltz.  Information understanding: integrating data fusion and data mining processes , 1998, ISCAS '98. Proceedings of the 1998 IEEE International Symposium on Circuits and Systems (Cat. No.98CH36187).

[51]  Jani Mäntyjärvi,et al.  Managing Context Information in Mobile Devices , 2003, IEEE Pervasive Comput..

[52]  Michael Stal,et al.  An architectural view of distributed objects and components in CORBA, Java RMI and COM/DCOM , 1998, Softw. Concepts Tools.

[53]  David Garlan,et al.  Project Aura: Toward Distraction-Free Pervasive Computing , 2002, IEEE Pervasive Comput..

[54]  Glenn Shafer,et al.  A Mathematical Theory of Evidence , 2020, A Mathematical Theory of Evidence.

[55]  Rainer Stiefelhagen,et al.  Tracking and modeling focus of attention in meetings , 2002 .

[56]  Albrecht Schmidt,et al.  There is more to context than location , 1999, Comput. Graph..

[57]  David S. Rosenblum,et al.  Issues in supporting event-based architectural styles , 1998, ISAW '98.

[58]  Armando Fox,et al.  System Software for Ubiquitous Computing , 2022 .

[59]  Paola Inverardi,et al.  A Lightweight Infrastructure for Reconfiguring Applications , 2003, SCM.

[60]  Richard R. Muntz,et al.  Managing context data for smart spaces , 2000, IEEE Wirel. Commun..

[61]  Datong Chen,et al.  An Architecture for Multi-Sensor Fusion in Mobile Environments , 2006 .

[62]  Ted Kremenek,et al.  A Probabilistic Room Location Service for Wireless Networked Environments , 2001, UbiComp.

[63]  Gregory D. Abowd,et al.  Distributed Mediation of Imperfectly Sensed Context in Aware Environments , 2000 .

[64]  A.,et al.  A Formal Approach to Information Fusion , 1999 .

[65]  Wilfried Elmenreich,et al.  Sensor Fusion in Time-Triggered Systems , 2002 .

[66]  David Levine,et al.  PICO: A Middleware Framework for Pervasive Computing , 2003, IEEE Pervasive Comput..

[67]  Mongi A. Abidi,et al.  Data fusion in robotics and machine intelligence , 1992 .

[68]  S. Sitharama Iyengar,et al.  Distributed multi-resolution data integration using mobile agents , 2001, 2001 IEEE Aerospace Conference Proceedings (Cat. No.01TH8542).

[69]  James Llinas,et al.  An introduction to multi-sensor data fusion , 1998, ISCAS '98. Proceedings of the 1998 IEEE International Symposium on Circuits and Systems (Cat. No.98CH36187).

[70]  Hans-Werner Gellersen,et al.  Beyond Prototypes: Challenges in Deploying Ubiquitous Systems , 2002, IEEE Pervasive Comput..

[71]  Todor Tagarev,et al.  Computational Intelligence in Multi-Source Data and Information Fusion , 1999 .

[72]  Hairong Qi,et al.  Multisensor Data Fusion in Distributed Sensor Networks Using Mobile Agents , 2001 .

[73]  Ren C. Luo,et al.  Multisensor integration and fusion in intelligent systems , 1989, IEEE Trans. Syst. Man Cybern..

[74]  Gaetano Borriello,et al.  Location Systems for Ubiquitous Computing , 2001, Computer.

[75]  Allen H. Dutoit,et al.  A Software Architecture for Knowledge Acquisition and Retrieval for Global Distributed Teams , 2003 .

[76]  S. Sitharama Iyengar,et al.  Multiresolution data integration using mobile agents in distributed sensor networks , 2001, IEEE Trans. Syst. Man Cybern. Part C.

[77]  Gregory D. Abowd,et al.  Towards a Better Understanding of Context and Context-Awareness , 1999, HUC.

[78]  Gregory D. Abowd,et al.  The Human Experience , 2002, IEEE Pervasive Comput..

[79]  Jie Yang,et al.  Sensor Fusion Using Dempster-Shafer Theory , 2002 .

[80]  David S. Rosenblum,et al.  Design of a Scalable Event Notification Service: Interface and Architecture , 1998 .

[81]  Alexander H. Waibel,et al.  A real-time face tracker , 1996, Proceedings Third IEEE Workshop on Applications of Computer Vision. WACV'96.

[82]  Bing Ma,et al.  Parametric and nonparametric approaches for multisensor data fusion. , 2001 .

[83]  Andrew Martin Robert Ward,et al.  Sensor-driven computing , 1999 .

[84]  Jason Pascoe,et al.  Adding generic contextual capabilities to wearable computers , 1998, Digest of Papers. Second International Symposium on Wearable Computers (Cat. No.98EX215).

[85]  Richard N. Taylor,et al.  Using off-the-shelf middleware to implement connectors in distributed software architectures , 1999, Proceedings of the 1999 International Conference on Software Engineering (IEEE Cat. No.99CB37002).

[86]  Alexander H. Waibel,et al.  Estimating focus of attention based on gaze and sound , 2001, PUI '01.

[87]  Joëlle Coutaz,et al.  Foundations for a Theory of Contextors , 2002, CADUI.

[88]  Nenad Medvidovic,et al.  Domains of Concern in Software Architectures and Architecture Description Languages , 1997, DSL.

[89]  I.A. Essa,et al.  Ubiquitous sensing for smart and aware environments , 2000, IEEE Wirel. Commun..

[90]  Amy L. Murphy,et al.  A Declarative Approach to Agent-Centered Context-Aware Computing in Ad Hoc Wireless Environments , 2002, SELMAS.

[91]  Douglas C. Schmidt,et al.  Patterns for concurrent and networked objects , 2000 .

[92]  Mel Siegel,et al.  Sensor fusion using Dempster-Shafer theory II: static weighting and Kalman filter-like dynamic weighting , 2003, Proceedings of the 20th IEEE Instrumentation Technology Conference (Cat. No.03CH37412).

[93]  D. L. Hall,et al.  Mathematical Techniques in Multisensor Data Fusion , 1992 .

[94]  Robert M. Losee,et al.  A Discipline Independent Definition of Information , 1997, J. Am. Soc. Inf. Sci..

[95]  David Garlan,et al.  A Formal Approach to Software Architectures , 1992, IFIP Congress.

[96]  Robert L. Nord,et al.  Applied Software Architecture , 1999, Addison Wesley object technology series.

[97]  Jun-Hong Cui,et al.  Websigns: Hyperlinking Physical Locations to the Web , 2001, Computer.

[98]  David Howie.  Interpreting Probability: Probability up to the Twentieth Century , 2002 .

[99]  Michael S. Landy,et al.  A Statistical Viewpoint on the Theory of Evidence , 1986, IEEE Trans. Pattern Anal. Mach. Intell..

[100]  Richard T. Antony,et al.  Principles of Data Fusion Automation , 1995 .

[101]  Barry Brumitt,et al.  EasyLiving: Technologies for Intelligent Environments , 2000, HUC.

[102]  A. Dey.  Providing Architectural Suppor t for Building Context-Aware Applications , 2000 .

[103]  Luciano Baresi,et al.  Architectures for an Event Notification Service Scalable to Wide-area Networks , 2000 .

[104]  Albrecht Schmidt,et al.  Sensor-Based Context-Awareness for Situated Computing , 2000 .

[105]  Albrecht Schmidt,et al.  What GPS doesn't tell you: determining one's context with low-level sensors , 1999, ICECS'99. Proceedings of ICECS '99. 6th IEEE International Conference on Electronics, Circuits and Systems (Cat. No.99EX357).

[106]  Gaurav S. Sukhatme,et al.  Connecting the Physical World with Pervasive Networks , 2002, IEEE Pervasive Comput..

[107]  Robin R. Murphy,et al.  Sensor fusion , 1998 .

[108]  Bill N. Schilit,et al.  Context-aware computing applications , 1994, Workshop on Mobile Computing Systems and Applications.

[109]  Ronald Mahler,et al.  Unified data fusion: fuzzy logic, evidence, and rules , 1996, Defense, Security, and Sensing.

[110]  Jennifer A. Healey,et al.  Wearable and automotive systems for affect recognition from physiology , 2000 .

[111]  Dov Dori,et al.  System function and architecture: OPM-based definitions and operational templates , 2003, CACM.

[112]  Robert J. Allen.  A formal approach to software architecture , 1997 .

[113]  Jürg Kohlas,et al.  A Mathematical Theory of Hints , 1995 .

[114]  William Noah Schilit,et al.  A system architecture for context-aware mobile computing , 1995 .

[115]  Bill N. Schilit,et al.  Disseminating active map information to mobile hosts , 1994, IEEE Network.

[116]  David R. Morse,et al.  FieldNote: a Handheld Information System for the Field , 1999 .

[117]  Richard N. Taylor,et al.  A Component- and Message-Based Architectural Style for GUI Software , 1995, 1995 17th International Conference on Software Engineering.

[118]  N. Meyers,et al.  H = W. , 1964, Proceedings of the National Academy of Sciences of the United States of America.

[119]  Marvin H. J. Gruber.  The Subjectivity of Scientists and the Bayesian Approach , 2002, Technometrics.

[120]  Jay Gowdy,et al.  A Case Study for Outdoor Mobile Robots , 2000 .

[121]  Alan N. Steinberg,et al.  Revisions to the JDL data fusion model , 1999, Defense, Security, and Sensing.

[122]  Mark Stefik,et al.  Introduction to knowledge systems , 1995 .

[123]  Gaetano Borriello,et al.  Disappearing Hardware , 2002, IEEE Pervasive Comput..

[124]  John William McManus,et al.  Design and analysis techniques for concurrent blackboard systems , 1996, IEEE Trans. Syst. Man Cybern. Part A.

[125]  Scott F. Midkiff.  Mobile Computing "Killer App" Competition , 2002, IEEE Pervasive Comput..

[126]  Gregory D. Abowd,et al.  Providing architectural support for building context-aware applications , 2000 .

[127]  Gregory D. Abowd,et al.  Charting past, present, and future research in ubiquitous computing , 2000, TCHI.

[128]  James A. Landay,et al.  An Infrastructure Approach to Context-Aware Computing , 2001, Hum. Comput. Interact..

[129]  Gregory D. Abowd,et al.  An Architecture to Support Context-Aware Applications , 1999 .

[130]  Joseph Y. Halpern,et al.  A Logic for Reasoning about Upper Probabilities , 2001, UAI.

[131]  James L. Crowley,et al.  Perceptual Components for Context Aware Computing , 2002, UbiComp.