Hybrid human-computing distributed sense-making: extending the soa paradigm for dynamic adjudication and optimization of human and computer roles

In many evolving systems, inputs can be derived from both human observations and physical sensors. Additionally, many computation and analysis tasks can be performed by either human beings or artificial intelligence (AI) applications. For example, weather prediction, emergency event response, assistive technology for various human sensory and cognitive impairments, individual and community medical systems, energy efficient buildings/processes, and a host of other complex management and sense-making applications have the potential to be implemented as hybrid human/computing systems in which: (1) observational data can be provided by either physical sensors or humans acting as observers (or a combination of such input), and (2) sense-making can be performed by either automated inference algorithms (computer automated reasoning/pattern recognition) or by human cognition (or both). This category of hybrid system (referred to as “hard and soft information fusion”) has wide-ranging promise for analysis of both physical data and abstract concepts. However, there are many challenges related to the effective storage, representation, and transmission of the vastly heterogeneous data necessary for scalable, loosely-coupled service-based communication between physical sensors, human observers, AI-based machine cognition tools, and human analysts. Additionally, there is currently a lack of techniques for adjudicating which tasks should be assigned to humans and which should be assigned to machine/ computer systems. This research explores the current state of the art in distributed hard and soft information fusion and seeks to address the above-mentioned gaps and challenges through a novel integration of paradigms and techniques such as service oriented architecture (SOA), multi-agent software systems (MAS), complex event processing (CEP), sonification (auditory display), message oriented middleware (MOM), and community standard data representation. Additionally, it provides a prototype system implementation and a simulation experiment to evaluate the efficacy of the proposed techniques.