The Situation Assessment Problem: Toward a Research Agenda

ions at which potentially useful assessments can be made, depending on the role and the level of authority and responsibility of the decision-makers. • In addition to multiple functions and echelons, individual tastes, preferences and command styles of the human decision-makers produce a strong impact on how an assessment should be performed. An Assessment System requires models of individual decision-makers, both automated and human. • Most of today’s information technologies are strongly dependent on models of the phenomena or systems that they serve or analyze. However, one must question the availability and applicability of models in the military domain, such as models of Battlespace and Command. Although certain aspects of weapons and forces have been extensively modeled, comprehensive models of Battlespace and Command do not exist. Perhaps even more importantly, one may question if any model of military matters, which by necessity are biased toward “the last war,” will ever be applicable to “the next war.” Perhaps we must look for model-independent approaches, or for means to construct models automatically and dynamically. We will return to this point later. 6. Current Research Directions: Applicability and Gaps to Overcome In this section we discuss a number of research fields from the perspective of the Assessment Problem. For each field we discuss (a) the potential contributions of this research into the Assessment Problem and (b) the limitations of the research that must be overcome or mitigated in order to apply it to the Assessment Problem. Research field: Model-Based Diagnosis, e.g., [1], [2]. Potential contributions of this research to the Assessment Problem: approaches to inferencing the requirements for specific information from the decision-making model or from the system model Current limitations of this research: 1. MBD research has not addressed the need for focusing on a specific most valuable subset of the available information. 2. The most significant similar current use of MBD is in Robotics and most work tends to be of ad hoc nature. 3. There is limited understanding of how MBD can be applied in real-time, resource-bound situations. This topic deserves a more detailed discussion. The work on Model-Based Diagnosis has made fairly significant progress in this respect: significant enough that it’s being used in Deep Space One, which does involve a dynamically changing situation and real-time constraints [2]. However, getting MBD to work for DS1 involves an extensive amount of engineering; moreover the designers of the MBD component were also working closely with the systems engineers, so that they fully understood the performance of the system, and to some extent could control it. In a warfare situation, no side can have a complete understanding of the enemy behavior, and they certainly don’t have much control over it in advance Research field: Execution Monitoring, particularly the recent work on Rationale-Based Monitoring, e.g., [10]; [7], [9], [8]. Potential contributions of this research to the Assessment Problem: approaches to determining most valuable information to monitor Current limitations of this research: Only very preliminary work has been done on this so far; further, it has so far only been tested as a technique for monitoring during the planning process, not for monitoring during execution. Although the latter is an intended extension of the idea, it hasn’t been implemented and analyzed yet. Also, it hasn’t addressed the question of links between different monitors (i.e., the “if you see X, then check Y” issue.) Research field: Machine Learning and Data Mining, e.g., [21], [22], [23]. Potential contributions of this research to the Assessment Problem: Machine learning and data mining techniques provide a way to discover significant patterns in the massive amount of data that are being input to the assessment system. ML/DM techniques divide into three main classes: supervised, reinforcement, and unsupervised. In supervised learning, the system is given immediate feedback about what the “correct” answer was. In reinforcement learning, the system occasionally receives rewards or penalties, but not direct feedback about the correct answer. In unsupervised learning, the system must identify clusters of data that are similar in some way, without external input about the quality of its results. “Data mining” is a term used to describe the application of machine learning techniques to very large, typically distributed, databases, which may contain rich implicit regularities. One form of data mining that may be particularly relevant to assessment systems involves learning the structure of Bayesian networks. If Bayesian networks could be inferred from the incoming data, they could provide a significant amount of assessment information. ML/DM approaches are particularly attractive to the Assessment Problem because they are at least partially model-independent. Current limitations of this research: Supervised learning may be problematic for the assessment problem, because it requires human input to provide correct results during training, but often the human will not understand enough about the situation to do so. (If we had this knowledge, we wouldn’t need an assessment system at all: we could just have a human assistant perform the assessment.) Reinforcement learning could in principle address this problem, but reinforcement learning is computationally complex, and often doesn’t converge until after a large amount of experience has been gained. In a highly dynamic environment, a reinforcement learning algorithm may not have time to learn a model before it has changed. The highly dynamic nature of the environment will also pose a challenge for unsupervised learning techniques; in addition, they do not address the real-time issues (i.e., they do not include explicit mechanisms for trading solution quality against computation time). Although in principle data-mining approaches have an important advantage of being model-independent, unless the system has some idea of what it is looking for, there are likely to be vastly too many patterns found in the data. Thus, some form of model is needed to focus on what’s interesting, and identify “useful” information. Research field: Aggregation of Dynamic Modes [24], [25]; [26]; [27]. Potential contributions of this research to the Assessment Problem: Techniques for creating reduced-order models reflecting the significant dynamics for assessment Current limitations of this research: Principally based on off-line analysis of detailed analytical models. Research field: Recursive State Estimation (Kalman Filters) Potential contributions of this research to the Assessment Problem: On-line estimation of the quantified elements of the current state of the world Current limitations of this research: strictly numerical, limited robustness results (robustness with respect to model inaccuracies; confidence measures depend on priors); highly model-dependent; issue of selective relevancy is not addressed. Research field: Model Validation, e.g., [28]. Potential contributions of this research to the Assessment Problem: determining quality of models from on-line data and selecting which of the possible models of the system is applicable given the observations Current limitations of this research: principally based on linear dynamic models (in the dynamic systems literature); alternative literature exists for discrete event simulation and training (training set / validation set concepts) Research field: System Identification. There is a very rich literature in this field, and many successful applications. Potential contributions of this research to the Assessment Problem: recursive numerical approaches to estimating parameters of the system model Current limitations of this research: Quantitative, modelbased in that the model structure needs to be given; has not been explored in application to systems in which hostile and deceptive agents are present. Research field: Plan and Intent Recognition research Potential contributions of this research to the Assessment Problem: approaches to recognition of enemy plans / intent Current limitations of this research: current work has focused on non-hostile agents; little work has been done with respect to hostile, deceiving agent’s plan and intent Research field: Decision Theory, e.g., [4], [6]. Potential contributions of this research to the Assessment Problem: determining value of information Current limitations of this research: real-time aspects are not addressed. While DT literature deals rather explicitly with certain temporal aspects, such as the time or stage of the decision process at which the information will be received, it doesn’t consider deadline-type constraints. Research field: Contingency Planning, e.g., [12], [13], [14], [15]. Potential contributions of this research to the Assessment Problem: understanding the expected contingencies and the possible responses to the contingencies provides basis for deciding what is worth observing Current limitations of this research: Computing the expected impact on the plan of all contingencies is computationally very expensive; known techniques will probably not scale up to full-fledged military scenarios. Also, the real-time issues have not yet been addressed in this work. In this connection, we also point out an emerging research direction: integration or middle ground between AI planning and MDP planning. These two views are on the opposite ends of the spectrum of approaches to the Assessment Problem. The former views alternative actions (and required observations) as an outcome of the contingency planning process; the latter uses abstraction / aggregation / envelope techniques to define every action (and required observations) as a function of the set of states. We join a number of researchers in both of the two communities who believe that there is an opportunit