Adapting to the User , s Time and Working Memory Limitations: New Directions of Research

As interactive computing devices become increasingly mobi le, the user ,s time limitations and situational distractions are becoming important deter minants of the quality of interaction with these devices. Automatic adaptation to such situa tion l constraints is a challenging goal for user modeling research. This paper first briefly s ummarizes research in the first three-year phase of the R EADY project, in which a prototype of such a system was developed. It then discusses four new lines of research that will be pursued in this project during its second phase: extension to multimodal interacti on; allowing for different degrees of granularity in the modeling of the user; learning th e system,s Bayesian networks automatically from empirical data; and dealing with more co mplex discourse structures. Computing power is moving off of the desktop and into the hustle and bustle of everyday life. This trend generates new challenges and opportunities for those who develop ada ptive systems. Users of a desktop computer may have to cope with deadlines and distrac tions, but on the whole they can measure their time in minutes and hours, rather than in sec ond ; and they can typically concentrate fully on whatever task they are using the comput er for at the moment. By contrast, some of the more recent additions to the family of computing devi ces— such as hand-held computers and smart mobile phones—often constitute just one element i n a complex environment that the user has to deal with. An interaction may be have to be completed within seconds, while the user is simultaneously conducting a conversation and/or na vigating through unfamiliar territory. In such a situation, two critical factors t hat influence the success of interaction with the system are: 1. the amount of time that the user ( U ) has available; and 2. the extent to whichU can concentrate on the interaction, as opposed to dealing with situational distractions. Somewhat more abstractly, we can speak of bottlenecks caused by temporary li mitations of the user,s resourcesof time and working memory. 1 It is not obvious that the user ,s temporary resource limitations need to be recognized and adapted to automatically. First of all, systems of this sort are typicall y designed from the start in such a way that they can be used quickly and without much mental effort. But no si ngle type The research outlined in this paper will be funded from 1999 though 200 1 by the German Science Foundation (DFG) in its Collaborative Research Center on Resource-Adaptive Cog nitive Processes, SFB 378, Project B2, READY. The principal investigators are Wolfgang Wahlster and Anth o y Jameson. The complete text of the research proposal (Wahlster & Jameson, 1998a) is available on request . The project Web page is http://w5.cs.uni-sb.de/ ̃ready/ . 1The Collaborative Research Center on Resource-Adaptive Cognitive Processes , where the research discussed here is being conducted, investigates ways in which the performance of cogni tive systems adapts to limitations in resources of various types (see http://www.coli.uni-sb.de/sfb378/ ). of interaction is optimal for all configurations of resource availability. For e xample, if a system is designed for the worst case of a rushed and distracted user, it will probably m ke too little use of the user ,s resources when more of these are available. What about the strategy of making these systems adaptablerather thanadaptive? For example, the designers could give users the opportunity to choose, in each situation, how the y would like to enter their input to the system and what type of output they desire. Unfortunate ly, a user who is rushed and/or distracted is in a poor position to adapt his system use accordingl y—or even to signal the problem to the system. In sum, it seems to be a reasonable goal to enable some types of interactive comput ing devices to do what people do when interacting with others: The device should assess , without being told, roughly what resources the person has available, and it should adapt its beha vior accordingly. To be sure, realizing such automatic adaptation is a challenging task, for the f ollowing reasons, among others: 1. Theevidencethat can permit a system ( S) to recognize resource limitations tends to be scant and not very reliable. A usefully accurate assessment is often only pos sible when several pieces of evidence can be combined. 2. Thetemporal variabilityof these resource limitations means that S is tracking a poorly visible moving target. 3. Even if the user ,s resource availability is perfectly known, it is often hard for a system to decidewhat type of adaptation is appropriate . For example, an especially concise instruction may initially save a user some time, but if it is too dense do be unde rstood completely, it may in the end cause the user to waste time or make errors. The READY project, which began in 1996, represents an initial effort to meet these challenges. This paper will first briefly summarize the research of the first thr ee-year phase. 2 It will then discuss the new directions of research that will be pursued during the se cond phase. The Current Prototype The example scenario that was used during the first phase of research is illust rated in Figure 1: Users are drivers whose cars need minor repairs; they request assist ance from the system in natural language via mobile phone. Exploratory empirical studies in this scenario (and with a different scenario involving emergency calls to a fire station) provided s ome background information concerning the following points: Causesof resource bottlenecks: factors (such as the difficulty of the current task, or the need to perform another task simulteneously) that create a situation in which the user ,s resources may not be adequate for successful interaction. Symptomsof resource bottlenecks: aspects of the user ,s behavior (e.g., speech disfluencies) that indirectly indicate the extent to which U ,s resources are adequate to meet the current demands. Strategiesfor dealing with resource bottlenecks: general ways in which systems (or persons) can adapt their behavior when a resource bottleneck is perceived. 2Jameson, Schäfer, Weis, Berthold, and Weyrath (in press) give a recent Eng lish-language overview of the research of this phase. Schäfer, Weis, Weyrath, and Jameson (1997) presented an earlier overview in German. The report by Wahlster and Jameson (1998b) is available on request. Papers pres nted at previous ABIS workshops include those by Schäfer and Weyrath (1996) and by Berthold (1997). Figure 1: Illustration of READY,s first example scenario, showing the similarity of R EADY,s role to that of an auto repairman who offers advice by phone. These studies guided the development of the initial R EADY prototype. The basic workings of the prototype can be explained with reference to the sketch of the architectur e in he left-hand side of Figure 2. The User Interface is designed so as to make it unnecessary for R EADY to deal with the challenging problems of speech processing raised by the scenario. Input is done via a nat ural language menu interface with which the “user” can compose utterances and speci fy a number of aspects of their form, such as the position and length of pauses. What is sent to theDialog Management component is a representation of U ,s utterance that contains the information that R EADY needs in order to update its user model and determine an appropriate response, i.e.: the meaning of U ,s utterance in the current dialog context plus any evidence that can be extracted from the input (including noises from the environment) t hat bears onU ,s current resource limitations. TheDialog Management component uses its knowledge about the current dialog state and about possible diagnosis and repair actions to determine a set of possible dialog contri buti s (e.g., instructions) that might make sense in the current situation—not yet taking into account U ,s resource limitations. These contributions may differ in their basic content (e.g., prescribing a simple or a complex action) and/or in their form (e.g., using simple, redundant formula tions or concise, technical ones). The Dialog Management component then sends the possible contributions to theUser Modeling component, which decides which one seems best in the light of U ,s current resource limitations. The User Modeling component employs dynamic Bayesian networks as a representation and processing formalism, because they are well suited to dealing with (a ) t e large amount of uncertainty involved in inferences about a user , resource limitations and (b) the dynamic nature of these resource limitations, which often change quite rapidly. Network Schemata represent assumptions about the causal relationships between resource limitations and thei r caus s and symptoms. Figure 3 gives an idea of the way in which this prototype adapts to a “user ,s” perceived resource limitations. The numbers on the right are the expected values of the syste m, current assessments of key variables, including available WM capacity ( vAGK) and time pressure ( Zeitdruck). In this case, the user ( P) has just input the problem description Kühlwasser-Warnlampe