Applying One Reason Decision-making: The Prioritisation of Literature Searches

The prioritisation of literature searches aims to order the large numbers of articles returned by a simple search so that the ones most likely to be relevant are at the top of the list. Prioritisation relies on having a good model of human decision-making that can learn from the articles users select as being relevant to make predictions about which of the remaining articles will be relevant. We develop and evaluate two psychological decision-making models for prioritisation: A "rational" model that considers all of the available information, and a "one reason" model that uses limited information to make decisions. The models are evaluated in an experiment where users rate the relevance of every article returned by PsyclNFO for a number of different research topics, with the results showing that both models achieve a level of prioritisation that significantly improves upon the default ordering of PsycINFO. The one reason model is shown to be superior to the rational model, especially when there are only a few relevant articles. The implications of the results for developing prioritisation systems in applied settings are discussed, together with implications for the general modeling of human decision-making. yvTh en a researcher first does a literature search, they Wv usually are only able to supply general search criteria, such as one or two keywords, to indicate their broad topic of interest. Typically, these initial searches will return a large number of potentially relevant articles. Faced with this information overload, one option for the researcher is to refine their search, and hope that a more manageable list of articles is returned. Often, however, this refinement is difficult, because the researcher is unsure exactly what sorts of materials are available, and there is a need to "sample" or "explore" the large initial list of articles before a more detailed search can be constructed with any confidence. Prioritisation offers a different approach to dealing with the information overload. The basic idea is to begin presenting the articles, requiring the user to indicate whether or not that article is of interest. As each article is examined, prioritisation acts to re-order the remaining articles so that the relevant ones are placed at the top of the list. If prioritisation is effective, the problem of information overload is solved without the user ever having to construct a refined search. They only need to work from the top of the prioritised list until they reach the point where the articles are no longer of sufficient relevance to be worth pursuing. While the prioritisation problem has been tackled in a variety of information retrieval contexts using machine learning techniques (e.g., Balabanovic, 1998; Macskassy, Dayanik, & Hirsch 1999; Mehran, Dumais, Heckerman, & Horvitz 1998), it has typically not been tackled from a cognitive modeling perspective. This is unfortunate, because prioritisation rests on the ability to predict whether or not a user will evaluate an article as a relevant one, and so requires an effective model of human decision-making to be successful. In this paper, we develop and evaluate two cognitive models for the prioritisation of literature searches. One is a "rational" model, that performs exhaustive calculations, while the other is a "one reason" model, that requires only limited time by making assumptions about the nature of its environment. In the next section, we describe how literature searches are represented by these models, and how information about them is learned. We then describe the two models in detail, before presenting the results of an experiment where both are evaluated on real-world data. Finally, we draw some conclusions regarding the theoretical implications of the results for understanding human decision-making, and the applied implications for building a literature search prioritisation system.