Preface Organizing Committee Program Committee Usefulness as the Criterion for Evaluation of Interactive Information Retrieval Systems Semi-supervised Priors for Microblog Language Identification Scope of Negation Detection in Sentiment Analysis a Multi-dimensional Model for Search Intent Result Div

DIR 2011, the 11th Dutch-Belgian Information Retrieval Workshop, was organized by the Information and Language Processing group (ILPS) of the University of Amsterdam in collaboration with the Centrum Wiskunde en Informatica (CWI). Two types of submissions were accepted for the workshop: research papers describing original research, compressed contributions presenting a summary of previously published work, and demonstrations. There were many people who helped organize DIR 2011, making it a success. We would like to thank them all. In particular, we are gratefull to our keynote speakers, Nick Belkin (Rutgers University) and Gabriella Kazai (Microsoft Research). Relevance has been the classic criterion for evaluation of the effectiveness of information retrieval (IR) systems since the earliest days of IR system evaluation. This criterion has been understood as the ability of an IR system to recognize documents relevant to a person's " information need " , and understood as the ability of the system to provide to the person all of the documents in an information resource relevant to that need, and only those documents relevant to the need. The measures of effectiveness of the system have thus been understood as recall and precision. These measures have been applied in the evaluation of the performance of an IR system as referring to the system's ability to maximize these measures in its response to a single query (representation of the information need) put to the system. This criterion, these measures, and the application of the measures depend crucially on both a specific model of IR, and a specific model of the user's desired results, both of which are based on the example of the special purpose bibliography of a topic constructed on demand by documentalists and science librarians in the early and middle 20th century. In this presentation, I argue that the criterion, measures, and application of those measures based on this example are inappropriate for the general interactive IR situation and evaluation of interactive IR systems, and propose that the usefulness of the IR system in supporting the goal or task which led the person to engage in information seeking should be the basic criterion according to which an IR system is evaluated. In particular, I argue that the relevance criterion and its associated measures cannot be used alone to evaluate the performance of an IR system over an information seeking episode, and that usefulness is a criterion which can be used to …