How do users evaluate individual documents? An analysis of dimensions of evaluation activities

Introduction. Evaluation plays an important role in users' information searching and retrieving processes. While previous research mainly focuses on applied criteria, less research is on other dimensions of evaluation. This study explores the dimensions of evaluation activities including criteria applied, elements examined, activity engaged in and time spent in evaluating individual documents. Method. Conducting their own tasks, thirty-one participants with different demographic backgrounds, participated in the study. Multiple data collection methods were used: pre-questionnaires, interaction diaries, think-aloud protocols, transaction logs and post-questionnaires. Analysis. Types of evaluation criteria, elements, evaluation activities and their associated pre- and post-activities were analysed by open-coding. Descriptive analysis was applied to analyse the most and leasdt time spent on individual documents with associated factors. Results. The findings of this study reveal that evaluation activities are complicated and dynamic. Eighteen types of evaluation criteria, seven types of evaluation elements, six types of evaluation activities and their associated pre- and post-activities were identified. In addition, factors affecting time spent in evaluating individual documents were analysed. Conclusions. The authors offer suggestions for information retrieval system design to support effective evaluation.