Understanding Engagement through Search Behaviour

Evaluating user engagement with search is a critical aspect of understanding how to assess and improve information retrieval systems. While standard techniques for measuring user engagement use questionnaires, these are obtrusive to user interaction, and can only be collected at acceptable intervals. The problem we address is whether there is a less obtrusive and more automatic way to assess how users perceive the search process and outcome. Log files collect behavioural signals (e.g., clicks, queries) from users on a large scale. In this paper, we investigate the potential to predict how users perceive engagement with search by modelling behavioural signals from log files using supervised learning methods. We focus on different engagement dimensions (Perceived Usability, Felt Involvement, Endurability and Novelty) and examine how 37 behavioural features can inform these dimensions. Our results, obtained from 377 in-lab participants undergoing goal-based search tasks, support the connection between perceived engagement and search behaviour. More specifically, we show that time- and query-related features are best suited for predicting user perceived engagement, and suggest that different behavioural features better reflect specific dimensions. We demonstrate the possibility of predicting user-perceived engagement using search behavioural features.