Assessing regret-based preference elicitation with the UTPREF recommendation system

Product recommendation and decision support systems must generally develop a model of user preferences by querying or otherwise interacting with a user. Recent approaches to elicitation using minimax regret have proven to be very powerful in simulation. In this work, we test both the effectiveness of regret-based elicitation, and user comprehension and acceptance of minimax regret in user studies. We report on a study involving 40 users interacting with the UTPref Recommendation System, which helps students navigate and find rental accommodation. UTPref maintains an explicit (but incomplete) generalized additive utility (GAI) model of user preferences, and uses minimax regret for recommendation. We assess the following general questions: How effective is regret-based elicitation in finding optimal or near-optimal products? Do users understand and accept the minimax regret criterion in practice? Do decision-theoretically valid queries for GAI models result in more accurate assessment than simpler, ad hoc queries? On the first two issues, we find that the minimax regret decision criterion is effective, understandable, and intuitively appealing. On the third issue, we find that simple, semantically ambiguous query types perform as well as more demanding, semantically valid queries for GAI models. We also assess the relative difficulty of specific query types.