Thurstonian Cognitive Models for Aggregating Top-n Lists

In a top-n task, people produce a list of items that they believe are ordered relative to a criterion, and can include any number of items in their list. We develop Thurstonian cognitive models of the individual differences and decision-making processes involved in producing top-n lists, and apply it to the problem of inferring an aggregated list from individual responses. We present 3 applications—involving predicting movie popularity, predicting the outcome of the 2014 World Cup tournament, and rating the worst U.S. presidents—using real-world data from the crowd-sourced opinion website ranker.com. The movie popularity application demonstrates the ability of the model to make relatively accurate predictions, partly through its ability to infer individual expertise. The World Cup application demonstrates the ability of the model to make accurate predictions, partly through its ability to incorporate relevant prior information, and partly through its ability to model multiple relevant behavioral data jointly. The U.S. presidents application demonstrates the ability of the model to allow for multiple latent rankings, so that a subgroup of contaminant opinions can be separated from those that reflect established historical opinion. On the basis of these applications, we argue that the psychological goals of developing models of how people produce top-n lists, and the applied goals of making accurate predictions, are usefully tackled simultaneously.