Evaluation Games - How to Make the Crowd your Jury

This paper introduces the application of online evaluation games as method to elicit promising contributions in innovation contests. The “style your smart” design contest serves as field experiment to explore applicability and use of games for the evaluation of designs. Results indicate that online evaluation games help identifying the most promising designs, while being limited to submissions with certain characteristics and not being free of fraud. 1 Innovation Contests Pushed through concepts like crowdsourcing [Ho08, KHS08], co-creation [Wi05], and open innovation [Ch03], firms increasingly use the creativity, skills, and intelligence of billions of individuals encountered on the Internet as source for innovative ideas. Building on the means of competition, innovation contests are one particular method to do so. In innovation contests a company or institution posts a challenge it is facing. Subsequent interested individuals contribute to the challenge by offering potential ideas or solutions to the problem. These are assessed and winners granted a prize. While such contests ensure a large variety of submissions, the identification of the best and most promising ones often causes large efforts. On the one hand, the mere magnitude of ideas generated can be overwhelming, on the other hand most approaches do not increase the chance of really selecting the best submissions nor do they reduce the risk of relaying on the wrong ones [BW08]. The existence of social media and new information and communication technologies (ICT) offers new opportunities to encounter this challenge. Participants can share their ideas, communicate with each other, establish relationships and even comment and evaluate others ideas. The latter both are also referred to as open evaluation [Ha10; MHB10]. Recent research indicates that open evaluation bears plenty of potential to support the selection of relevant submissions [MHB10; BBHLK09]. Still, it is also recognized that many methods are prone to fraud. By using multiple accounts, participants can vote for themselves to increase their chance of winning or by voting down competitors respectively. Hence, effective open evaluation has to avoid these pitfalls, while still tapping into the wisdom of the crowd. Online games seem to be promising approach. First experiments show the suitability of online games to elicit users’ preferences, while making it harder to cheat [HA09]. This paper introduces and discusses the use of games with a purpose for the evaluation and identification of most