How to Harvest Collective Wisdom on Complex Problems: An Introduction to the MIT Deliberatorium

Humanity now finds itself faced with pressing and highly complex problems – such as climate change, the spread of disease, international and economic security, and so on that call upon us to bring together large numbers of experts and stakeholders to deliberate collectively on a global scale. Such large-scale deliberations are themselves complex processes, however, with emergent properties that often prevent us from adequately harnessing the community's collective wisdom. Collocated meetings, beside being expensive, are prone to such well-known effects as polarization, power dynamics, and groupthink. Social media (such as email, blogs, wikis, chat rooms, and web forums) enable more concurrent input but still typically generate more heat than light when applied to complex controversial topics. Large-scale argumentation systems represent a promising approach for addressing this important challenge, by virtue of providing a simple systematic structure that radically reduces redundancy and encourages clarity. This paper will describe the efforts we have made to explore this approach, giving an overview of the key underlying concepts, the ways we have translated these concepts into a working system (the MIT Deliberatorium), and the experiences we have had evaluating this system in a range of contexts. The Challenge Decision-making in large communities rarely fully harvests the collective wisdom of its members, even for very high-stakes problems where it can make the difference between disaster and success. It can be simply too expensive to bring the key players into one room, and too difficult to manage the interactions of large groups to get the best that the members have to offer. Only one person can talk at a time, loud voices can dominate a discussion, and emergent dynamics can lead such groups to either deadlock without a solution (polarization) or prematurely settle on a solution without sufficiently exploring the space of promising alternatives (groupthink) [1]. In recent years, social media technologies (e.g. email, web forums, chat rooms, blogs, wikis, and idea forums) have emerged that have the potential to address this important gap. Such tools have enabled diverse communities to weigh in on topics they care about at unprecedented scale, in turn leading to remarkably powerful emergent phenomena [2] [1] [3] [4] such as: • Idea synergy: when users share their creations in a common forum, it can enable a synergistic explosion of creativity as people develop new ideas by combining and extending ideas that have been put out by others. • The long tail: social computing systems enable access to a much greater diversity of ideas than they would otherwise: “small voices” (the tail of the frequency distribution) that would otherwise not be heard can now have significant impact. • Many eyes: social computing efforts can produce remarkably high-quality results by virtue of the fact that there are many eyes continuously checking the shared content for errors and correcting them. • Wisdom of the crowds: large groups of (appropriately independent, motivated and informed) contributors can collectively make better judgments than the individuals that make them up, even experts, because their collective judgment cancels out the biases and gaps that individuals are prone to have on their own. Social media technologies often create, however, more heat than light when applied to complex, controversial problems: • Disorganized content: Existing social media generally create very disorganized content, so it's timeconsuming to find what has been said on any topic of interest. This fosters unsystematic coverage, since users are often unable to quickly identify which areas aren't yet well-covered and need more attention. • Low signal-to-noise ratio. Social media content is notorious for producing highly redundant content, so important points often got lost in the crowd. • Quantity rather than Depth. Social media systems often elicit many relatively small contributions, rather than a smaller number of more deeply-considered ideas, because collaborative refinement is not inherently supported. • Polarization: Users of social media systems often self-assemble into groups that share the same opinions, so they see only a subset of the issues, ideas, and arguments potentially relevant to a problem. People thus tend to take on more extreme, but not more broadly informed, versions of the opinions they already had. • Dysfunctional argumentation: Existing social media systems do not inherently encourage or enforce well-considered argumentation, so postings are often biasrather than evidenceor logic-based. Social media technologies thus often capture only a fraction of the collective wisdom of a community, and enormous effort is typically required to "harvest" this wisdom to inform better, more broadly-supported decisions. Intel, to give a typical example, ran a web forum on organizational health that elicited a total of 1000 posts from 300 participants. A post-discussion analysis team invested over 160 person-hours to create a useful summary of these contributions (at 10 minutes a post, probably longer than it took to write many of the posts in the first place). The team found that there was lots of redundancy, little genuine debate, and few actionable ideas, so that in the end many of the ideas they reported came from the analysis team members themselves, rather than the forum. It could be argued that many of these concerns are addressed by topic-centric tools such as wikis, where each discussion topic is captured in it’s own unique, collaborativey-authored, article. But wikis are deeply 1 Based on personal communication with Catherine Spence, Information Technology Enterprise Architect, Computing Director/Manager at Intel. challenged by complex and controversial topics [5] [6]. They capture, by their nature, the “least-commondenominator” consensus between many authors (any non-consensus element presumably being edited out by those that do not agree with it). This consensus is often achieved only by dint of wasteful “edit wars” as different authors repeatedly undo each other’s contributions, and the controversial core of deliberations are typically moved to massive talk pages for the article, which are essentially time-centric venues prone to all the limitations we noted above. To give a concrete example, at the time of writing, the wikipedia “global warming” article was 6500 words long, but the discussion area for this article consisted of 64 archives, each ranging from 2,000 to 60,000 words in length. Articles for more controversial topics are often “locked” altogether to prevent sabotage. Indeed, if we look at the effort users allocate to different wikipedia activities, it appears that the site is increasingly being bogged-down in activities that deal with contention over article contents: Changing percentage of edits over time in Wikipedia, showing decreasing direct work (article edits) and increasing indirect work (article talk and so on). [5] Another promising social media technology is question-centric tools such as Dell’s Ideastorm.com, the Obama administrations’ Open for Questions web site, solution contest sites such as innocentive.com, and Google’s project10tothe100.com. Such tools are organized around questions: a question is posted and the community is asked to contribute ideas for how to answer that question. Such sites can elicit huge levels of activity – the Obama site for example elicited 70,000 ideas and 4 million votes in three weeks – but they are prone to several serious shortcomings. One is redundancy: in all of these sites, many of the ideas represent minor variations of each other. When the volume of posts is large, important ideas that happen to appear relatively few times can be easily overlooked. Pruning such lists to consolidate equivalent posts is, moreover, a massive undertaking. In Google’s case, for example, the company had to recruit 3,000 employees to filter and consolidate the 150,000 ideas they received in a process that put them 9 months behind their original schedule. One could argue that asking the user community to rate submissions can allow the best ideas to rise to the top, but unfortunately ratings systems are prone to dysfunctional ranking dynamics in such contexts. When people are asked to rate a very long list of items, one can expect that the system will quickly “lock” into a fairly static, and arbitrary, ranking. People are more likely to vote for ideas that already have high ratings and, if there are thousands of ideas, people will in all likelihood stop looking after the first few. So the first few winners take all, even if they are inferior to many other ideas in the list. Researchers observed this property when looking at music markets [7]: when people could see each other’s rankings, much of the ranking results were essentially random. A final issue is depth vs breadth. Idea aggregation sites tend to elicit many fairly simple ideas. The ideas generated by the google project, for example, (e.g. make government more transparent, help social entrepreneurs, support public transport, create user-generated news services) were in large part not novel and light on detail. The Better World Campaign used an idea sharing system to come up with a proposal for Obama’s first actions upon entering office. After a year of debate, over 5,000 idea submissions, and close to one hundred thousand votes, the result was the proposal that Obama plant an organic garden at the White House. Surely that massive amount of effort could have been used to compose a smaller number of more deeplyconsidered ideas, but idea aggregation sites provide no support for this, because people can not collaboratively refine submitted ideas. To summarize: social media technologies enable unprecedented levels of community input but often capture only a fraction of the community’s collective wisdom, especially with complex and controv