Finding Place in a Design Space

Many organizations have adopted design processes that integrate community voices to discover the real problems that communities face. Online discussion forums offer a familiar and flexible technology that can help facilitate discussion around problems and potential solutions. However, we lack understanding about what information community members share, how that information is structured, and how social interactions affect design processes at scale. This paper presents a mixed-methods analysis of Canvas, a learning management system, which enables users to contribute to the design of the platform by sharing and deliberating on problems and solutions in a discussion forum. We collected and analyzed 1412 ideas and 18,335 associated comments shared on the Canvas discussion forum. We found that the distributed nature of design information, the presence of duplicate ideas, and contributors' gaming behaviors made it difficult for the community to make sense of the design discussion. These gaming behaviors also constitute a new concern for participatory design research. Finally, we reflect on how Canvas community members contribute information to a shared design space and how future systems could more effectively coordinate community design efforts.

[1]  Martin A. Andresen Asynchronous Discussion Forums: Success Factors, Outcomes, Assessments, and Limitations , 2009, J. Educ. Technol. Soc..

[2]  J. Füller,et al.  Why Co‐Creation Experience Matters? Creative Experience and its Impact on the Quantity and Quality of Creative Contributions , 2011 .

[3]  Jennifer Ann Rode,et al.  Reflexivity in digital anthropology , 2011, CHI.

[4]  Celine Latulipe,et al.  IneqDetect: A Visual Analytics System to Detect Conversational Inequality and Support Reflection during Active Learning , 2019, CompEd.

[5]  John P. Kretzman,et al.  Building Communities from the Inside Out: A Path Toward Finding and Mobilizing a Community's Assets , 1993 .

[6]  Yann Mathet,et al.  The Unified and Holistic Method Gamma (γ) for Inter-Annotator Agreement Measure and Alignment , 2015, CL.

[7]  Michael S. Bernstein,et al.  Context Trees: Crowdsourcing Global Understanding from Local Views , 2014, HCOMP.

[8]  Pelle Ehn,et al.  Work-oriented design of computer artifacts , 1989 .

[9]  R. J. Bogumil,et al.  The reflective practitioner: How professionals think in action , 1985, Proceedings of the IEEE.

[10]  John Levine,et al.  Effects of Collective Socialization on Newcomers’ Response to Feedback in Online Communities , 2018, ACM Trans. Soc. Comput..

[11]  Peter van den Besselaar,et al.  From small scale to large scale user participation: a case study of participatory design in e-government systems , 2004, PDC 04.

[12]  Henry Mainsah,et al.  Participatory design through a cultural lens: insights from postcolonial theory , 2014, PDC '14.

[13]  Ronald S Pimentel Kendall's Tau and Spearman's Rho for Zero-Inflated Data , 2009 .

[14]  James Grimmelmann,et al.  The Virtues of Moderation , 2015 .

[15]  A. Stetsenko,et al.  Vygotskian collaborative project of social transformation History , politics , and practice in knowledge construction , 2004 .

[16]  Jeanette Blomberg,et al.  An ethnographic approach to design , 2002 .

[17]  Kazjon Grace,et al.  Designing with and for the Crowd: A Cognitive Study of Design Processes in NatureNet , 2018, Design Computing and Cognition '18.

[18]  Dario Taraborelli,et al.  MoodBar: Increasing New User Retention in Wikipedia through Lightweight Socialization , 2014, CSCW.

[19]  Kenneth Benoit,et al.  Natural Sentences as Valid Units for Coded Political Texts , 2012 .

[20]  Bernardo A. Huberman,et al.  Usage patterns of collaborative tagging systems , 2006, J. Inf. Sci..

[21]  T. Gowers,et al.  Massively collaborative mathematics , 2009, Nature.

[22]  Cynthia J. Atman,et al.  Comparing freshman and senior engineering design processes: an in-depth follow-up study , 2005 .

[23]  Kees Dorst,et al.  The core of ‘design thinking’ and its application , 2011 .

[24]  D. Lei,et al.  Collaborative Innovation with Customers: A Review of the Literature and Suggestions for Future Research , 2012 .

[25]  Bella Martin,et al.  Universal Methods of Design: 100 Ways to Research Complex Problems, Develop Innovative Ideas, and Design Effective Solutions , 2012 .

[26]  P. Davies Beacons of hope , 2005 .

[27]  Barry L. Bayus,et al.  Crowdsourcing New Product Ideas over Time: An Analysis of the Dell IdeaStorm Community , 2013, Manag. Sci..

[28]  R. Mooney,et al.  Learnable similarity functions and their application to record linkage and clustering , 2006 .

[29]  Gabriela Goldschmidt,et al.  Linkography : Assessing Design Productivity , 1990 .

[30]  John S. Gero,et al.  Advances in Formal Design Methods for CAD , 1996, IFIP — The International Federation for Information Processing.

[31]  Klaus Krippendorff,et al.  On the reliability of unitizing textual continua: Further developments , 2016 .

[32]  Colleen M. Lewis,et al.  How Equity and Inequity Can Emerge in Pair Programming , 2015, ICER.

[33]  Robert E. Kraut,et al.  Socialization tactics in wikipedia and their effects , 2010, CSCW '10.

[34]  Monica M. C. Schraefel,et al.  Exploring, Defining, & Advancing Community-Driven Design for Social Impact , 2019, Conference on Designing Interactive Systems.

[35]  Jane Sinclair,et al.  Exploring the use of MOOC discussion forums , 2014 .

[36]  Steven Dow,et al.  IdeaGens: Enabling Expert Facilitation of Crowd Brainstorming , 2016, CSCW '16 Companion.

[37]  Karim R. Lakhani,et al.  InnoCentive.Com (B) , 2011 .

[38]  Alexander J. Quinn,et al.  BlueSky: Crowd-Powered Uniform Sampling of Idea Spaces , 2017, Creativity & Cognition.

[39]  Michael S. Bernstein,et al.  Mechanical Novel: Crowdsourcing Complex Work through Reflection and Revision , 2016, CSCW.

[40]  Susanne Bødker,et al.  When second wave HCI meets third wave challenges , 2006, NordiCHI '06.

[41]  Cristina Nita-Rotaru,et al.  A survey of attack and defense techniques for reputation systems , 2009, CSUR.

[42]  Sue Moon,et al.  Identifying and Quantifying Coordinated Manipulation of Upvotes and Downvotes in Naver News Comments , 2020, ICWSM.

[43]  Aaron Halfaker,et al.  Don't bite the newbies: how reverts affect the quantity and quality of Wikipedia work , 2011, Int. Sym. Wikis.

[44]  Gilly Leshed,et al.  The Magic Sauce: Practices of Facilitation in Online Policy Deliberation , 2016, Regular Issue.

[45]  Vinod Goel,et al.  The Structure of Design Problem Spaces , 1992, Cogn. Sci..

[46]  Lincoln Dahlberg The Internet and Democratic Discourse: Exploring The Prospects of Online Deliberative Forums Extending the Public Sphere , 2001 .

[47]  Mung Chiang,et al.  Personalized Thread Recommendation for MOOC Discussion Forums , 2018, ECML/PKDD.

[48]  A. Mathie,et al.  From clients to citizens: Asset-based Community Development as a strategy for community-driven development , 2003 .

[49]  A. Russo,et al.  Getting Personal : Reflexivity , Positionality , and Feminist Research , 2022 .

[50]  Amy Bruckman,et al.  Leadership in online creative collaboration , 2008, CSCW.

[51]  Amy X. Zhang,et al.  Digital Juries: A Civics-Oriented Approach to Platform Governance , 2020, CHI.

[52]  Amy Bruckman,et al.  Redistributing leadership in online creative collaboration , 2013, CSCW.

[53]  Jennifer Stromer-Galley Measuring Deliberation’s Content: A Coding Scheme , 2007, Regular Issue.

[54]  Krzysztof Z. Gajos,et al.  IdeaHound: Improving Large-scale Collaborative Ideation with Crowd-Powered Real-time Semantic Modeling , 2016, UIST.

[55]  Kate Starbird,et al.  Centralized, Parallel, and Distributed Information Processing during Collective Sensemaking , 2017, CHI.

[56]  Celine Latulipe,et al.  Dimensional Reasoning and Research Design Spaces , 2017, Creativity & Cognition.

[57]  Ulrike Cress,et al.  Knowledge Construction in Wikipedia: A Systemic-Constructivist Analysis , 2014 .

[58]  Justin Reich,et al.  Addressing Common Analytic Challenges to Randomized Experiments in MOOCs: Attrition and Zero-Inflation , 2015, L@S.

[59]  Michael J. Muller,et al.  Social Tagging and Self-Tagging for Impression Management * , 2006 .

[60]  Elizabeth B.-N. Sanders,et al.  From user-centered to participatory design approaches , 2002 .

[61]  Scott R. Klemmer,et al.  Docent: transforming personal intuitions to scientific hypotheses through content learning and process training , 2018, L@S.

[62]  Betty Vandenbosch,et al.  Information Overload in a Groupware Environment: Now You See It, Now You Don't , 1998, J. Organ. Comput. Electron. Commer..

[63]  A. Stetsenko,et al.  Constructing and Deconstructing the Self: Comparing Post-Vygotskian and Discourse-Based Versions of Social Constructivism , 1997 .

[64]  Nigel Cross,et al.  Creativity in the design process: co-evolution of problem–solution , 2001 .

[65]  Aniket Kittur,et al.  The polymath project: lessons from a successful online collaboration in mathematics , 2011, CHI.

[66]  C. H. Dorst,et al.  Frame Innovation: Create New Thinking by Design , 2015 .

[67]  Andreas Hotho,et al.  The anti-social tagger: detecting spam in social bookmarking systems , 2008, AIRWeb '08.

[68]  Dafna Shahaf,et al.  31 SOLVENT : A Mixed Initiative System for Finding Analogies between Research Papers , 2018 .

[69]  Tiffany Tseng Build in Progress : Building Process-Oriented Documentation , 2015 .

[70]  Marisol Wong-Villacres,et al.  Culture in Action: Unpacking Capacities to Inform Assets-Based Design , 2020, CHI.

[71]  Yla R. Tausczik,et al.  Will Too Many Editors Spoil The Tag? , 2018, Proc. ACM Hum. Comput. Interact..

[72]  S. Rafaeli,et al.  Psychological Aspects of Cyberspace: Online Motivational Factors: Incentives for Participation and Contribution in Wikipedia , 2008 .

[73]  C. Lewis FROM PUBLIC SPACES TO PUBLIC SPHERE Rethinking systems for reader comments on online news sites , 2015 .

[74]  Peter Dalsgaard,et al.  Mapping the Landscape of Creativity Support Tools in HCI , 2019, CHI.

[75]  Kees Dorst,et al.  Briefing and reframing: A situated practice , 2011 .

[76]  Jure Leskovec,et al.  Antisocial Behavior in Online Discussion Communities , 2015, ICWSM.

[77]  Kim-Kwang Raymond Choo,et al.  Manipulating Visibility of Political and Apolitical Threads on Reddit via Score Boosting , 2018, 2018 17th IEEE International Conference On Trust, Security And Privacy In Computing And Communications/ 12th IEEE International Conference On Big Data Science And Engineering (TrustCom/BigDataSE).

[78]  Johnny Saldaña,et al.  The Coding Manual for Qualitative Researchers , 2009 .

[79]  Volkmar Pipek,et al.  Community driven development as participation?: involving user communities in a software design process , 2008, PDC.

[80]  Svenja Boberg,et al.  The Moral Gatekeeper? Moderation and Deletion of User-Generated Content in a Leading News Forum , 2018, Media and Communication.

[81]  Marc Roper,et al.  What's in a bug report? , 2014, ESEM '14.

[82]  Douglas Schuler,et al.  Participatory Design: Principles and Practices , 1993 .

[83]  Tone Bratteteig,et al.  Unpacking the Notion of Participation in Participatory Design , 2016, Computer Supported Cooperative Work (CSCW).

[84]  Chinmay Kulkarni,et al.  $1 Conversational Turn Detector: Measuring How Video Conversations Affect Student Learning in Online Classes , 2016, L@S.

[85]  Alexander van Deursen,et al.  Internet skills and the digital divide , 2011, New Media Soc..

[86]  Eric Gilbert,et al.  Widespread underprovision on Reddit , 2013, CSCW.

[87]  M. Klein How to Harvest Collective Wisdom on Complex Problems: An Introduction to the MIT Deliberatorium , 2012 .

[88]  Robert E. Kraut,et al.  Wikipedia classroom experiment: bidirectional benefits of students' engagement in online production communities , 2013, CHI.

[89]  Kent Marett,et al.  The Decision to Share Information and Rumors: Examining the Role of Motivation in an Online Discussion Forum , 2009, Commun. Assoc. Inf. Syst..

[90]  R. Estrella,et al.  Stuck in the Shallow End Education , Race , and Computing , 2008 .

[91]  Leysia Palen,et al.  "Beacons of hope" in decentralized coordination: learning from on-the-ground medical twitterers during the 2010 Haiti earthquake , 2012, CSCW.

[92]  Pieter Jan Stappers,et al.  Co-creation and the new landscapes of design , 2008 .

[93]  BruckmanAmy,et al.  Human-Machine Collaboration for Content Regulation , 2019 .

[94]  Alan Borning,et al.  Supporting reflective public thought with considerit , 2012, CSCW.

[95]  Richard Reeve,et al.  Facilitating Problem Framing in Project-Based Learning , 2016, Interactional Research Into Problem-Based Learning.

[96]  B. Nardi,et al.  Ethnography and Virtual Worlds: A Handbook of Method , 2012 .

[97]  Michael Hammond,et al.  A REVIEW OF RECENT PAPERS ON ONLINE DISCUSSION IN TEACHING AND LEARNING IN HIGHER EDUCATION , 2019, Online Learning.

[98]  David R. Karger,et al.  Wikum: Bridging Discussion Forums and Wikis Using Recursive Summarization , 2017, CSCW.

[99]  Andy Dong,et al.  Exploring the Relationship Between Lexical Behavior and Concept Formation in Design Conversations , 2005 .

[100]  Todd Graham,et al.  Discursive Equality and Everyday Talk Online: The Impact of "Superparticipants" , 2014, J. Comput. Mediat. Commun..

[101]  J. R. Landis,et al.  The measurement of observer agreement for categorical data. , 1977, Biometrics.

[102]  Osamuyimen Stewart,et al.  Crowdsourcing participation inequality: a SCOUT model for the enterprise domain , 2010, HCOMP '10.

[103]  Robin Adams,et al.  The Informed Design Teaching and Learning Matrix , 2012 .

[104]  Anirban Dasgupta,et al.  Superposter behavior in MOOC forums , 2014, L@S.

[105]  Thomas Zimmermann,et al.  What Makes a Good Bug Report? , 2008, IEEE Transactions on Software Engineering.

[106]  Jock D. Mackinlay,et al.  The design space of input devices , 1990, CHI '90.

[107]  Tone Bratteteig,et al.  Disentangling power and decision-making in participatory design , 2012, PDC '12.

[108]  Mary Lou Maher,et al.  Formalising Design Exploration as Co-Evolution , 1996 .

[109]  Gabriele Bavota,et al.  Detecting missing information in bug descriptions , 2017, ESEC/SIGSOFT FSE.

[110]  Laura E. Kozak,et al.  Sticks, Ropes, Land: Confronting Colonial Practices in Public Space Design , 2020, PDC.

[111]  Eric Horvitz,et al.  What's your idea?: a case study of a grassroots innovation pipeline within a large software company , 2010, CHI.

[112]  Amy X. Zhang,et al.  Making Sense of Group Chat through Collaborative Tagging and Summarization , 2018, Proc. ACM Hum. Comput. Interact..

[113]  Colin M. Gray,et al.  Supporting Distributed Critique through Interpretation and Sense-Making in an Online Creative Community , 2017, Proc. ACM Hum. Comput. Interact..

[114]  Cynthia J. Atman,et al.  Engineering Design Processes: A Comparison of Students and Expert Practitioners , 2007 .

[115]  Cliff Lampe,et al.  Defining, Understanding, and Supporting Open Collaboration , 2013 .

[116]  S. Coleman,et al.  Bowling Together: Online Public Engagement in Policy Deliberation , 2001 .

[117]  F. Baum,et al.  Participatory action research , 2006, Journal of Epidemiology and Community Health.

[118]  Kim Halskov,et al.  A constraint-based understanding of design spaces , 2014, Conference on Designing Interactive Systems.