Interpretable Directed Diversity: Leveraging Model Explanations for Iterative Crowd Ideation

Feedback in creativity support tools can help crowdworkers to improve their ideations. However, current feedback methods require human assessment from facilitators or peers. This is not scalable to large crowds. We propose Interpretable Directed Diversity to automatically predict ideation quality and diversity scores, and provide AI explanations — Attribution, Contrastive Attribution, and Counterfactual Suggestions — to feedback on why ideations were scored (low), and how to get higher scores. These explanations provide multi-faceted feedback as users iteratively improve their ideations. We conducted formative and controlled user studies to understand the usage and usefulness of explanations to improve ideation diversity and quality. Users appreciated that explanation feedback helped focus their efforts and provided directions for improvement. This resulted in explanations improving diversity compared to no feedback or feedback with predictions only. Hence, our approach opens opportunities for explainable AI towards scalable and rich feedback for iterative crowd ideation and creativity support tools.

[1]  Raymond Fok,et al.  Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance , 2020, CHI.

[2]  Amrita Sadarangani,et al.  Mental Models of Mere Mortals with Explanations of Reinforcement Learning , 2020, ACM Trans. Interact. Intell. Syst..

[3]  Leonard Adelman,et al.  Examining the Effect of Causal Focus on the Option Generation Process: An Experiment Using Protocol Analysis , 1995 .

[4]  Aniket Kittur,et al.  Crowdlines: Supporting Synthesis of Diverse Information Sources through Crowdsourced Outlines , 2015, HCOMP.

[5]  Peter Dalsgaard,et al.  Mapping the Landscape of Creativity Support Tools in HCI , 2019, CHI.

[6]  Gökhan Tür,et al.  Building a Conversational Agent Overnight with Dialogue Self-Play , 2018, ArXiv.

[7]  Larry Ambrose,et al.  The power of feedback. , 2002, Healthcare executive.

[8]  Gary Hsieh,et al.  Send Me a Different Message: Utilizing Cognitive Space to Create Engaging Message Triggers , 2017, CSCW.

[9]  Krzysztof Z. Gajos,et al.  Semantically Far Inspirations Considered Harmful?: Accounting for Cognitive States in Collaborative Ideation , 2017, Creativity & Cognition.

[10]  Sonia Chernova,et al.  Leveraging rationales to improve human task performance , 2020, IUI.

[11]  Jeffrey V. Nickerson,et al.  Cooks or cobblers?: crowd creativity through combination , 2011, CHI.

[12]  Wolfgang Stroebe,et al.  How the Group Affects the Mind: A Cognitive Model of Idea Generation in Groups , 2006, Personality and social psychology review : an official journal of the Society for Personality and Social Psychology, Inc.

[13]  Scott Lundberg,et al.  A Unified Approach to Interpreting Model Predictions , 2017, NIPS.

[14]  Ankur Taly,et al.  Axiomatic Attribution for Deep Networks , 2017, ICML.

[15]  Michael S. Bernstein,et al.  Soylent: a word processor with a crowd inside , 2010, UIST.

[16]  Carlos Guestrin,et al.  "Why Should I Trust You?": Explaining the Predictions of Any Classifier , 2016, ArXiv.

[17]  Joe Tullio,et al.  How it works: a field study of non-technical users interacting with an intelligent system , 2007, CHI.

[18]  Nan Hua,et al.  Universal Sentence Encoder , 2018, ArXiv.

[19]  Jeffrey V. Nickerson,et al.  A literature review on individual creativity support systems , 2017, Comput. Hum. Behav..

[20]  Andrés Lucero,et al.  ImageSense: An Intelligent Collaborative Ideation Tool to Support Diverse Human-Computer Partnerships , 2020, Proc. ACM Hum. Comput. Interact..

[21]  Mark Fuge,et al.  Interpreting Idea Maps: Pairwise Comparisons Reveal What Makes Ideas Novel , 2019, Journal of Mechanical Design.

[22]  Jonathan A. Fugelsang,et al.  Neural correlates of creativity in analogical reasoning. , 2012, Journal of experimental psychology. Learning, memory, and cognition.

[23]  E. A. Locke,et al.  Building a practically useful theory of goal setting and task motivation. A 35-year odyssey. , 2002, The American psychologist.

[24]  Lydia B. Chilton,et al.  Metaphoria: An Algorithmic Companion for Metaphor Creation , 2019, CHI.

[25]  Catherine Havasi,et al.  ConceptNet 5.5: An Open Multilingual Graph of General Knowledge , 2016, AAAI.

[26]  Arkalgud Ramaprasad,et al.  On the definition of feedback , 1983 .

[27]  Jan Marco Leimeister,et al.  Rating Scales for Collective Intelligence in Innovation Communities: Why Quick and Easy Decision Making Does Not Get it Right , 2010, ICIS.

[28]  Paula Phillips Carson,et al.  Managing Creativity Enhancement Through Goal-Setting and Feedback† , 1993 .

[29]  Krzysztof Z. Gajos,et al.  Toward Collaborative Ideation at Scale: Leveraging Ideas from Others to Generate More Creative and Diverse Ideas , 2015, CSCW.

[30]  Tom Vanallemeersch,et al.  Intellingo: An Intelligible Translation Environment , 2018, CHI.

[31]  Pao Siangliulue,et al.  Critter: Augmenting Creative Work with Dynamic Checklists, Automated Quality Assurance, and Contextual Reviewer Feedback , 2019, CHI.

[32]  Joke Meheus,et al.  Analogical Reasoning in Creative Problem Solving Processes: Logico-Philosophical Perspectives , 2000 .

[33]  Abhishek Das,et al.  Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization , 2016, 2017 IEEE International Conference on Computer Vision (ICCV).

[34]  Vinod Goel,et al.  Neural basis of thinking: laboratory problems versus real-world problems. , 2010, Wiley interdisciplinary reviews. Cognitive science.

[35]  Cheng-Zhi Anna Huang,et al.  Novice-AI Music Co-Creation via AI-Steering Tools for Deep Generative Models , 2020, CHI.

[36]  Mark Klein,et al.  High-Speed Idea Filtering with the Bag of Lemons , 2014, IEEE Intell. Informatics Bull..

[37]  Mohan S. Kankanhalli,et al.  Trends and Trajectories for Explainable, Accountable and Intelligible Systems: An HCI Research Agenda , 2018, CHI.

[38]  Yejin Choi,et al.  COMET: Commonsense Transformers for Automatic Knowledge Graph Construction , 2019, ACL.

[39]  Brian Y. Lim,et al.  Towards Relatable Explainable AI with the Perceptual Process , 2021, ArXiv.

[40]  Anind K. Dey,et al.  Why and why not explanations improve the intelligibility of context-aware intelligent systems , 2009, CHI.

[41]  Scott R. Klemmer,et al.  Shepherding the crowd yields better work , 2012, CSCW.

[42]  Anind K. Dey,et al.  Design of an intelligible mobile context-aware application , 2011, Mobile HCI.

[43]  Steven M. Smith.  The constraining effects of initial ideas. , 2003 .

[44]  Alexander Binder,et al.  Layer-Wise Relevance Propagation for Deep Neural Network Architectures , 2016 .

[45]  Sean A. Munson,et al.  Crowdsourcing Exercise Plans Aligned with Expert Guidelines and Everyday Constraints , 2018, CHI.

[46]  Joanna L. Y. Ho,et al.  Decision problem structuring: generating options , 1988, IEEE Trans. Syst. Man Cybern..

[47]  B. Nijstad,et al.  Cognitive stimulation and interference in groups: Exposure effects in an idea generation task , 2002 .

[48]  Martin Wattenberg,et al.  Human-Centered Tools for Coping with Imperfect Algorithms During Medical Decision-Making , 2019, CHI.

[49]  Kurt VanLehn,et al.  A model of the self-explanation effect. , 1992 .

[50]  Ming-Wei Chang,et al.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , 2019, NAACL.

[51]  Krzysztof Z. Gajos,et al.  IdeaHound: Improving Large-scale Collaborative Ideation with Crowd-Powered Real-time Semantic Modeling , 2016, UIST.

[52]  Scott R. Klemmer,et al.  The efficacy of prototyping under time constraints , 2009, C&C '09.

[53]  Tim Miller,et al.  Explanation in Artificial Intelligence: Insights from the Social Sciences , 2017, Artif. Intell..

[54]  Vanessa Evers,et al.  Crowd-Designed Motivation: Motivational Messages for Exercise Adherence Based on Behavior Change Theory , 2016, CHI.

[55]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[56]  O. Bjelland,et al.  An Inside View of IBM's 'Innovation Jam' , 2008 .

[57]  T. Lubart.  Models of the Creative Process: Past, Present and Future , 2001 .

[58]  John M. Carroll,et al.  Exploring and Promoting Diagnostic Transparency and Explainability in Online Symptom Checkers , 2021, CHI.

[59]  Jonathan E. Butner,et al.  Compliance with a Request in Two Cultures: The Differential Influence of Social Proof and Commitment/Consistency on Collectivists and Individualists , 1999 .

[60]  Mark O. Riedl,et al.  Rationalization: A Neural Machine Translation Approach to Generating Natural Language Explanations , 2017, AIES.

[61]  Peter Dalsgaard,et al.  Twenty Years of Creativity Research in Human-Computer Interaction: Current State and Future Directions , 2018, Conference on Designing Interactive Systems.

[62]  E. A. Locke,et al.  A theory of goal setting & task performance , 1990 .

[63]  Anind K. Dey,et al.  Evaluating Intelligibility Usage and Usefulness in a Context-Aware Application , 2013, HCI.

[64]  Jesse Chandler,et al.  Online panels in social science research: Expanding sampling methods beyond Mechanical Turk , 2019, Behavior Research Methods.

[65]  Elizabeth Gerber,et al.  Listen to Others, Listen to Yourself: Combining Feedback Review and Reflection to Improve Iterative Design , 2017, Creativity & Cognition.

[66]  Daniel L. Schwartz,et al.  Parallel prototyping leads to better design results, more divergence, and increased self-efficacy , 2010, TCHI.

[67]  S. Derry,et al.  Learning from Examples: Instructional Principles from the Worked Examples Research , 2000 .

[68]  C. Gettys,et al.  MINERVA-DM: A memory processes model for judgments of likelihood. , 1999 .

[69]  Simo Hosio,et al.  Design recommendations for augmenting creative tasks with computational priming , 2019, MUM.

[70]  Margaret A. Boden,et al.  Chapter 9 – Creativity , 1996 .

[71]  Vishal Gupta,et al.  Recent automatic text summarization techniques: a survey , 2016, Artificial Intelligence Review.

[72]  Ming Yin,et al.  Are Explanations Helpful? A Comparative Study of the Effects of Explanations in AI-Assisted Decision-Making , 2021, IUI.

[73]  Ben Shneiderman,et al.  Creativity support tools: accelerating discovery and innovation , 2007, CACM.

[74]  Deborah Silver,et al.  Feature Visualization , 1994, Scientific Visualization.

[75]  Sharon Bailin.  CREATIVITY IN CONTEXT , 2002 .

[76]  Brian P. Bailey,et al.  Voyant: generating structured feedback on visual designs using a crowd of non-experts , 2014, CSCW.

[77]  M. Boden.  The creative mind : myths & mechanisms , 1991 .

[78]  Vincent Aleven,et al.  Can Crowds Customize Instructional Materials with Minimal Expert Guidance? , 2021, Proc. ACM Hum. Comput. Interact..

[79]  O. Houdé,et al.  How minimal executive feedback influences creative idea generation , 2017, PloS one.

[80]  J. Pennebaker,et al.  The Psychological Meaning of Words: LIWC and Computerized Text Analysis Methods , 2010 .

[81]  Qian Yang,et al.  Designing Theory-Driven User-Centric Explainable AI , 2019, CHI.

[82]  Martin Wattenberg,et al.  Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV) , 2017, ICML.

[83]  Simo Hosio,et al.  Supporting Creative Work with Crowd Feedback Systems , 2020, ArXiv.

[84]  Brian Y. Lim,et al.  Show or Suppress? Managing Input Uncertainty in Machine Learning Model Explanations , 2021, Artif. Intell..

[85]  Simo Hosio,et al.  Hardhats and Bungaloos: Comparing Crowdsourced Design Feedback with Peer Design Feedback in the Classroom , 2021 .

[86]  Timo Mäntylä,et al.  Option generation in decision making: ideation beyond memory retrieval , 2015, Front. Psychol..

[87]  Eric Brill,et al.  Beyond PageRank: machine learning for static ranking , 2006, WWW '06.

[88]  Senthil K. Chandrasegaran,et al.  Spinneret: Aiding Creative Ideation through Non-Obvious Concept Associations , 2020, CHI.

[89]  Jonas Oppenlaender,et al.  Crowdsourcing Personalized Weight Loss Diets , 2020, Computer.

[90]  Brian Y. Lim,et al.  Directed Diversity: Leveraging Language Embedding Distances for Collective Creativity in Crowd Ideation , 2021, CHI.

[91]  Baptiste Barbot,et al.  The Dynamics of Creative Ideation: Introducing a New Assessment Paradigm , 2018, Front. Psychol..

[92]  Alexander J. Quinn,et al.  BlueSky: Crowd-Powered Uniform Sampling of Idea Spaces , 2017, Creativity & Cognition.

[93]  Simo Hosio,et al.  CrowdUI: Supporting Web Design with the Crowd , 2020, Proc. ACM Hum. Comput. Interact..

[94]  Noah A. Smith,et al.  Creative Writing with a Machine in the Loop: Case Studies on Slogans and Stories , 2018, IUI.

[95]  Xiaojuan Ma,et al.  Exploring the Effects of Technological Writing Assistance for Support Providers in Online Mental Health Community , 2020, CHI.