Crowd-centric requirements engineering: A method based on crowdsourcing and gamification

Requirements Engineering (RE) is an essential process in the continuous development of software products. User involvement has a large potential for improving the quality of RE and thereby the quality of software. Shortcomings of current RE methods and threats in the complex environment of Software Product Organizations (SPO) trigger the need for an innovative method. Crowdsourcing and gamification are two emerging trends that provide opportunities to overcome the shortcomings and threats. In this thesis, the Crowd-Centric Requirements Engineering (CCRE) method, which guides SPOs in involving all stakeholders in the RE process, is described. A prototype, Refine, is built to demonstrate the method. Although some obstacles to large-scale adoption exist, the evaluation of this demonstration shows that CCRE can provide a useful process, useful requirements, engaged stakeholders and valuable interaction among those stakeholders. Through these aspects, the method has shown potential to improve requirements engineering in software production.

[1]  Bashar Nuseibeh,et al.  Requirements engineering: a roadmap , 2000, ICSE '00.

[2]  Martin Höst,et al.  An Industrial Case Study on Distributed Prioritisation in Market-Driven Requirements Engineering for Packaged Software , 2001, Requirements Engineering.

[3]  Navneet Kaur,et al.  Opinion mining and sentiment analysis , 2016, 2016 3rd International Conference on Computing for Sustainable Global Development (INDIACom).

[4]  Inge van de Weerd,et al.  A Situational Assessment Method for Software Product Management , 2010, ECIS.

[5]  Miguel Mira da Silva,et al.  Gamifying requirement elicitation: Practical implications and outcomes in improving stakeholders collaboration , 2014, Entertain. Comput..

[6]  Lai Xu,et al.  Concepts of product software , 2007, Eur. J. Inf. Syst..

[7]  Franz Lehner,et al.  Requirements Engineering as a Success Factor in Software Projects , 2001, IEEE Softw..

[8]  Fred D. Davis,et al.  User Acceptance of Computer Technology: A Comparison of Two Theoretical Models , 1989 .

[9]  Samir Chatterjee,et al.  A Design Science Research Methodology for Information Systems Research , 2008 .

[10]  Scott Nicholson.  A User-Centered Theoretical Framework for Meaningful Gamification , 2012 .

[11]  Matti A. Kaulio.  Customer, consumer and user involvement in product development: A framework and a review of selected methods , 1998 .

[12]  S. Haynes,et al.  Content validity in psychological assessment: A functional approach to concepts and methods. , 1995 .

[13]  Axel van Lamsweerde,et al.  Requirements engineering in the year 00: a research perspective , 2000, Proceedings of the 2000 International Conference on Software Engineering. ICSE 2000 the New Millennium.

[14]  Daniela E. Damian,et al.  StakeSource2.0: using social networks of stakeholders to identify and prioritise requirements , 2011, 2011 33rd International Conference on Software Engineering (ICSE).

[15]  Mahmood Hosseini,et al.  Towards Crowdsourcing for Requirements Engineering , 2014, REFSQ Workshops.

[16]  Brendan T. O'Connor,et al.  Cheap and Fast – But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks , 2008, EMNLP.

[17]  Daniel J. Veit,et al.  More than fun and money. Worker Motivation in Crowdsourcing - A Study on Mechanical Turk , 2011, AMCIS.

[18]  Ivan A. Garcia,et al.  A systematic literature review of stakeholder identification methods in requirements elicitation , 2012, J. Syst. Softw..

[19]  Padmini Srinivasan,et al.  Quality through flow and immersion: gamifying crowdsourced relevance assessments , 2012, SIGIR '12.

[20]  Eric Schenk,et al.  Crowdsourcing: What can be Outsourced to the Crowd, and Why ? , 2009 .

[21]  Dana Chandler,et al.  Breaking Monotony with Meaning: Motivation in Crowdsourcing Markets , 2012, ArXiv.

[22]  M. Brewer,et al.  Research Design and Issues of Validity , 2000 .

[23]  Lennart E. Nacke,et al.  From game design elements to gamefulness: defining "gamification" , 2011, MindTrek.

[24]  Daren C. Brabham.  MOVING THE CROWD AT THREADLESS , 2010 .

[25]  Kent L. Beck,et al.  Extreme programming explained - embrace change , 1990 .

[26]  Pär Carlshamre,et al.  Release Planning in Market-Driven Software Product Development: Provoking an Understanding , 2002, Requirements Engineering.

[27]  Rory V. O'Connor,et al.  The situational factors that affect the software development process: Towards a comprehensive reference framework , 2012, Inf. Softw. Technol..

[28]  Inge van de Weerd,et al.  Towards a Reference Framework for Software Product Management , 2006, 14th IEEE International Requirements Engineering Conference (RE'06).

[29]  Inge van de Weerd,et al.  Situational requirements engineering for the development of Content Management System-based web applications , 2007, Int. J. Web Eng. Technol..

[30]  Leela Damodaran,et al.  User involvement in the systems design process-a practical guide for users , 1996, Behav. Inf. Technol..

[31]  H. Holmstrom,et al.  Virtual communities for software maintenance , 2004, 37th Annual Hawaii International Conference on System Sciences, 2004. Proceedings of the.

[32]  Betty H. C. Cheng,et al.  Research Directions in Requirements Engineering , 2007, Future of Software Engineering (FOSE '07).

[33]  Salvatore T. March,et al.  Design and natural science research on information technology , 1995, Decis. Support Syst..

[34]  Juho Hamari,et al.  Does Gamification Work? -- A Literature Review of Empirical Studies on Gamification , 2014, 2014 47th Hawaii International Conference on System Sciences.

[35]  Ivy Hooks,et al.  What Is A Requirement , 1993 .

[36]  J. Michael Spivey,et al.  An introduction to Z and formal specifications , 1989, Softw. Eng. J..

[37]  Björn Regnell,et al.  Market-Driven Requirements Engineering for Software Products , 2005 .

[38]  Fernando González-Ladrón-de-Guevara,et al.  Towards an integrated crowdsourcing definition , 2012, J. Inf. Sci..

[39]  Marjo Kauppinen,et al.  The role of user involvement in requirements quality and project success , 2005, 13th IEEE International Conference on Requirements Engineering (RE'05).

[40]  Mahbubur Rahman Syed,et al.  Handbook of research on modern systems analysis and design technologies and applications , 2008 .

[41]  Pericles Loucopoulos,et al.  System Requirements Engineering , 1995, System Requirements Engineering.

[42]  Venugopal Balijepally,et al.  Theoretical reflections on agile development methodologies , 2007, Commun. ACM.

[43]  Christof Ebert,et al.  The impacts of software product management , 2007, J. Syst. Softw..

[44]  David M. Pennock,et al.  Mining the peanut gallery: opinion extraction and semantic classification of product reviews , 2003, WWW '03.

[45]  Alan R. Hevner,et al.  Design Science in Information Systems Research , 2004, MIS Q..

[46]  Björn Regnell,et al.  Can We Beat the Complexity of Very Large-Scale Requirements Engineering? , 2008, REFSQ.

[47]  Inge van de Weerd,et al.  Meta-Modeling for Situational Analysis and Design Methods , 2009 .

[48]  Aniket Kittur,et al.  Crowdsourcing user studies with Mechanical Turk , 2008, CHI.

[49]  Murali Chemuturi.  Requirements Engineering and Management for Software Development Projects , 2013, Springer New York.

[50]  Daren C. Brabham.  Moving the crowd at iStockphoto: The composition of the crowd and motivations for participation in a crowdsourcing application , 2008, First Monday.

[51]  Steve Sawyer,et al.  Software Development: Processes and Performance , 1998, IBM Syst. J..

[52]  Slinger Jansen,et al.  Customer Involvement in Requirements Management: Lessons from Mass Market Software Development , 2009, 2009 17th IEEE International Requirements Engineering Conference.

[53]  Sari Kujala,et al.  User involvement: A review of the benefits and challenges , 2003, Behav. Inf. Technol..

[54]  Dale E. Zand,et al.  Theory of Change and the Effective Use of Management Science. , 1975 .

[55]  Karl E. Wiegers.  First Things First: Prioritizing Requirements , 1999 .

[56]  A. Adam.  Whatever happened to information systems ethics? Caught between the devil and the deep blue sea , 2004 .

[57]  Marjo Kauppinen,et al.  Bridging the Gap between User Needs and User Requirements , 2001 .

[58]  F. Scientific.  Online Shopping , 1998, Current Biology.

[59]  Davor Svetinovic,et al.  CrowdREquire: A Requirements Engineering Crowdsourcing Platform , 2012, AAAI Spring Symposium: Wisdom of the Crowd.

[60]  Sjaak Brinkkemper,et al.  Situational method engineering for informational system project approaches , 1994, Methods and Associated Tools for the Information Systems Life Cycle.

[61]  Klaus Pohl,et al.  The Three Dimensions of Requirements Engineering , 1993, CAiSE.

[62]  Slinger Jansen,et al.  The Agile Requirements Refinery: Applying SCRUM Principles to Software Product Management , 2009, 2009 Third International Workshop on Software Product Management.

[63]  Inge van de Weerd,et al.  A Framework for Process Improvement in Software Product Management , 2010, EuroSPI.

[64]  Norbert Seyff,et al.  Using Mobile RE Tools To Give End-Users a Voice , 2015 .

[65]  W. J. Bekkers,et al.  Situational Process Improvement in Software Product Management , 2012 .

[66]  Michael Vitale,et al.  The Wisdom of Crowds , 2015, Cell.

[67]  Sjaak Brinkkemper,et al.  Software product management - An industry evaluation , 2014, J. Syst. Softw..

[68]  Adam Wojciechowski,et al.  Extreme programming modified: embrace requirements engineering practices , 2002, Proceedings IEEE Joint International Conference on Requirements Engineering.

[69]  Andrey Maglyas,et al.  What do we know about software product management? - a systematic mapping study , 2011, 2011 Fifth International Workshop on Software Product Management (IWSPM).

[70]  Mark C. Paulk,et al.  Capability Maturity Model for Software , 2001 .

[71]  Khaled El Emam,et al.  User participation in the requirements engineering process: An empirical study , 1996, Requirements Engineering.

[72]  Juho Hamari,et al.  Social Motivations To Use Gamification: An Empirical Study Of Gamifying Exercise , 2013, ECIS.

[73]  Susanne Robra-Bissantz,et al.  Gamification of online idea competitions: insights from an explorative case , 2011, GI-Jahrestagung.

[74]  Beum-Seuk Lee,et al.  Two-level grammar as an object-oriented requirements specification language , 2002, Proceedings of the 35th Annual Hawaii International Conference on System Sciences.

[75]  Daren C. Brabham.  Crowdsourcing as a Model for Problem Solving , 2008 .

[76]  Salvatore T. March,et al.  Information Systems Research , 2004, IFIP International Federation for Information Processing.

[77]  Pamela Zave.  Classification of research efforts in requirements engineering , 1997, ACM Comput. Surv..

[78]  Jaime G. Carbonell,et al.  Active Learning and Crowd-Sourcing for Machine Translation , 2010, LREC.

[79]  Paul Grünbacher,et al.  Complementing XP with Requirements Negotiation , 2002 .

[80]  A Straw,et al.  Guide to the Software Engineering Body of Knowledge , 1998 .

[81]  Christoph Treude,et al.  The impact of social media on software engineering practices and tools , 2010, FoSER '10.

[82]  Mahmood Hosseini,et al.  The four pillars of crowdsourcing: A reference model , 2014, 2014 IEEE Eighth International Conference on Research Challenges in Information Science (RCIS).

[83]  Agile Manifesto,et al.  Manifesto for Agile Software Development , 2001 .

[84]  Tony Gorschek,et al.  Introduction of a process maturity model for market‐driven product management and requirements engineering , 2012, J. Softw. Maintenance Res. Pract..

[85]  Claes Wohlin,et al.  Systematic literature studies: Database searches vs. backward snowballing , 2012, Proceedings of the 2012 ACM-IEEE International Symposium on Empirical Software Engineering and Measurement.

[86]  Robert C. Martin,et al.  Extreme programming in practice , 2001, OOPSLA '00.

[87]  Daniele Quercia,et al.  StakeSource: harnessing the power of crowdsourcing and social networks in stakeholder analysis , 2010, 2010 ACM/IEEE 32nd International Conference on Software Engineering.

[88]  Anthony Finkelstein,et al.  Ieee Transactions on Software Engineering, Manuscript Id Stakerare: Using Social Networks and Collaborative Filtering for Large-scale Requirements Elicitation , 2022 .

[89]  John Mylopoulos,et al.  Understanding "why" in software process modelling, analysis, and design , 1994, Proceedings of 16th International Conference on Software Engineering.

[90]  Harlan D. Mills.  Software Development , 1976, IEEE Transactions on Software Engineering.

[91]  Colin Potts,et al.  Software-engineering research revisited , 1993, IEEE Software.

[92]  Jari Juhani Jussila,et al.  Can crowdsourcing really be used in B2B innovation? , 2012, MindTrek.

[93]  Sjaak Brinkkemper,et al.  Method engineering: engineering of information systems development methods and tools , 1996, Inf. Softw. Technol..

[94]  Miguel Mira da Silva,et al.  iThink: A Game-Based Approach Towards Improving Collaboration and Participation in Requirement Elicitation , 2012, VS-GAMES.

[95]  Jeff Sutherland,et al.  Manifesto for Agile Software Development , 2013 .