The impact of domain knowledge on the effectiveness of requirements engineering activities

One factor that seems to influence an individual’s effectiveness in requirements engineering activities is her knowledge of the problem being solved, i.e., domain knowledge. While in-depth domain knowledge enables a requirements analyst to understand the problem easier, she can fall for tacit assumptions and fail to consider issues that she believes to be obvious. This paper investigates the impact of domain knowledge on requirements engineering activities. Its main research question is “How does one form the most effective team, consisting of some mix of domain ignorants and domain awares, for a requirements engineering activity involving knowledge about the domain of the computer-based system whose requirements are being determined by the team?” For completeness, a number of other factors, such as educational background, are considered for their effect on teams’ effectiveness. Two controlled experiments test a number of hypotheses derived from the question, including mainly that for a computer-based system in a particular domain, a team consisting of a mix of requirements analysts that are both ignorant and aware of the domain, is more effective at requirement idea generation than a team consisting of only analysts that are aware of the domain. The results show no significant effect of the mix by itself on effectiveness in requirement idea generation. However, the results do show surprising significant effects of the educational background and of the mix combined with the educational background. Combining the results, the main conclusion is that the presence of a domain ignorant Communicated by: Daniel Amyot Electronic supplementary material The online version of this article (doi:10.1007/s10664-015-9416-2) contains supplementary material, which is available to authorized users. Daniel Berry dberry@uwaterloo.ca Ali Niknafs aniknafs@uwaterloo.ca 1 David R. Cheriton School of Computer Science, University of Waterloo, Waterloo, Ontario, Canada Empir Software Eng (2017) 22:80–133 81 with a computer science or software engineering background improves the effectiveness of a requirement idea generation team.

[1]  H. D. Rombach,et al.  The Goal Question Metric Approach , 1994 .

[2]  Frederick P. Brooks,et al.  The Mythical Man-Month: Essays on Softw , 1978 .

[3]  Brian Berenbach,et al.  Software & Systems Requirements Engineering: In Practice , 2009 .

[4]  Robert I. Sutton.  Weird Ideas That Work: 11 1/2 Practices for Promoting, Managing, and Sustaining Innovation , 2001 .

[5]  Lorenzo Magnani,et al.  Model-Based Reasoning in Scientific Discovery , 1999, Springer US.

[6]  Reiner R. Dumke,et al.  Techniques of Successful Application of Factor Analysis in Software Measurement , 2004, Empirical Software Engineering.

[7]  David A. Bell,et al.  The role of domain knowledge in data mining , 1995, CIKM '95.

[8]  Soren Lauesen,et al.  Software Requirements: Styles & Techniques , 2002 .

[9]  Steve Easterbrook,et al.  Communication Problems in Requirements Engineering: A Field Study , 1996 .

[10]  Janis A. Bubenko,et al.  Theories underlying requirements engineering: an overview of NATURE at Genesis , 1993, [1993] Proceedings of the IEEE International Symposium on Requirements Engineering.

[11]  Claes Wohlin,et al.  Using Students as Subjects—A Comparative Study of Students and Professionals in Lead-Time Impact Assessment , 2000, Empirical Software Engineering.

[12]  Kevin Dunbar,et al.  How Scientists Build Models In Vivo Science as a Window on the Scientific Mind , 1999 .

[13]  Jonah Lehrer.  Imagine: How Creativity Works , 2012 .

[14]  Irit Hadar,et al.  The Role of Domain Knowledge in Requirements Elicitation: an Exploratory Study , 2010, MCIS.

[15]  Gunnar Blom,et al.  Statistical Estimates and Transformed Beta-Variables. , 1960 .

[16]  Tore Dybå,et al.  A systematic review of statistical power in software engineering experiments , 2006, Inf. Softw. Technol..

[17]  Daniel M. Berry,et al.  The impact of domain knowledge on the effectiveness of requirements idea generation during requirements elicitation , 2012, 2012 20th IEEE International Requirements Engineering Conference (RE).

[18]  Daniel M. Berry,et al.  Vi.iv, a bi-directional version of the vi full-screen editor , 1990 .

[19]  Daniela E. Damian,et al.  Selecting Empirical Methods for Software Engineering Research , 2008, Guide to Advanced Empirical Software Engineering.

[20]  Tom DeMarco,et al.  Peopleware: Productive Projects and Teams , 1987 .

[21]  Robert Feldt,et al.  Validity Threats in Empirical Software Engineering Research - An Initial Survey , 2010, SEKE.

[22]  Daniel M. Berry.  The importance of ignorance in requirements engineering: An earlier sighting and a revisitation , 2002, J. Syst. Softw..

[23]  G. Glass,et al.  Consequences of Failure to Meet Assumptions Underlying the Fixed Effects Analyses of Variance and Covariance , 1972 .

[24]  T. Cook,et al.  Quasi-experimentation: Design & analysis issues for field settings , 1979 .

[25]  Claes Wohlin,et al.  Experimentation in software engineering: an introduction , 2000 .

[26]  Mika Mäntylä,et al.  The Role of the Tester's Knowledge in Exploratory Software Testing , 2013, IEEE Transactions on Software Engineering.

[27]  R. O'Brien.  An Overview of the Methodological Approach of Action Research , 2008 .

[28]  Jing Dong,et al.  User’s manual as a requirements specification: case studies , 2004, Requirements Engineering.

[29]  Nikolaos M. Avouris,et al.  The Role of Domain Knowledge in a Large Scale Data Mining Project , 2002, SETN.

[30]  Carl Frappaolo.  Implicit knowledge , 2008 .

[31]  K. W. Phillips,et al.  Rethinking the Baseline in Diversity Research , 2014, Perspectives on psychological science : a journal of the Association for Psychological Science.

[32]  Daniel M. Berry.  Stretching letter and slanted‐baseline formatting for Arabic, Hebrew, and Persian with ditroff/ffortid and dynamic PostScript fonts , 1999 .

[33]  Gerhard Fischer,et al.  Symmetry of igorance, social creativity, and meta-design , 1999, Creativity & Cognition.

[34]  Manish Kumar,et al.  A Method and Framework for Domain Knowledge Assisted Requirements Evolution (K-RE) , 2009 .

[35]  Anthony Finkelstein,et al.  Requirements engineering: a review and research agenda , 1994, Proceedings of 1st Asia-Pacific Software Engineering Conference.

[36]  Daniel M. Berry,et al.  The importance of ignorance in requirements engineering , 1995, J. Syst. Softw..

[37]  Nancy Van Schooenderwoert.  Embedded Agile Project by the Numbers With Newbies , 2006, AGILE.

[38]  A. Luchins,et al.  New Experimental Attempts at Preventing Mechanization in Problem Solving , 1950 .

[39]  Jason Rubens.  Business analysis and requirements engineering: the same, only different? , 2007, Requirements Engineering.

[40]  Phillip A. Laplante,et al.  Requirements Engineering for Software and Systems , 2009 .

[41]  Brian Randell,et al.  Software engineering : report on a conference sponsored by the NATO Science Committee, Garmisch, Germany, 7th to 11th October 1968 , 1969 .

[42]  Jeffrey C. Carver,et al.  The Impact of Educational Background on the Effectiveness of Requirements Inspections: An Empirical Study , 2008, IEEE Transactions on Software Engineering.

[43]  Per Runeson,et al.  Guidelines for conducting and reporting case study research in software engineering , 2009, Empirical Software Engineering.

[44]  J. Wiley.  Expertise as mental set: The effects of domain knowledge in creative problem solving , 1998, Memory & cognition.

[45]  Calvin W. Taylor,et al.  Instructional media and creativity : the proceedings of the Sixth Utah Creativity Research Conference held at Torrey Pines Inn, La Jolla, California , 1966 .

[46]  Klaus Schmid,et al.  Selecting Creativity Techniques for Innovative Requirements Engineering , 2008, 2008 Third International Workshop on Multimedia and Enjoyable Requirements Engineering - Beyond Mere Descriptions and with More Fun and Games.

[47]  A. Gustafsson,et al.  Harnessing the Creative Potential among Users , 2004 .

[48]  Gunnar Blom,et al.  Statistical Estimates and Transformed Beta-Variables. , 1960 .

[49]  Rachel K. E. Bellamy,et al.  Moving into a new software project landscape , 2010, 2010 ACM/IEEE 32nd International Conference on Software Engineering.

[50]  R. B. Johnson,et al.  Educational Research: Quantitative, Qualitative, and Mixed Approaches , 2007 .

[51]  Rachel K. E. Bellamy,et al.  A qualitative study on project landscapes , 2009, 2009 ICSE Workshop on Cooperative and Human Aspects on Software Engineering.

[52]  G. Mehrotra,et al.  Role of Domain Ignorance in Software Development , 2011 .

[53]  A. Luchins.  Mechanization in problem solving: The effect of Einstellung. , 1942 .

[54]  R. Warner.  Applied Statistics: From Bivariate through Multivariate Techniques [with CD-ROM]. , 2007 .

[55]  Vladimir Tosic,et al.  Guidelines for industrially-based multiple case studies in software engineering , 2009, 2009 Third International Conference on Research Challenges in Information Science.

[56]  Daniela E. Damian,et al.  The role of domain knowledge and cross-functional communication in socio-technical coordination , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[57]  Helen Sharp.  The role of domain knowledge in software design , 1991 .

[58]  Nazim H. Madhavji,et al.  The Impact of Requirements Knowledge and Experience on Software Architecting: An Empirical Study , 2007, 2007 Working IEEE/IFIP Conference on Software Architecture (WICSA'07).

[59]  Daniel M. Berry,et al.  An industrial case study of the impact of domain ignorance on the effectiveness of requirements idea generation during requirements elicitation , 2013, 2013 21st IEEE International Requirements Engineering Conference (RE).

[60]  Tore Dybå,et al.  Evidence-Based Software Engineering for Practitioners , 2005, IEEE Softw..

[61]  Perry R. Hinton.  SPSS Explained , 2004 .

[62]  Natalia Juristo Juzgado,et al.  Understanding the Customer: What Do We Know about Requirements Elicitation? , 2008, IEEE Software.

[63]  Barry W. Boehm,et al.  Software Engineering Economics , 1993, IEEE Transactions on Software Engineering.

[64]  Jr. Frederick P. Brooks,et al.  The mythical man-month (anniversary ed.) , 1995 .

[65]  Irit Hadar,et al.  The role of domain knowledge in requirements elicitation via interviews: an exploratory study , 2012, Requirements Engineering.

[66]  P. Kidwell,et al.  The mythical man-month: Essays on software engineering , 1996, IEEE Annals of the History of Computing.