Software engineering process theory: A multi-method comparison of Sensemaking-Coevolution-Implementation Theory and Function-Behavior-Structure Theory

ContextSoftware engineering has experienced increased calls for attention to theory, including process theory and general theory. However, few process theories or potential general theories have been proposed and little empirical evaluation has been attempted. ObjectiveThe purpose of this paper is to empirically evaluate two previously untested software development process theories - Sensemaking-Coevolution-Implementation Theory (SCI) and the Function-Behavior-Structure Framework (FBS). MethodA survey of more than 1300 software developers is combined with four longitudinal, positivist case studies to achieve a simultaneously broad and deep empirical evaluation. Instrument development, statistical analysis of questionnaire data, case data analysis using a closed-ended, a priori coding scheme and data triangulation are described. ResultsCase data analysis strongly supports SCI, as does analysis of questionnaire response distributions (p<0.001; chi-square goodness of fit test). Furthermore, case-questionnaire triangulation found no evidence that support for SCI varied by participants' gender, education, experience, nationality or the size or nature of their projects. ConclusionsSCI is supported. No evidence of an FBS subculture was found. This suggests that instead of iterating between weakly-coupled phases (analysis, design, coding, testing), it is more accurate and useful to conceptualize development as ad hoc oscillation between making sense of the project context (Sensemaking), simultaneously improving mental representations of the context and design space (Coevolution) and constructing, debugging and deploying software artifacts (Implementation).

[1]  David E. Avison,et al.  Information Systems Development: Methodologies, Techniques and Tools , 1988 .

[2]  Paul Ralph,et al.  Requirements fixation , 2014, ICSE.

[3]  MirakhorliMehdi,et al.  The Twin Peaks of Requirements and Architecture , 2013 .

[4]  Meir M. Lehman,et al.  Approach to a theory of software evolution , 2001, Eighth International Workshop on Principles of Software Evolution (IWPSE'05).

[5]  Terry Bollinger.  The Interplay of Art and Science in Software , 1997, Computer.

[6]  Peter Freeman,et al.  A science of design for software-intensive systems , 2004, CACM.

[7]  A. V. D. Ven,et al.  Explaining Development and Change in Organizations , 1995 .

[8]  Shirley Gregor,et al.  The Nature of Theory in Information Systems , 2006, MIS Q..

[9]  Jens Liegle,et al.  The efficacy of matching information systems development methodologies with application characteristics - an empirical study , 1999, J. Syst. Softw..

[10]  Willard Van Orman Quine,et al.  The Pursuit of Truth , 1990 .

[11]  A. Hunter,et al.  Foundations of multimethod research : synthesizing styles , 2006 .

[12]  Ch Kees Dorst,et al.  On the conceptual framework of John Gero's FBS-model and the prescriptive aims of design methodology , 2007 .

[13]  Kees Dorst,et al.  Comparing paradigms for describing design activity , 1995 .

[14]  Lerina Aversano,et al.  An empirical study on the maintenance of source code clones , 2010, Empirical Software Engineering.

[15]  Meir M. Lehman,et al.  Approach to a theory of software evolution , 2001, Eighth International Workshop on Principles of Software Evolution (IWPSE'05).

[16]  Michael E. Holmes,et al.  Organizational Change and Innovation Processes: Theory and Methods for Research , 2000 .

[17]  Václav Rajlich,et al.  Changing the paradigm of software engineering , 2006, CACM.

[18]  Richard Baskerville,et al.  Generalizing Generalizability in Information Systems Research , 2003, Inf. Syst. Res..

[19]  W. W. Royce,et al.  Managing the development of large software systems: concepts and techniques , 1987, ICSE '87.

[20]  Paul Ralph,et al.  Improving Coverage of Design in Information Systems Education , 2012, ICIS.

[21]  R. Yin.  Case Study Research: Design and Methods , 1984 .

[22]  Paul Ralph,et al.  Is Requirements Engineering Inherently Counterproductive? , 2015, 2015 IEEE/ACM 5th International Workshop on the Twin Peaks of Requirements and Architecture.

[23]  Jonathan Grudin,et al.  Personas, Participatory Design and Product Development: An Infrastructure for Engagement , 2002 .

[24]  Stuart K. Card,et al.  The cost structure of sensemaking , 1993, INTERCHI.

[25]  Nigel Cross,et al.  Design cognition: results from protocol and other empirical studies of design activity , 2016 .

[26]  Frederick P. Brooks,et al.  The Design of Design: Essays from a Computer Scientist , 2010 .

[27]  Barry W. Boehm,et al.  A spiral model of software development and enhancement , 1986, Computer.

[28]  C. Churchman,et al.  Experience and Reflection , 1959 .

[29]  Mary Lou Maher,et al.  Formalising Design Exploration as Co-Evolution , 1996 .

[30]  C. H. Dorst,et al.  Describing Design - A comparison of paradigms , 1997 .

[31]  P ? ? ? ? ? ? ? % ? ? ? ? , 1991 .

[32]  Frank Buschmann.  A Week in the Life of an Architect , 2012, IEEE Software.

[33]  Robert F. DeVellis,et al.  Scale Development: Theory and Applications. , 1992 .

[34]  Chad Saunders,et al.  Cognitive heuristics in software engineering applying and extending anchoring and adjustment to artifact reuse , 2004, IEEE Transactions on Software Engineering.

[35]  John S. Gero,et al.  An Ontological Model of Emergent Design in Software Engineering , 2007 .

[36]  Gary James Jason,et al.  The Logic of Scientific Discovery , 1988 .

[37]  Stefan Biffl,et al.  Investigating the impact of reading techniques on the accuracy of different defect content estimation techniques , 2001, Proceedings Seventh International Software Metrics Symposium.

[38]  Nigel Cross,et al.  Research in design Thinking , 1992 .

[39]  Helen Sharp,et al.  A systematic review of theory use in studies investigating the motivations of software engineers , 2009, TSEM.

[40]  Paul Ralph.  The illusion of requirements in software development , 2012, Requirements Engineering.

[41]  Gérard P. Cachon,et al.  Perspective: Complexity Theory and Organization Science , 1999, Organization Science.

[42]  Kees Dorst,et al.  Design Problems and Design Paradoxes , 2006, Design Issues.

[43]  Ken Schwaber,et al.  Agile Project Management with Scrum , 1980 .

[44]  William M. K. Trochim,et al.  Research methods knowledge base , 2001 .

[45]  Kent L. Beck,et al.  Extreme programming explained - embrace change , 1990 .

[46]  John S. Gero,et al.  The Situated Function - Behaviour - Structure Framework , 2002, AID.

[47]  Donald A. Sch.  The reflective practitioner: how professionals think in action , 1983 .

[48]  Pedro Isaias,et al.  Information Systems Development Methodologies , 2015 .

[49]  Philippe Kruchten,et al.  Casting software design in the function-behavior-structure framework , 2005, IEEE Software.

[50]  Sergiu M. Dascalu,et al.  Unit-level test adequacy criteria for visual dataflow languages and a testing methodology , 2008, TSEM.

[51]  Nigel Cross,et al.  Creativity in the design process: co-evolution of problem–solution , 2001 .

[52]  Scott Curland Chase,et al.  Transforming shape in design: observations from studies of sketching , 2009 .

[53]  D. Schoen,et al.  The Reflective Practitioner: How Professionals Think in Action , 1985 .

[54]  Christopher Alexander.  Notes on the Synthesis of Form , 1964 .

[55]  Ritu Agarwal,et al.  Comprehending Object and Process Models: An Empirical Study , 1999, IEEE Trans. Software Eng..

[56]  Daniel L. Moody,et al.  The “Physics” of Notations: Toward a Scientific Basis for Constructing Visual Notations in Software Engineering , 2009, IEEE Transactions on Software Engineering.

[57]  Paul Ralph,et al.  Developing and Evaluating Software Engineering Process Theories , 2015, 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering.

[58]  Paul Ralph,et al.  How to Develop a General Theory of Software Engineering: Report on the GTSE 2014 Workshop , 2014, SOEN.

[59]  W. W. Royce,et al.  Managing the development of large software systems , 1970 .

[60]  Paul Ralph,et al.  The Sensemaking-Coevolution-Implementation Theory of software design , 2013, Sci. Comput. Program..

[61]  Tore Dybå,et al.  A Systematic Review of Theory Use in Software Engineering Experiments , 2007, IEEE Transactions on Software Engineering.

[62]  David Paul Ralph,et al.  Fundamentals of software design science , 2010 .

[63]  Line Dubé,et al.  Rigor in Information Systems Positivist Case Research: Current Practices , 2003, MIS Q..

[64]  Detmar W. Straub,et al.  Validating Instruments in MIS Research , 1989, MIS Q..

[65]  C. Churchman,et al.  The design of inquiring systems: basic concepts of systems and organization , 1971 .

[66]  G. Goldschmidt.  The dialectics of sketching , 1991 .

[67]  Michele Lanza,et al.  Evaluating defect prediction approaches: a benchmark and an extensive comparison , 2011, Empirical Software Engineering.

[68]  Floyd J. Fowler,et al.  Improving Survey Questions: Design and Evaluation , 1995 .

[69]  R. Audi.  The Cambridge Dictionary of Philosophy , 1995 .

[70]  James R Cook.  Engaged Scholarship: A Guide for Organizational and Social Research , 2014 .

[71]  J. Platt.  Strong Inference , 2007 .

[72]  Paul Ralph,et al.  The Two Paradigms of Software Design , 2013, ArXiv.

[73]  John S. Gero,et al.  Social models of creativity: Integrating the DIFI and FBS frameworks to study creative design , 2005 .

[74]  Per Runeson,et al.  Guidelines for conducting and reporting case study research in software engineering , 2009, Empirical Software Engineering.

[75]  J. Eekels,et al.  On the fundamentals of engineering design science: The geography of engineering design science. Part 1 , 2000 .

[76]  M. Kendall,et al.  The Logic of Scientific Discovery. , 1959 .

[77]  John S. Gero,et al.  Design Prototypes: A Knowledge Representation Schema for Design , 1990, AI Mag..

[78]  Paul Ralph,et al.  Toward a Theory of Debiasing Software Development , 2011, SIGSAND.

[79]  Khaled El Emam,et al.  An Internally Replicated Quasi-Experimental Comparison of Checklist and Perspective-Based Reading of Code Documents , 2001, IEEE Trans. Software Eng..

[80]  Nam P. Suh,et al.  Axiomatic Design: Advances and Applications , 2001 .

[81]  J. Platt.  Strong Inference: Certain systematic methods of scientific thinking may produce much more rapid progress than others. , 1964, Science.

[82]  Mark Harman,et al.  A Theoretical and Empirical Study of Search-Based Testing: Local, Global, and Hybrid Search , 2010, IEEE Transactions on Software Engineering.

[83]  John S. Gero,et al.  Function–behavior–structure paths and their role in analogy-based design , 1996, Artificial Intelligence for Engineering Design, Analysis and Manufacturing.

[84]  Ivar Jacobson,et al.  The Unified Software Development Process , 1999 .

[85]  Per Runeson,et al.  Alignment practices affect distances in software development: a theory and a model , 2014, GTSE 2014.

[86]  John Worrall,et al.  The Methodology of Scientific Research Programmes: Volume 1: Philosophical Papers , 1980 .

[87]  Venkataraman Ramesh,et al.  Research in software engineering: an analysis of the literature , 2002, Inf. Softw. Technol..

[88]  E. Glover.  DICTIONARY OF PSYCHOLOGY , 1959 .

[89]  Jan Mendling,et al.  A Study Into the Factors That Influence the Understandability of Business Process Models , 2011, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[90]  Tony Proctor.  Creative Problem Solving for Managers , 2018 .

[91]  Kjetil Moløkken-Østvold,et al.  A review of software surveys on software effort estimation , 2003, 2003 International Symposium on Empirical Software Engineering, 2003. ISESE 2003. Proceedings..

[92]  Nimal Jayaratna,et al.  Understanding and Evaluating Methodologies: NIMSAD, a Systematic Framework , 1994 .

[93]  Richard Baskerville,et al.  Amethodical systems development: the deferred meaning of systems development methods , 2000 .

[94]  Brian Fitzgerald,et al.  The Transformation of Open Source Software , 2006, MIS Q..

[95]  Paul Ralph,et al.  Report on the first SEMAT workshop on general theory of software engineering (GTSE 2012) , 2013, SOEN.

[96]  W. L. Smith,et al.  On the limitations of software process assessment and the recognition of a required re-orientation for global process improvement , 1998, Software Quality Journal.

[97]  Ivar Jacobson,et al.  Where's the Theory for Software Engineering? , 2012, IEEE Software.

[98]  Paul Ralph,et al.  Comparing Two Software Design Process Theories , 2010, DESRIST.

[99]  Paul Ralph,et al.  The dimensions of software engineering success , 2014, ICSE.

[100]  Kalle Lyytinen,et al.  Boundary Objects in Design: An Ecological View of Design Artifacts , 2007, J. Assoc. Inf. Syst..

[101]  Brenda Dervin,et al.  Sense-making theory and practice: an overview of user interests in knowledge seeking and use , 1998, J. Knowl. Manag..

[102]  R. Wolfe.  ORGANIZATIONAL INNOVATION: REVIEW, CRITIQUE AND SUGGESTED RESEARCH DIRECTIONS* , 1994 .

[103]  Kent Beck,et al.  Extreme Programming Explained: Embrace Change (2nd Edition) , 2004 .

[104]  Paul Ralph,et al.  A Proposal for a Formal Definition of the Design Concept , 2009 .

[105]  Per Galle,et al.  The ontology of Gero's FBS model of designing , 2009 .

[106]  D. Wegner.  Transactive Memory: A Contemporary Analysis of the Group Mind , 1987 .

[107]  Colin Atkinson,et al.  An experimental comparison of reading techniques for defect detection in UML design documents , 2000, J. Syst. Softw..

[108]  John S. Gero,et al.  A function–behavior–structure ontology of processes , 2007, Artificial Intelligence for Engineering Design, Analysis and Manufacturing.

[109]  Paul Ralph,et al.  Possible core theories for software engineering , 2013, 2013 2nd SEMAT Workshop on a General Theory of Software Engineering (GTSE).

[110]  Franz von Kutschera,et al.  Causation , 1993, J. Philos. Log..

[111]  Philippe Kruchten,et al.  The Rational Unified Process: An Introduction , 1998 .

[112]  Kathleen M. Sutcliffe,et al.  Special Issue: Frontiers of Organization Science, Part 1 of 2: Organizing and the Process of Sensemaking , 2005, Organ. Sci..