Empirical research for software architecture decision making: An analysis

Abstract Context Despite past empirical research in software architecture decision making, we have not yet systematically studied how to perform such empirical research. Software architecture decision making involves humans, their behavioral issues and practice. As such, research on decision making needs to involve not only engineering but also social science research methods. Objective This paper studies empirical research on software architecture decision making. We want to understand what research methods have been used to study human decision making in software architecture. Further, we want to provide guidance for future studies. Method We analyzed research papers on software architecture decision making. We classified the papers according to different sub-dimensions of empirical research design like research logic, research purpose, research methodology and process. We introduce the study focus matrix and the research cycle to capture the focus and the goals of a software architecture decision making study. We identify gaps in current software architecture decision making research according to the classification and discuss open research issues inspired by social science research. Conclusion We show the variety of research designs and identify gaps with respect to focus and goals. Few papers study decision making behavior in software architecture design. Also these researchers study mostly the process and much less the outcome and the factors influencing decision making. Furthermore, there is a lack of improvements for software architecture decision making and in particular insights into behavior have not led to new practices. The study focus matrix and the research cycle are two new instruments for researchers to position their research clearly. This paper provides a retrospective for the community and an entry point for new researchers to design empirical studies that embrace the human role in software architecture decision making.

[1]  Danny Weyns,et al.  From Craft to Science: The Road Ahead for Empirical Software Engineering Research , 2018, 2018 IEEE/ACM 40th International Conference on Software Engineering: New Ideas and Emerging Technologies Results (ICSE-NIER).

[2]  John W. Creswell,et al.  Research Design: Qualitative, Quantitative, and Mixed Methods Approaches , 2010 .

[3]  D. Kahneman.  Maps of Bounded Rationality: Psychology for Behavioral Economics , 2003 .

[4]  Pearl Brereton,et al.  Systematic literature reviews in software engineering - A systematic literature review , 2009, Inf. Softw. Technol..

[5]  Tua Björklund,et al.  Initial mental representations of design problems: Differences between experts and novices , 2013 .

[6]  John M. Carroll,et al.  Design rationale: concepts, techniques, and use , 1996 .

[7]  Danny Weyns,et al.  Empirical Research in Software Architecture: How Far have We Come? , 2016, 2016 13th Working IEEE/IFIP Conference on Software Architecture (WICSA).

[8]  Jan Bosch,et al.  Software Architecture as a Set of Architectural Design Decisions , 2005, 5th Working IEEE/IFIP Conference on Software Architecture (WICSA'05).

[9]  Claes Wohlin,et al.  Towards a decision-making structure for selecting a research design in empirical software engineering , 2014, Empirical Software Engineering.

[10]  Jami J. Shah,et al.  Empirical Studies of Designer Thinking: Past, Present, and Future , 2015 .

[11]  D. Kahneman.  Thinking, Fast and Slow , 2011 .

[12]  Alan J. Mayne,et al.  Decision Research: A Field Guide , 1990 .

[13]  Adam A. Porter,et al.  Empirical studies of software engineering: a roadmap , 2000, ICSE '00.

[14]  Masaki Suwa,et al.  What do architects and students perceive in their design sketches? A protocol analysis , 1997 .

[15]  David C. Brown,et al.  Reasoning with Design Rationale , 2000, AID.

[16]  Muhammad Ali Babar,et al.  Software Architecture Knowledge Management: Theory and Practice , 2009 .

[17]  Claes Wohlin,et al.  Guidelines for snowballing in systematic literature studies and a replication in software engineering , 2014, EASE '14.

[18]  Roel Wieringa,et al.  Design Science Methodology for Information Systems and Software Engineering , 2014, Springer Berlin Heidelberg.

[19]  Sung Pil Moon,et al.  Supporting Complex Decision Making Through Option Awareness , 2013 .

[20]  Philippe Kruchten,et al.  What do software architects really do? , 2008, J. Syst. Softw..

[21]  Antony Tang,et al.  Decision making in software architecture , 2016, J. Syst. Softw..

[22]  Daniela E. Damian,et al.  Selecting Empirical Methods for Software Engineering Research , 2008, Guide to Advanced Empirical Software Engineering.

[23]  Janet McDonnell,et al.  Analysing Design Behaviour: The Design Thinking Research Symposia Series , 2007 .

[24]  Jeff Tyree,et al.  Architecture decisions: demystifying architecture , 2005, IEEE Software.

[25]  H. Simon,et al.  Models of Bounded Rationality: Empirically Grounded Economic Reason , 1997 .

[26]  Kees Dorst,et al.  The core of ‘design thinking’ and its application , 2011 .

[27]  Claes Wohlin,et al.  Systematic literature studies: Database searches vs. backward snowballing , 2012, Proceedings of the 2012 ACM-IEEE International Symposium on Empirical Software Engineering and Measurement.

[28]  R. Thaler.  Misbehaving: The Making of Behavioral Economics , 2015 .

[29]  Claes Wohlin,et al.  Experimentation in Software Engineering , 2012, Springer Berlin Heidelberg.

[30]  Matthew M. Mehalik,et al.  What Constitutes Good Design? A Review of Empirical Studies of Design Processes* , 2006 .

[31]  Rafael Capilla,et al.  In two minds: how reflections influence software design thinking , 2016, J. Softw. Evol. Process..

[32]  Muhammad Ali Babar,et al.  10 years of software architecture knowledge management: Practice and future , 2016, J. Syst. Softw..

[33]  Tom-Michael Hesse,et al.  Human Aspects in Software Architecture Decision Making: A Literature Review , 2017, 2017 IEEE International Conference on Software Architecture (ICSA).

[34]  Per Runeson,et al.  Guidelines for conducting and reporting case study research in software engineering , 2009, Empirical Software Engineering.