What works for whom, where, when, and why? On the role of context in empirical software engineering

Context is a central concept in empirical software engineering. It is one of the distinctive features of the discipline and it is an indispensable part of software practice. It is likely responsible for one of the most challenging methodological and theoretical problems: study-to-study variation in research findings. Still, empirical software engineering research is mostly concerned with attempts to identify universal relationships that are independent of how work settings and other contexts interact with the processes important to software practice. The aim of this paper is to provide an overview of how context affects empirical research and how empirical software engineering research can be better `contextualized' in order to provide a better understanding of what works for whom, where, when, and why. We exemplify the importance of context with examples from recent systematic reviews and offer recommendations on the way forward.

[1]  E. Trist,et al.  The Evolution of Socio-Technical Systems: A Conceptual Framework and an Action Research Program , 1981 .

[2]  Reidar Conradi,et al.  Empirical Methods and Studies in Software Engineering: Experiences from Esernet , 2003 .

[3]  Peng Xu,et al.  Software Process Tailoring: An Empirical Investigation , 2007, J. Manag. Inf. Syst..

[4]  D. Bonn,et al.  Two heads are better than one , 1999, Nature Methods.

[5]  Tore Dybå,et al.  The Future of Empirical Methods in Software Engineering Research , 2007, Future of Software Engineering (FOSE '07).

[6]  Robert L. Glass,et al.  Science and substance: a challenge to software engineers , 1994, IEEE Software.

[7]  Tore Dybå,et al.  Conducting realistic experiments in software engineering , 2002, Proceedings International Symposium on Empirical Software Engineering.

[8]  Y. Fried,et al.  Location, location, location: contextualizing organizational research* , 2001 .

[9]  Victor R. Basili,et al.  The TAME Project: Towards Improvement-Oriented Software Environments , 1988, IEEE Trans. Software Eng..

[10]  D. Whetten.  An Examination of the Interface between Context and Theory Applied to the Study of Chinese Organizations , 2009, Management and Organization Review.

[11]  Snejina Michailova,et al.  Contextualizing in International Business research: Why do we need more of it and how can we be better at it? , 2011 .

[12]  G. Johns.  The Essential Impact of Context on Organizational Behavior , 2006 .

[13]  Dag I. K. Sjøberg,et al.  Evaluating methods and technologies in software engineering with Respect to Developers' skill level , 2012, EASE.

[14]  Shari Lawrence Pfleeger,et al.  Preliminary Guidelines for Empirical Research in Software Engineering , 2002, IEEE Trans. Software Eng..

[15]  Dag I. K. Sjøberg,et al.  Evaluating the effect of a delegated versus centralized control style on the maintainability of object-oriented software , 2004, IEEE Transactions on Software Engineering.

[16]  Luiz Fernando Capretz,et al.  Making Sense of Software Development and Personality Types , 2010, IT Professional.

[17]  C. Goodwin,et al.  Rethinking Context: Language as an Interactive Phenomenon , 1992 .

[18]  Tore Dybå,et al.  Inferring Skill from Tests of Programming Performance: Combining Time and Quality , 2011, 2011 International Symposium on Empirical Software Engineering and Measurement.

[19]  Forrest Shull,et al.  Building Knowledge through Families of Experiments , 1999, IEEE Trans. Software Eng..

[20]  Larry V. Hedges,et al.  How hard is hard science, how soft is soft science? The empirical cumulativeness of research. , 1987 .

[21]  Tore Dybå,et al.  Evaluating Pair Programming with Respect to System Complexity and Programmer Expertise , 2007, IEEE Transactions on Software Engineering.

[22]  Elaine Chin.  Redefining “Context” in Research on Writing , 1994 .

[23]  Tore Dybå,et al.  A Dynamic Model of Software Engineering Knowledge Creation , 2003 .

[24]  Tore Dybå,et al.  Qualitative research in software engineering , 2011, Empirical Software Engineering.

[25]  Tore Dybå,et al.  Factors of software process improvement success in small and large organizations: an empirical study in the scandinavian context , 2003, ESEC/FSE-11.

[26]  Rupert Brown,et al.  Group Processes: Dynamics Within and Between Groups , 1988 .

[27]  Tim Menzies,et al.  Local vs. global models for effort estimation and defect prediction , 2011, 2011 26th IEEE/ACM International Conference on Automated Software Engineering (ASE 2011).

[28]  D. A. Kenny,et al.  The moderator-mediator variable distinction in social psychological research: conceptual, strategic, and statistical considerations. , 1986, Journal of personality and social psychology.

[29]  Daniela Cruzes,et al.  Recommended Steps for Thematic Synthesis in Software Engineering , 2011, 2011 International Symposium on Empirical Software Engineering and Measurement.

[30]  C Freitas,et al.  [I believe in...]. , 2000, Servir.

[31]  Gary Johns,et al.  Substantive and methodological constraints on behavior and attitudes in organizational research , 1991 .

[32]  Tore Dybå,et al.  The effectiveness of pair programming: A meta-analysis , 2009, Inf. Softw. Technol..

[33]  Mark A. Griffin,et al.  Specifying organizational contexts: systematic links between contexts and processes in organizational behavior , 2007 .

[34]  Kipling D. Williams,et al.  PROCESSES Social Loafing: A Meta-Analytic Review and Theoretical Integration , 2022 .

[35]  Tore Dybå,et al.  Challenges and Recommendations When Increasing the Realism of Controlled Software Engineering Experiments , 2003, ESERNET.

[36]  Tore Dybå,et al.  Measuring software methodology usage: challenges of conceptualization and operationalization , 2005, 2005 International Symposium on Empirical Software Engineering, 2005..

[37]  K. Petersen,et al.  Context in industrial software engineering research , 2009, 2009 3rd International Symposium on Empirical Software Engineering and Measurement.

[38]  Venugopal Balijepally,et al.  Are Two Heads Better than One for Software Development? The Productivity Paradox of Pair Programming , 2009, MIS Q..

[39]  Tore Dybå,et al.  Overcoming Barriers to Self-Management in Software Teams , 2009, IEEE Software.

[40]  Tore Dybå.  Improvisation in Small Software Organizations , 2000, IEEE Software.

[41]  Rory V. O'Connor,et al.  The situational factors that affect the software development process: Towards a comprehensive reference framework , 2012, Inf. Softw. Technol..

[42]  Amela Karahasanovic,et al.  A survey of controlled experiments in software engineering , 2005, IEEE Transactions on Software Engineering.

[43]  Forrest Shull,et al.  How Effective Is Test-Driven Development? , 2011, Making Software.

[44]  Daniela Cruzes,et al.  Research synthesis in software engineering: A tertiary study , 2011, Inf. Softw. Technol..

[45]  R. I. Sutton,et al.  Organizational behavior: linking individuals and groups to organizational contexts. , 1993, Annual review of psychology.

[46]  D. Whetten.  What Constitutes a Theoretical Contribution , 1989 .

[47]  Claes Wohlin,et al.  Using Students as Subjects—A Comparative Study of Students and Professionals in Lead-Time Impact Assessment , 2000, Empirical Software Engineering.

[48]  Tore Dybå,et al.  Evidence-Based Software Engineering for Practitioners , 2005, IEEE Softw..

[49]  N. Charness,et al.  Expert Performance Its Structure and Acquisition , 2002 .

[50]  Tore Dybå,et al.  Building Theories in Software Engineering , 2008, Guide to Advanced Empirical Software Engineering.

[51]  Friederike Welter,et al.  Contextualizing Entrepreneurship—Conceptual Challenges and Ways Forward , 2011 .

[52]  Forrest Shull,et al.  Are Two Heads Better than One? On the Effectiveness of Pair Programming , 2007, IEEE Software.