Characterizing industry-academia collaborations in software engineering: evidence from 101 projects

Research collaboration between industry and academia supports improvement and innovation in industry and helps ensure the industrial relevance of academic research. However, many researchers and practitioners in the community believe that the level of joint industry-academia collaboration (IAC) projects in Software Engineering (SE) research is relatively low, creating a barrier between research and practice. The goal of the empirical study reported in this paper is to explore and characterize the state of IAC with respect to industrial needs, developed solutions, impacts of the projects and also a set of challenges, patterns and anti-patterns identified by a recent Systematic Literature Review (SLR) study. To address the above goal, we conducted an opinion survey among researchers and practitioners with respect to their experience in IAC. Our dataset includes 101 data points from IAC projects conducted in 21 different countries. Our findings include: (1) the most popular topics of the IAC projects, in the dataset, are: software testing, quality, process, and project managements; (2) over 90% of IAC projects result in at least one publication; (3) almost 50% of IACs are initiated by industry, busting the myth that industry tends to avoid IACs; and (4) 61% of the IAC projects report having a positive impact on their industrial context, while 31% report no noticeable impacts or were “not sure”. To improve this situation, we present evidence-based recommendations to increase the success of IAC projects, such as the importance of testing pilot solutions before using them in industry. This study aims to contribute to the body of evidence in the area of IAC, and benefit researchers and practitioners. Using the data and evidence presented in this paper, they can conduct more successful IAC projects in SE by being aware of the challenges and how to overcome them, by applying best practices (patterns), and by preventing anti-patterns.

[1]  Albert Banal-Estañol,et al.  Research Output From University–Industry Collaborative Projects , 2013 .

[2]  Carlo Ghezzi,et al.  Determining the Impact of Software Engineering Research on Practice , 2008, Computer.

[3]  Wolfgang Emmerich,et al.  The Impact of Research on Middleware Technology , 2007, 29th International Conference on Software Engineering (ICSE'07 Companion).

[4]  Per Runeson,et al.  Guidelines for conducting and reporting case study research in software engineering , 2009, Empirical Software Engineering.

[5]  Forrest Shull,et al.  Impact of research on practice in the field of inspections, reviews and walkthroughs: learning from successful industrial uses , 2008, SOEN.

[6]  Vahid Garousi,et al.  Using Citation Behavior to Rethink Academic Impact in Software Engineering , 2015, 2015 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM).

[7]  J. R. Larson,et al.  Research strategies and tactics in I/O psychology , 1990 .

[8]  Udo Lindemann,et al.  Impact of Design Research on Industrial Practice , 2016 .

[9]  David Eichelberger,et al.  Handbook Of Psychological Testing , 2016 .

[10]  Claes Wohlin,et al.  Software Engineering Research under the Lamppost , 2013, ICSOFT.

[11]  Ann Branaman,et al.  Status, Power and Ritual Interaction: A Relational Reading of Durkheim, Goffman and Collins , 2014 .

[12]  Vahid Garousi,et al.  Software test-code engineering: A systematic mapping , 2015, Inf. Softw. Technol..

[13]  Richard E. Fairley,et al.  Guide to the Software Engineering Body of Knowledge (SWEBOK(R)): Version 3.0 , 2014 .

[14]  Vahid Garousi,et al.  Development of Scientific Software: a Systematic Mapping, a bibliometrics Study, and a Paper Repository , 2013, Int. J. Softw. Eng. Knowl. Eng..

[15]  Daniela Cruzes,et al.  Research synthesis in software engineering: A tertiary study , 2011, Inf. Softw. Technol..

[16]  I. R. Pashby,et al.  Effective University - Industry Interaction: A Multi-case Evaluation of Collaborative R&D Projects , 2002 .

[17]  Ivar Jacobson,et al.  Where's the Theory for Software Engineering? , 2012, IEEE Software.

[18]  Mike Cohn,et al.  Succeeding with Agile: Software Development Using Scrum , 2009 .

[19]  David Lo,et al.  How practitioners perceive the relevance of software engineering research , 2015, ESEC/SIGSOFT FSE.

[20]  Vahid Garousi,et al.  Dataset For Survey Of Industry-Academia Collaboration In Software Engineering (Phase 1) , 2017 .

[21]  Per Runeson,et al.  Towards a framework to support large scale sampling in software engineering surveys , 2014, ESEM '14.

[22]  Vahid Garousi,et al.  Challenges and best practices in industry-academia collaborations in software engineering: A systematic literature review , 2016, Inf. Softw. Technol..

[23]  Amela Karahasanovic,et al.  A survey of controlled experiments in software engineering , 2005, IEEE Transactions on Software Engineering.

[24]  Vahid Garousi,et al.  Questions For Survey Of Industry-Academia Collaborations In Software Engineering , 2017 .

[25]  Colin Potts,et al.  Software-engineering research revisited , 1993, IEEE Software.

[26]  Muhammad Ali Babar,et al.  Conducting empirical studies in industry: Balancing rigor and relevance , 2013, 2013 1st International Workshop on Conducting Empirical Studies in Industry (CESI).

[27]  Antony Tang.  On the Worthiness of Software Engineering Research , 2017 .

[28]  Lones Smith,et al.  Assortative Matching and Search , 2000 .

[29]  Jun-You Lin.  Balancing industry collaboration and academic innovation: The contingent role of collaboration-specific attributes , 2017 .

[30]  Ana M. Bernardos Barbolla,et al.  Critical factors for success in university–industry research projects , 2009 .

[31]  Tony Gorschek,et al.  A Model for Technology Transfer in Practice , 2006, IEEE Software.

[32]  R. Ferber.  Research By Convenience , 1977 .

[33]  A. Huberman,et al.  Qualitative Data Analysis: A Methods Sourcebook , 1994 .

[34]  Thomas Arts,et al.  Agile Collaborative Research: Action Principles for Industry-Academia Collaboration , 2011, IEEE Software.

[35]  Klaus Marius Hansen,et al.  Ready! Set! Go! An Action Research Agenda for Software Architecture Research , 2008, Seventh Working IEEE/IFIP Conference on Software Architecture (WICSA 2008).

[36]  Vahid Garousi,et al.  Industry–academia collaborations in software testing: experience and success stories from Canada and Turkey , 2016, Software Quality Journal.

[37]  Kai Petersen,et al.  Survey Guidelines in Software Engineering: An Annotated Review , 2016, ESEM.

[38]  S. Fox,et al.  Reframing the relevance of research to practice , 2016 .

[39]  Tom DeMarco,et al.  Software Creativity 2.0 , 2006 .

[40]  Jeffrey C. Carver,et al.  Industry-Academia Collaboration in Software Engineering , 2018, IEEE Softw..

[41]  M. G. Vedavyas.  Proposed Maturity Model for Industry-Academia Collaboration , 2016 .

[42]  T. D. Kemper,et al.  Status, Power and Ritual Interaction: A Relational Reading of Durkheim, Goffman and Collins , 2013 .

[43]  Guilherme Horta Travassos,et al.  Action research use in software engineering: An initial survey , 2009, ESEM 2009.

[44]  Anselm L. Strauss,et al.  Basics of qualitative research : techniques and procedures for developing grounded theory , 1998 .

[45]  Mehrdad Sabetzadeh,et al.  The Case for Context-Driven Software Engineering Research: Generalizability Is Overrated , 2017, IEEE Softw..

[46]  Vahid Garousi,et al.  Comparing automated visual GUI testing tools: an industrial case study , 2017, A-TEST@ESEC/SIGSOFT FSE.

[47]  Giancarlo Succi,et al.  What do software engineers care about? gaps between research and practice , 2017, ESEC/SIGSOFT FSE.

[48]  Mehrdad Sabetzadeh,et al.  Software Engineering Research and Industry: A Symbiotic Relationship to Foster Impact , 2018, IEEE Software.

[49]  Alena Buchalcevová,et al.  Test Governance Framework for contracted IS development: Ethnographically informed action research , 2015, Information and Software Technology.

[50]  Kai Petersen,et al.  Guidelines for conducting systematic mapping studies in software engineering: An update , 2015, Inf. Softw. Technol..

[51]  G. Gobo.  Sampling, Representativeness and Generalizability , 2004 .

[52]  S. Ankrah,et al.  Universities-Industry Collaboration: A Systematic Review , 2015 .

[53]  Walter F. Tichy,et al.  Summary of the Dagstuhl workshop on future directions in software engineering: February 17–21, 1992, Schloß Dagstuhl , 1993, SOEN.

[54]  Christof Ebert,et al.  50 Years of Software Engineering: Progress and Perils , 2018, IEEE Software.

[55]  Daniela Cruzes,et al.  Recommended Steps for Thematic Synthesis in Software Engineering , 2011, 2011 International Symposium on Empirical Software Engineering and Measurement.

[56]  N. Hoffart.  Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory , 2000 .

[57]  Vahid Garousi,et al.  What We Know about Software Test Maturity and Test Process Improvement , 2017, IEEE Software.

[58]  Robert L. Glass,et al.  The software-research crisis , 1994, IEEE Software.

[59]  Guilherme Horta Travassos,et al.  Action research use in software engineering: An initial survey , 2009, 2009 3rd International Symposium on Empirical Software Engineering and Measurement.

[60]  Vahid Garousi,et al.  Cost, benefits and quality of software development documentation: A systematic mapping , 2015, J. Syst. Softw..

[61]  Lefteris Angelis,et al.  The Success Factors Powering Industry-Academia Collaboration , 2012, IEEE Software.

[62]  Nicolas Carayol,et al.  Objectives , Agreements and Matching in Science-Industry Collaborations : Reassembling the Pieces of the Puzzle , 2003 .

[63]  Simon P. Philbin.  Process model for university‐industry research collaboration , 2008 .

[64]  Claes Wohlin,et al.  Using Students as Subjects—A Comparative Study of Students and Professionals in Lead-Time Impact Assessment , 2000, Empirical Software Engineering.

[65]  Lionel C. Briand.  Useful software engineering research - leading a double-agent life , 2011, ICSM.

[66]  Daniela Cruzes,et al.  Synthesizing evidence in software engineering research , 2010, ESEM '10.

[67]  S. Ankrah,et al.  Social capital to facilitate ‘engineered’ university–industry collaboration for technology transfer: A dynamic perspective , 2016 .

[68]  Denisa Mindruta,et al.  Value Creation in University-Firm Research Collaborations: A Matching Approach , 2008 .

[69]  Egon Berghout,et al.  The Goal/Question/Metric method: a practical guide for quality improvement of software development , 1999 .

[70]  Lionel Briand,et al.  Embracing the Engineering Side of Software Engineering , 2012, IEEE Softw..

[71]  Shirley Gregor,et al.  The Nature of Theory in Information Systems , 2006, MIS Q..

[72]  Vahid Garousi,et al.  Industry-academia collaborations in software engineering: An empirical analysis of challenges, patterns and anti-patterns in research projects , 2017, EASE.

[73]  Albert Banal-Estañol,et al.  Endogenous Matching in University-Industry Collaboration: Theory and Empirical Evidence from the United Kingdom , 2018, Manag. Sci..

[74]  Kai Petersen,et al.  Action research as a model for industry-academia collaboration in the software engineering context , 2014, WISE@ASE.

[75]  S. J. Lamprecht,et al.  Models for technology research collaboration between industry and academia in South Africa , 2012, Proceedings of the 2012 4th IEEE Software Engineering Colloquium (SE).

[76]  Rajen K. Gupta,et al.  Making academic research more relevant: A few suggestions , 2014 .

[77]  Daniela E. Damian,et al.  Selecting Empirical Methods for Software Engineering Research , 2008, Guide to Advanced Empirical Software Engineering.

[78]  Katriina Viikki,et al.  Integrating Human-Centered Design into Software Development: An Action Research Study in the Automation Industry , 2011, 2011 37th EUROMICRO Conference on Software Engineering and Advanced Applications.

[79]  Vahid Garousi,et al.  Selecting the Right Topics for Industry-Academia Collaborations in Software Testing: An Experience Report , 2016, 2016 IEEE International Conference on Software Testing, Verification and Validation (ICST).

[80]  Manfred Broy.  Yesterday, Today, and Tomorrow: 50 Years of Software Engineering , 2018, IEEE Software.

[81]  Claes Wohlin.  Empirical software engineering research with industry: Top 10 challenges , 2013, 2013 1st International Workshop on Conducting Empirical Studies in Industry (CESI).

[82]  Ann Zimmerman,et al.  A Theory of Remote Scientific Collaboration , 2008 .

[83]  Jeffrey S. Foster,et al.  SCORE: agile research group management , 2010, Commun. ACM.

[84]  William G. Griswold,et al.  The Birth of Refactoring: A Retrospective on the Nature of High-Impact Software Engineering Research , 2015, IEEE Software.

[85]  Guilherme Horta Travassos,et al.  Sampling improvement in software engineering surveys , 2014, ESEM '14.

[86]  Dar-Zen Chen,et al.  Technological Forecasting & Social Change How can academic innovation performance in university–industry collaboration be improved? , 2017 .

[87]  H. Dieter Rombach,et al.  Research Collaborations between Academia and Industry , 2007, Future of Software Engineering (FOSE '07).

[88]  Judith Segal,et al.  Developing Scientific Software , 2008, IEEE Software.

[89]  Claes Wohlin,et al.  Experimentation in software engineering: an introduction , 2000 .

[90]  Robert M. Mason.  Not Either/Or: Research in Pasteur's Quadrant , 2001, Commun. Assoc. Inf. Syst..

[91]  H. Kölsch,et al.  Reporting bias in medical research - a narrative review , 2010, Trials.