Qualitative software engineering research -- reflections and guidelines

Researchers are increasingly recognizing the importance of human aspects in software development and since qualitative methods are used to, in-depth, explore human behavior, we believe that studies using such techniques will become more common. Existing qualitative software engineering guidelines do not cover the full breadth of qualitative methods and knowledge on using them found in the social sciences. The aim of this study was thus to extend the software engineering research community's current body of knowledge regarding available qualitative methods and provide recommendations and guidelines for their use. With the support of an epistemological argument and a literature review, we suggest that future research would benefit from (1) utilizing a broader set of research methods, (2) more strongly emphasizing reflexivity, and (3) employing qualitative guidelines and quality criteria. We present an overview of three qualitative methods commonly used in social sciences but rarely seen in software engineering research, namely interpretative phenomenological analysis, narrative analysis, and discourse analysis. Furthermore, we discuss the meaning of reflexivity in relation to the software engineering context and suggest means of fostering it. Our paper will help software engineering researchers better select and then guide the application of a broader set of qualitative research methods.

[1]  James Noble,et al.  Developing a grounded theory to explain the practices of self-organizing Agile teams , 2012, Empirical Software Engineering.

[2]  Jolanta Aritz,et al.  Cognitive Organization and Identity Maintenance in Multicultural Teams , 2010 .

[3]  Barbara Kitchenham,et al.  Procedures for Performing Systematic Reviews , 2004 .

[4]  Emily Foster,et al.  Adolescents' Experience of 'Adjustment' to Life with Diabetes: An Interpretative Phenomenological Analysis , 2010 .

[5]  Jonathan A. Smith,et al.  Interpretative phenomenological analysis. , 2008, Qualitative research in psychology: Expanding perspectives in methodology and design (2nd ed.)..

[6]  John McLeod,et al.  Qualitative approaches to research in counselling and psychotherapy: Issues and challenges , 1996 .

[7]  Martha S. Feldman,et al.  Making Sense of Stories: A Rhetorical Approach to Narrative Analysis , 2002 .

[8]  Kathy Murphy,et al.  Ethical challenges in qualitative research: examples from practice. , 2010, Nurse researcher.

[9]  Per Runeson,et al.  Guidelines for conducting and reporting case study research in software engineering , 2009, Empirical Software Engineering.

[10]  A. Langley.  Strategies for Theorizing from Process Data , 1999 .

[11]  S. Sofaer.  Qualitative methods: what are they and why use them? , 1999, Health services research.

[12]  Reva Brown,et al.  Doing Your Dissertation in Business and Management: The Reality of Researching and Writing , 2006 .

[13]  Jim Highsmith,et al.  Agile Software Development Ecosystems , 2002 .

[14]  Paul Ralph,et al.  Grounded Theory in Software Engineering Research: A Critical Review and Guidelines , 2016, 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE).

[15]  F. Heider,et al.  An experimental study of apparent behavior , 1944 .

[16]  J. Brocki,et al.  A critical evaluation of the use of interpretative phenomenological analysis (IPA) in health psychology , 2006 .

[17]  R. Stebbins.  Exploratory research in the social sciences , 2001 .

[18]  Barbara Czarniawska.  A Narrative Approach to Organization Studies , 1998 .

[19]  Martha S. Feldman,et al.  The uniqueness paradox in organizational stories. , 1983 .

[20]  Brendan Gough,et al.  Reflexivity in qualitative psychological research , 2017 .

[21]  Nkoli Ezumah,et al.  Quality assurance of qualitative research: a review of the discourse , 2011, Health research policy and systems.

[22]  Lynne J. Millward,et al.  The transition to motherhood in an organizational context: An interpretative phenomenological analysis , 2006 .

[23]  Charlotte Burck,et al.  Comparing qualitative research methodologies for systemic research: the use of grounded theory, discourse analysis and narrative analysis , 2005 .

[24]  R. Josselson,et al.  The Ethical Attitude in Narrative Research: Principles and Practicalities. , 2007 .

[25]  Marie Clancy,et al.  Is reflexivity the key to minimising problems of interpretation in phenomenological research? , 2013, Nurse researcher.

[26]  Anselm L. Strauss,et al.  Basics of qualitative research : techniques and procedures for developing grounded theory , 1998 .

[27]  J. Overcash,et al.  Narrative research: a review of methodology and relevance to clinical practice. , 2003, Critical reviews in oncology/hematology.

[28]  Colum Cronin,et al.  Brief Encounters with Qualitative Methods in Health Research: Phenomenology and Interpretative Phenomenological Analysis , 2016 .

[29]  D. Clandinin,et al.  Narrative Inquiry: Experience and Story in Qualitative Research , 1999 .

[30]  R. Elliott,et al.  Evolving guidelines for publication of qualitative research studies in psychology and related fields. , 1999, The British journal of clinical psychology.

[31]  Nila Ginger Hofman.  Toward critical research ethics: transforming ethical conduct in qualitative health care research. , 2004, Health care for women international.

[32]  Robert Cheng Huat Chia.  Discourse Analysis Organizational Analysis , 2000 .

[33]  C. Riessman.  ANALYSIS OF PERSONAL NARRATIVES , 2001 .

[34]  Phillip A. Laplante,et al.  A content analysis process for qualitative software engineering research , 2017, Innovations in Systems and Software Engineering.

[35]  Gerald M. Weinberg,et al.  Psychology of computer programming , 1971 .

[36]  C. Willig.  Introducing Qualitative Research in Psychology , 2001 .

[37]  D. Boje,et al.  Language and Organization: The Doing of Discourse , 2004 .

[38]  Robert Feldt,et al.  Human Factors Related Challenges in Software Engineering -- An Industrial Perspective , 2015, 2015 IEEE/ACM 8th International Workshop on Cooperative and Human Aspects of Software Engineering.

[39]  Angela Morgan,et al.  Discourse Analysis: An Overview for the Neophyte Researcher , 2010 .

[40]  Joseph A. Maxwell,et al.  Qualitative Research Design: An Interactive Approach , 1996 .

[41]  Stefanie Buckner,et al.  TAKING THE DEBATE ON REFLEXIVITY FURTHER , 2005 .

[42]  Dermot Moran,et al.  Husserl's Crisis of the European Sciences and Transcendental Phenomenology: An Introduction , 2012 .

[43]  Caroline Bradbury-Jones,et al.  Enhancing rigour in qualitative health research: exploring subjectivity through Peshkin's I's. , 2007, Journal of advanced nursing.

[44]  Jonathan A. Smith,et al.  Interpretative phenomenological analysis and the psychology of health and illness , 2013 .

[45]  D. Watt,et al.  On Becoming a Qualitative Researcher: The Value of Reflexivity , 2007 .

[46]  M. Crossley,et al.  Introducing Narrative Psychology , 2000 .

[47]  M. Hoepfl.  Choosing Qualitative Research: A Primer for Technology Education Researchers , 1997 .

[48]  P Brew.  Now/I see it , 2015 .

[49]  M. Lecompte,et al.  Problems of Reliability and Validity in Ethnographic Research , 1982 .

[50]  R. Whittemore,et al.  Validity in Qualitative Research , 2001, Qualitative health research.

[51]  John Lyons,et al.  Introduction to Theoretical Linguistics , 1971 .

[52]  C. Harvey,et al.  A Conceptual Review of Functioning: Implications for the Development of Consumer Outcome Measures , 2001, The Australian and New Zealand journal of psychiatry.

[53]  Theodore R. Sarbin,et al.  Narrative Psychology: The Storied Nature of Human Conduct , 1986 .

[54]  John S. McClure.  Semiotics: The Basics , 2004 .

[55]  N. Petty,et al.  Ready for a paradigm shift? Part 2: introducing qualitative research methodologies and methods. , 2012, Manual therapy.

[56]  J. Maxwell.  Understanding and Validity in Qualitative Research , 1992 .

[57]  U. Flick.  An Introduction to Qualitative Research , 1998 .

[58]  M. Mitic,et al.  Understanding challenges of qualitative research: rhetorical issues and reality traps , 2012 .

[59]  Sarah J. Tracy.  Qualitative Quality: Eight “Big-Tent” Criteria for Excellent Qualitative Research , 2010 .

[60]  Janice Singer,et al.  For the Special issue on Qualitative Software Engineering Research , 2007, Inf. Softw. Technol..

[61]  P. Liamputtong.  Qualitative Research Methods , 2005 .

[62]  Douglas Harper,et al.  The visual ethnographic narrative 1 , 1987 .

[63]  C. Pope,et al.  Assessing quality in qualitative research , 2000, BMJ : British Medical Journal.

[64]  Michael J. Gill,et al.  Elite identity and status anxiety: An interpretative phenomenological analysis of management consultants , 2015 .

[65]  A. Onwuegbuzie,et al.  Mixed Methods Research: A Research Paradigm Whose Time Has Come , 2004 .

[66]  A. Strauss,et al.  The discovery of grounded theory: strategies for qualitative research aldine de gruyter , 1968 .

[67]  Philippe Kruchten,et al.  Reconciling perspectives: A grounded theory of how people manage the process of software development , 2012, J. Syst. Softw..

[68]  M. Crowe,et al.  The power of the word: some post-structural considerations of qualitative approaches in nursing research. , 1998, Journal of advanced nursing.

[69]  Asher Shkedi,et al.  Multiple Case Narrative: A qualitative approach to studying multiple populations , 2005 .

[70]  Jonathan A. Smith.  Evaluating the contribution of interpretative phenomenological analysis , 2011 .

[71]  Carolyn B. Seaman,et al.  Qualitative Methods in Empirical Studies of Software Engineering , 1999, IEEE Trans. Software Eng..

[72]  Deborah K. Padgett,et al.  Qualitative Methods in Social Work Research , 2008 .

[73]  P. Ricoeur,et al.  Time and Narrative , 1989 .

[74]  A. Tsui,et al.  Teachers of English to Speakers of Other Languages , Inc . ( TESOL ) Complexities of Identity Formation : A Narrative Inquiry of an EFL Teacher , 2012 .

[75]  C. Pope,et al.  Quality in qualitative health research , 2007 .

[76]  Brian Fitzgerald,et al.  The ABC of Software Engineering Research , 2018, ACM Trans. Softw. Eng. Methodol..

[77]  James Paul Gee,et al.  话语分析入门 : 理论与方法 = An introduction to discourse analysis : theory and method , 1999 .

[78]  Susan L. Morrow.  Quality and trustworthiness in qualitative research in counseling psychology. , 2005 .

[79]  Paul Ramcharan,et al.  Leveling the Playing Field? Exploring the Merits of the Ethics-as-Process Approach for Judging Qualitative Research Proposals , 2002, Qualitative health research.

[80]  Jonathan A. Smith.  Qualitative Psychology: A Practical Guide to Research Methods , 2006, QMiP Bulletin.

[81]  L. Davidson,et al.  Understanding and Evaluating Qualitative Research∗ , 2002, The Australian and New Zealand journal of psychiatry.

[82]  J. Popay,et al.  Rationale and Standards for the Systematic Review of Qualitative Literature in Health Services Research , 1998, Qualitative health research.

[83]  V. Braun,et al.  Using thematic analysis in psychology , 2006 .

[84]  Sally V Hunter,et al.  Analysing and representing narrative data: The long and winding road , 2010 .

[85]  Michael J. Gill,et al.  The Possibilities of Phenomenology for Organizational Research , 2014 .

[86]  Robin Kearns,et al.  Narrative analysis as a strategy for understanding interview talk in geographic research , 2005 .

[87]  Ian Parker,et al.  Critical Discursive Psychology , 2002 .

[88]  C. Stuhlmiller,et al.  Narrative methods in qualitative research: potential for therapeutic transformation , 2000 .

[89]  P. Sainsbury,et al.  Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. , 2007, International journal for quality in health care : journal of the International Society for Quality in Health Care.

[90]  Jonathan A. Smith,et al.  A practical guide to using Interpretative Phenomenological Analysis in qualitative research psychology , 2014 .

[91]  John P. Hewitt.  The Social Construction of Self-Esteem , 2016 .

[92]  M. Patton,et al.  Qualitative evaluation and research methods , 1992 .

[93]  D. Polkinghorne.  Validity Issues in Narrative Research , 2007 .

[94]  C. Willig,et al.  Reflections on the Use of a Phenomenological Method , 2007 .

[95]  K. Weick.  FROM SENSEMAKING IN ORGANIZATIONS , 2021, The New Economic Sociology.

[96]  C. Seale.  Quality in Qualitative Research , 1999 .