Using argumentation theory to analyse software practitioners' defeasible evidence, inference and belief

ContextSoftware practitioners are often the primary source of information for software engineering research. They naturally produce information about their experiences of software practice, and the beliefs they infer from their experiences. Researchers must evaluate the quality and quantity of this information for their research. ObjectiveTo examine how concepts and methods from argumentation research can be used to study practitioners evidence, inference and beliefs so as to better understand and improve software practice. MethodWe develop a preliminary framework and preliminary methodology, and use those to identify, extract and structure practitioners evidence, inference and beliefs. We illustrate the application of the framework and methodology with examples from a practitioner's blog post. ResultThe practitioner uses (factual) stories, analogies, examples and popular opinion as evidence, and uses that evidence in defeasible reasoning to justify his beliefs and to rebut the beliefs of other practitioners. ConclusionThe framework, methodology and examples could provide a foundation for software engineering researchers to develop a more sophisticated understanding of, and appreciation for, practitioners defeasible evidence, inference and belief. Further work needs to automate (parts of) the methodology to support larger-scale application of the methodology.

[1]  Will Hayes,et al.  Research synthesis in software engineering: a case for meta-analysis , 1999, Proceedings Sixth International Software Metrics Symposium (Cat. No.PR00403).

[2]  William Twining,et al.  Rethinking Evidence: Exploratory Essays , 1990 .

[3]  Sung-Hyon Myaeng,et al.  Detecting Experiences from Weblogs , 2010, ACL.

[4]  Magne Jørgensen,et al.  The Role of Deliberate Artificial Design Elements in Software Engineering Experiments , 2008, IEEE Transactions on Software Engineering.

[5]  Marie-Francine Moens,et al.  Argumentation mining , 2011, Artificial Intelligence and Law.

[6]  Daniela Cruzes,et al.  Applying theory of reasoned action in the context of software development practices: insights into team intention and behavior , 2013, EASE '13.

[7]  Pearl Brereton,et al.  A Study of Computing Undergraduates Undertaking a Systematic Literature Review , 2011, IEEE Transactions on Education.

[8]  Andrew D. Brown,et al.  Doomed to Failure: Narratives of Inevitability and Conspiracy in a Failed IS Project , 1998 .

[9]  Michiel van Genuchten,et al.  Why is Software Late? An Empirical Study of Reasons For Delay in Software Development , 1991, IEEE Trans. Software Eng..

[10]  Pearl Brereton,et al.  Systematic literature reviews in software engineering - A systematic literature review , 2009, Inf. Softw. Technol..

[11]  Shirley Gregor,et al.  The Nature of Theory in Information Systems , 2006, MIS Q..

[12]  Helen Sharp,et al.  An Ethnographic Study of XP Practice , 2004, Empirical Software Engineering.

[13]  Stephen N. Thomas,et al.  Practical Reasoning in Natural Language , 1981 .

[14]  James D. Herbsleb,et al.  Global Software Engineering: The Future of Socio-technical Coordination , 2007, Future of Software Engineering (FOSE '07).

[15]  Janice Singer,et al.  Learning from project history: a case study for software development , 2004, CSCW.

[16]  Philippe Kruchten,et al.  Generating a useful theory of software engineering , 2013, 2013 2nd SEMAT Workshop on a General Theory of Software Engineering (GTSE).

[17]  David S. Rosenblum,et al.  A study in software process data capture and analysis , 1993, [1993] Proceedings of the Second International Conference on the Software Process-Continuous Software Process Improvement.

[18]  S. Toulmin The uses of argument , 1960 .

[19]  Tore Dybå,et al.  Evidence-Based Software Engineering for Practitioners , 2005, IEEE Softw..

[20]  Philippe Kruchten,et al.  Using grounded theory to study the experience of software development , 2011, Empirical Software Engineering.

[21]  Björn Regnell,et al.  Evidence-Based Timelines for Agile Project Retrospectives - A Method Proposal , 2012, XP.

[22]  Austen Rainer,et al.  Identifying Practitioners' Arguments and Evidence in Blogs: Insights from a Pilot Study , 2016, 2016 23rd Asia-Pacific Software Engineering Conference (APSEC).

[23]  Austen Rainer,et al.  The longitudinal, chronological case study research strategy: A definition, and an example from IBM Hursley Park , 2011, Inf. Softw. Technol..

[24]  Trevor J. M. Bench-Capon,et al.  Arguing with Stories , 2017 .

[25]  David A. Schum,et al.  Analysis of Evidence: Frontmatter , 2005 .

[26]  Norman E. Fenton,et al.  Quantitative Analysis of Faults and Failures in a Complex Software System , 2000, IEEE Trans. Software Eng..

[27]  Tore Dybå,et al.  Building Theories in Software Engineering , 2008, Guide to Advanced Empirical Software Engineering.

[28]  Austen Rainer,et al.  A follow-up empirical evaluation of evidence based software engineering by undergraduate students , 2008, EASE.

[29]  Paul Ricoeur,et al.  Paul Ricoeur: Hermeneutics and the Human Sciences , 1983 .

[30]  Helen Sharp,et al.  Ethnographically-informed empirical studies of software practice , 2007, Inf. Softw. Technol..

[31]  Robert Feldt,et al.  Behavioral software engineering: A definition and systematic literature review , 2015, J. Syst. Softw..

[32]  Ivar Jacobson,et al.  Where's the Theory for Software Engineering? , 2012, IEEE Software.

[33]  Manfred Stede,et al.  From Argument Diagrams to Argumentation Mining in Texts: A Survey , 2013, Int. J. Cogn. Informatics Nat. Intell..

[34]  P. Ricoeur,et al.  Hermeneutics and the Human Sciences: Essays on Language, Action and Interpretation , 2016 .

[35]  Austen Rainer,et al.  A preliminary empirical investigation of the use of evidence based software engineering by under-graduate students , 2006 .

[36]  Pearl Brereton,et al.  The Case for Knowledge Translation , 2013, 2013 ACM / IEEE International Symposium on Empirical Software Engineering and Measurement.

[37]  Claes Wohlin An Evidence Profile for Software Engineering Research and Practice , 2013, Perspectives on the Future of Software Engineering.

[38]  Tore Dybå,et al.  Teaching evidence-based software engineering to university students , 2005, 11th IEEE International Software Metrics Symposium (METRICS'05).

[39]  Magne Jørgensen,et al.  Generalization and theory-building in software engineering research , 2004, ICSE 2004.

[40]  Daniela Cruzes,et al.  Analyzing the Impact of Beliefs in Software Project Practices , 2011, 2011 International Symposium on Empirical Software Engineering and Measurement.

[41]  Lutz Prechelt,et al.  An Empirical Comparison of Seven Programming Languages , 2000, Computer.

[42]  Lutz Prechelt,et al.  Plat_Forms: A Web Development Platform Comparison by an Exploratory Experiment Searching for Emergent Platform Properties , 2011, IEEE Transactions on Software Engineering.

[43]  Winifred Menezes,et al.  Marketing Technology to Software Practitioners , 2000, IEEE Softw..

[44]  Alexander L. Wolf,et al.  Discovering models of software processes from event-based data , 1998, TSEM.

[45]  Björn Regnell,et al.  Evidence-based timelines for project retrospectives — A method for assessing requirements engineering in context , 2012, 2012 Second IEEE International Workshop on Empirical Requirements Engineering (EmpiRE).

[46]  Austen Rainer,et al.  Persuading developers to "buy into" software process improvement: a local opinion and empirical evidence , 2003, 2003 International Symposium on Empirical Software Engineering, 2003. ISESE 2003. Proceedings..

[47]  Shari Lawrence Pfleeger,et al.  Understanding and improving technology transfer in software engineering , 1999, J. Syst. Softw..

[48]  William Hughes Critical Thinking: An Introduction to the Basic Skills , 1992 .

[49]  Carolyn B. Seaman,et al.  Qualitative Methods in Empirical Studies of Software Engineering , 1999, IEEE Trans. Software Eng..

[50]  Douglas Walton,et al.  Argument Evaluation and Evidence , 2015 .

[51]  Alec Fisher,et al.  The logic of real arguments: Philosophical assumptions , 2004 .

[52]  Barbara A. Kitchenham,et al.  Combining empirical results in software engineering , 1998, Inf. Softw. Technol..

[53]  Linda S. Lotto Qualitative Data Analysis: A Sourcebook of New Methods , 1986 .

[54]  Kentaro Inui,et al.  Experience Mining: Building a Large-Scale Database of Personal Experiences and Opinions from Web Documents , 2008, 2008 IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology.

[55]  Colin Potts,et al.  Collaborative conceptual design: A large software project case study , 1996, Computer Supported Cooperative Work (CSCW).

[56]  Torgeir Dingsøyr,et al.  Postmortem: Never Leave a Project without It , 2002, IEEE Softw..

[57]  Mehrdad Sabetzadeh,et al.  Classification, Structuring, and Assessment of Evidence for Safety -- A Systematic Literature Review , 2013, 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation.

[58]  D. Schum The Evidential Foundations of Probabilistic Reasoning , 1994 .

[59]  Janice Singer,et al.  Hipikat: a project memory for software development , 2005, IEEE Transactions on Software Engineering.

[60]  R. Yin Case Study Research: Design and Methods , 1984 .

[61]  Mehrdad Sabetzadeh,et al.  An extended systematic literature review on provision of evidence for safety certification , 2014, Inf. Softw. Technol..

[62]  Bill Curtis,et al.  A field study of the software design process for large systems , 1988, CACM.

[63]  Premkumar T. Devanbu,et al.  Belief & Evidence in Empirical Software Engineering , 2016, 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE).

[64]  Valentin Jijkoun,et al.  Mining User Experiences from Online Forums: An Exploration , 2010, HLT-NAACL 2010.

[65]  David A. Schum A science of evidence: contributions from law and probability , 2014, Legal Theory and the Natural Sciences Volume VI.

[66]  Austen Rainer,et al.  Case Study Research in Software Engineering - Guidelines and Examples , 2012 .

[67]  Bart Verheij Legal decision making as dialectical theory construction with argumentation schemes , 2001, ICAIL '01.