Ethical issues in empirical studies using student subjects: Re-visiting practices and perceptions

Context: Using student subjects in empirical studies has been discussed extensively from a methodological perspective in Software Engineering (SE), but there is a lack of similar discussion surrounding ethical aspects of doing so. As students are in a subordinate relationship to their instructors, such a discussion is needed. Objective: We aim to increase the understanding of practices and perceptions SE researchers have of ethical issues with student participation in empirical studies. Method: We conducted a systematic mapping study of 372 empirical SE studies involving students, following up with a survey answered by 100 SE researchers regarding their current practices and opinions regarding student participation. Results: The mapping study shows that the majority of studies does not report conditions regarding recruitment, voluntariness, compensation, and ethics approval. In contrast, the majority of survey participants supports reporting these conditions. The survey further reveals that less than half of the participants require ethics approval. Additionally, the majority of participants recruit their own students on a voluntary basis, and use informed consent with withdrawal options. There is disagreeG. Liebel Reykjavik University Menntavegur 1, 102 Reykjav́ık, Iceland ORCID: 0000-0002-3884-815X E-mail: grischal@ru.is S. Chakraborty Reykjavik University Menntavegur 1, 102 Reykjav́ık, Iceland E-mail: shalini19@ru.is 2 Grischa Liebel, Shalini Chakraborty ment among the participants whether course instructors should be involved in research studies and if should know who participates in a study. Conclusions: It is a positive sign that mandatory participation is rare, and that informed consent and withdrawal options are standard. However, we see immediate need for action, as study conditions are under-reported, and as opinions on ethical practices differ widely. In particular, there is little regard in SE on the power relationship between instructors and students.

[1]  Claes Wohlin,et al.  Using students as subjects - an empirical evaluation , 2008, ESEM '08.

[2]  Paul Voigt,et al.  The Eu General Data Protection Regulation (Gdpr): A Practical Guide , 2017 .

[3]  Kai Petersen,et al.  Systematic Mapping Studies in Software Engineering , 2008, EASE.

[4]  Margaret M. Burnett,et al.  A practical guide to controlled experiments of software engineering tools with human participants , 2013, Empirical Software Engineering.

[5]  Emerson R. Murphy-Hill,et al.  Do Developers Read Compiler Error Messages? , 2017, 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE).

[6]  Janice Singer,et al.  A Practical Guide to Ethical Research Involving Humans , 2008, Guide to Advanced Empirical Software Engineering.

[7]  Magne Jørgensen,et al.  Believing is Seeing: Confirmation Bias Studies in Software Engineering , 2015, 2015 41st Euromicro Conference on Software Engineering and Advanced Applications.

[8]  Paul Voigt,et al.  The EU General Data Protection Regulation (GDPR) , 2017 .

[9]  David Budgen,et al.  Reporting computing projects through structured abstracts: a quasi-experiment , 2011, Empirical Software Engineering.

[10]  Arnon Sturm,et al.  Evaluating the productivity of a reference-based programming approach: A controlled experiment , 2014, Inf. Softw. Technol..

[11]  Shari Lawrence Pfleeger,et al.  Personal Opinion Surveys , 2008, Guide to Advanced Empirical Software Engineering.

[12]  Brian Fitzgerald,et al.  The ABC of Software Engineering Research , 2018, ACM Trans. Softw. Eng. Methodol..

[13]  J. R. Landis,et al.  The measurement of observer agreement for categorical data. , 1977, Biometrics.

[14]  Janice Singer,et al.  Studying Software Engineers: Data Collection Techniques for Software Field Studies , 2005, Empirical Software Engineering.

[15]  Muhammad Ali Babar,et al.  An empirical investigation of the influence of persona with personality traits on conceptual design , 2017, J. Syst. Softw..

[16]  Natalia Juristo Juzgado,et al.  Are Students Representatives of Professionals in Software Engineering Experiments? , 2015, 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering.

[17]  Laurie A. Williams,et al.  Using templates to elicit implied security requirements from functional requirements - a controlled experiment , 2014, ESEM '14.

[18]  Paris Avgeriou,et al.  On integrating student empirical software engineering studies with research and teaching goals , 2012, EASE.

[19]  Keith W. Miller,et al.  Software engineering code of ethics , 1997, CACM.

[20]  Claes Wohlin,et al.  Empirical software engineering experts on the use of students and professionals in experiments , 2017, Empirical Software Engineering.

[21]  Anneliese Amschler Andrews,et al.  Ethical Issues in Empirical Software Engineering: The Limits of Policy , 2001, Empirical Software Engineering.

[22]  Jeffrey C. Carver,et al.  A checklist for integrating student empirical studies with research and teaching goals , 2010, Empirical Software Engineering.

[23]  Westley Weimer,et al.  Decoding the Representation of Code in the Brain: An fMRI Study of Code Review and Expertise , 2017, 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE).

[24]  D. Dittrich,et al.  Computer Science Security Research and Human Subjects: Emerging Considerations for Research Ethics Boards , 2011, Journal of Empirical Research on Human Research Ethics.

[25]  Ewan D. Tempero,et al.  An experiment on the impact of transparency on the effectiveness of requirements documents , 2015, Empirical Software Engineering.

[26]  Pearl Brereton,et al.  Performing systematic literature reviews in software engineering , 2006, ICSE.

[27]  Robert M. Davison,et al.  Research Ethics in Information Systems: Would a Code of Practice Help? , 2001, Commun. Assoc. Inf. Syst..

[28]  Stephan Diehl,et al.  Worse Than Spam: Issues In Sampling Software Developers , 2016, ESEM.

[29]  Shari Lawrence Pfleeger,et al.  Principles of survey research: part 1: turning lemons into lemonade , 2001, SOEN.

[30]  Joan E. Sieber,et al.  Protecting Research Subjects, Employees and Researchers: Implications for Software Engineering , 2001, Empirical Software Engineering.

[31]  Jeffrey C. Carver,et al.  Issues in using students in empirical studies in software engineering education , 2003, Proceedings. 5th International Workshop on Enterprise Networking and Computing in Healthcare Industry (IEEE Cat. No.03EX717).

[32]  Margaret-Anne D. Storey,et al.  Is it Ethical to Evaluate Web-based Learning Tools using Students? , 2004, Empirical Software Engineering.

[33]  John Aycock,et al.  Human Subjects, Agents, or Bots: Current Issues in Ethics and Computer Security Research , 2011, Financial Cryptography Workshops.

[34]  Claes Wohlin,et al.  Using Students as Subjects—A Comparative Study of Students and Professionals in Lead-Time Impact Assessment , 2000, Empirical Software Engineering.

[35]  Peretz Shoval,et al.  Comparing the impact of the OO-DFD and the Use Case methods for modeling functional requirements on comprehension and quality of models: a controlled experiment , 2012, Requirements Engineering.

[36]  Maria Letizia Jaccheri,et al.  Exploring the intersection between software industry and Software Engineering education - A systematic mapping of Software Engineering Trends , 2021, J. Syst. Softw..

[37]  Johnny Saldaña,et al.  The Coding Manual for Qualitative Researchers , 2009 .

[38]  Eirini Kalliamvakou,et al.  Methodology Matters: How We Study Socio-Technical Aspects in Software Engineering , 2019, ArXiv.

[39]  Anne Bowser,et al.  Supporting Ethical Web Research: A New Research Ethics Review , 2015, WWW.

[40]  Per Runeson,et al.  Using Students as Experiment Subjects – An Analysis on Graduate and Freshmen Student Data , 2003 .

[41]  Westley Weimer,et al.  Benefits and barriers of user evaluation in software engineering research , 2011, OOPSLA '11.

[42]  Laurie A. Williams,et al.  Identifying the implied: Findings from three differentiated replications on the use of security requirements templates , 2016, Empirical Software Engineering.

[43]  Luisa Mich,et al.  Group versus individual use of power-only EPMcreate as a creativity enhancement technique for requirements elicitation , 2017, Empirical Software Engineering.

[44]  Marsha Chechik,et al.  Modeling and Reasoning with Changing Intentions: An Experiment , 2017, 2017 IEEE 25th International Requirements Engineering Conference (RE).

[45]  Tracy Hall,et al.  Ethical Issues in Software Engineering Research: A Survey of Current Practice , 2001, Empirical Software Engineering.

[46]  Charles Weijer,et al.  The Ethics and Politics of Human Experimentation , 1995 .

[47]  Janice Singer,et al.  Ethical Issues in Empirical Studies of Software Engineering , 2002, IEEE Trans. Software Eng..