Dialog-based protocol: an empirical research method for cognitive activities in software engineering

This paper proposes dialog-based protocol for the study of the cognitive activities during software development and evolution. The dialog-based protocol, derived from the idea of pair programming, is a significant alternative to the common think-aloud protocol, because it lessens the Hawthorne and placebo effects. Using screen-capturing and voice recording instead of videotaping further reduces the Hawthorne effect. The self-directed learning theory provides an encoding scheme and can be used in analyzing the data. A case study illustrates this new approach.

[1]  John S. Gero,et al.  An approach to the analysis of design protocols , 1998 .

[2]  Jere Confrey,et al.  Constructivism and education: Voice and perspective: hearing epistemological innovation in students' words , 1998 .

[3]  A. Adam.  Whatever happened to information systems ethics? Caught between the devil and the deep blue sea , 2004 .

[4]  Iris Vessey,et al.  Research Report - The Relevance of Application Domain Knowledge: The Case of Computer Program Comprehension , 1995, Inf. Syst. Res..

[5]  Andrian Marcus,et al.  An empirical study of programmer learning during incremental software development , 2005, Fourth IEEE Conference on Cognitive Informatics, 2005. (ICCI 2005)..

[6]  J. G. Adair,et al.  The Hawthorne effect: A reconsideration of the methodological artifact. , 1984 .

[7]  B. Bloom,et al.  Taxonomy of Educational Objectives. Handbook I: Cognitive Domain , 1966 .

[8]  Shari Lawrence Pfleeger,et al.  Preliminary Guidelines for Empirical Research in Software Engineering , 2002, IEEE Trans. Software Eng..

[9]  Robert Rosenthal,et al.  The Effect of Early Data Returns on Data Subsequently Obtained by Outcome-Biased Experimenters , 1963 .

[10]  K. A. Ericsson,et al.  Protocol Analysis: Verbal Reports as Data , 1984 .

[11]  R. Rosenthal.  Experimenter effects in behavioral research , 1968 .

[12]  Victor R. Basili,et al.  The role of experimentation in software engineering: past, current, and future , 1996, Proceedings of IEEE 18th International Conference on Software Engineering.

[13]  J. Meijer,et al.  Analysis of solving problems , 1986 .

[14]  John B. Watson.  IS THINKING MERELY ACTION OF LANGUAGE MECHANISMS1? (V.) , 1920 .

[15]  Ted J. Biggerstaff,et al.  Program understanding and the concept assignment problem , 1994, CACM.

[16]  Françoise Détienne,et al.  Measuring cognitive activities in software engineering , 1998, Proceedings of the 20th International Conference on Software Engineering.

[17]  D. Broadbent,et al.  The role of instruction and verbalization in improving performance on complex search tasks , 1990 .

[18]  J. Piaget.  The construction of reality in the child , 1954 .

[19]  Anneliese Amschler Andrews,et al.  A Coding Scheme to Support Systematic Analysis of Software Comprehension , 1999, IEEE Trans. Software Eng..

[20]  Allen Newell,et al.  Human Problem Solving. , 1973 .

[21]  Shaochun Xu,et al.  Analogy of incremental program development and constructivist learning , 2003, The Second IEEE International Conference on Cognitive Informatics, 2003. Proceedings..

[22]  Shaochun Xu,et al.  Cognitive process during program debugging , 2004, Proceedings of the Third IEEE International Conference on Cognitive Informatics, 2004..

[23]  K. Beck,et al.  Extreme Programming Explained , 2002 .

[24]  S. Ian Robertson,et al.  Problem-solving , 2001, Human Thinking.

[25]  Godfrey H. Thomson.  IS THINKING MERELY THE ACTION OF LANGUAGE MECHANISMS1? (II.) , 1920 .

[26]  Judith S. Olson,et al.  Small Group Design Meetings: An Analysis of Collaboration , 1992, Hum. Comput. Interact..

[27]  Spencer Rugaber,et al.  Dowsing: a tool framework for domain-oriented browsing of software artifacts , 1998, Proceedings 13th IEEE International Conference on Automated Software Engineering (Cat. No.98EX239).

[28]  Nancy Pennington,et al.  Comprehension strategies in programming , 1987 .

[29]  David Lorge Parnas,et al.  The limits of empirical studies of software engineering , 2003, 2003 International Symposium on Empirical Software Engineering, 2003. ISESE 2003. Proceedings..

[30]  W T Fu,et al.  ACT-PRO action protocol analyzer: A tool for analyzing discrete action protocols , 2001, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[31]  John R. Anderson,et al.  Change-episodes in coding: when and how do programmers change their code? , 1987 .

[32]  K. Duncker,et al.  On problem-solving , 1945 .

[33]  Bernd Frohmann.  Discourse Analysis as a Research Method in Library and Information Science. , 1994 .

[34]  P. Lloyd,et al.  Can concurrent verbalization reveal design cognition , 1995 .

[35]  Carl von Ossietzky,et al.  The Role of Experimentation in Software Engineering , 2005 .

[36]  A. Shepherd,et al.  Hierarchical Task Analysis and Training Decisions. , 1985 .

[37]  Raquel Benbunan-Fich,et al.  Using protocol analysis to evaluate the usability of a commercial web site , 2001, Inf. Manag..

[38]  Anneliese Amschler Andrews,et al.  Identification of Dynamic Comprehension Processes During Large Scale Maintenance , 1996, IEEE Trans. Software Eng..