Are the Perspectives Really Different? – Further Experimentation on Scenario-Based Reading of Requirements

Perspective-BasedReading (PBR) is a scenario-based inspection technique whereseveral reviewers read a document from different perspectives(e.g. user, designer, tester). The reading is made accordingto a special scenario, specific for each perspective. The basicassumption behind PBR is that the perspectives find differentdefects and a combination of several perspectives detects moredefects compared to the same amount of reading with a singleperspective. This paper presents a study which analyses the differencesin perspectives. The study is a partial replication of previousstudies. It is conducted in an academic environment using graduatestudents as subjects. Each perspective applies a specific modellingtechnique: use case modelling for the user perspective, equivalencepartitioning for the tester perspective and structured analysisfor the design perspective. A total of 30 subjects were dividedinto 3 groups, giving 10 subjects per perspective. The analysisresults show that (1) there is no significant difference amongthe three perspectives in terms of defect detection rate andnumber of defects found per hour, (2) there is no significantdifference in the defect coverage of the three perspectives,and (3) a simulation study shows that 30 subjects is enough todetect relatively small perspective differences with the chosenstatistical test. The results suggest that a combination of multipleperspectives may not give higher coverage of the defects comparedto single-perspective reading, but further studies are neededto increase the understanding of perspective difference.

[1]  Earl R. Babbie,et al.  Survey Research Methods , 1984 .

[2]  Claes Wohlin,et al.  An Experimental Comparison of Usage-Based and Checklist-Based Reading , 2003, IEEE Trans. Software Eng..

[3]  L. A. Marascuilo,et al.  Statistical Methods for the Social and Behavioral Sciences. , 1989 .

[4]  John J. Marciniak,et al.  Encyclopedia of Software Engineering , 1994, Encyclopedia of Software Engineering.

[5]  Colin Atkinson,et al.  An experimental comparison of reading techniques for defect detection in UML design documents , 2000, J. Syst. Softw..

[6]  B. Bergman,et al.  Quality from Customer Needs to Customer Satisfaction , 1994 .

[7]  Carolyn B. Seaman,et al.  Qualitative Methods in Empirical Studies of Software Engineering , 1999, IEEE Trans. Software Eng..

[8]  Mark C. Paulk,et al.  Capability Maturity Model for Software , 2001 .

[9]  S. Siegel,et al.  Nonparametric Statistics for the Behavioral Sciences , 2022, The SAGE Encyclopedia of Research Design.

[10]  Claes Wohlin,et al.  An Experimental Evaluation of an Experience-Based Capture-Recapture Method in Software Code Inspections , 1998, Empirical Software Engineering.

[11]  Claes Wohlin,et al.  Towards integration of use case modelling and usage-based testing , 2000, J. Syst. Softw..

[12]  R. Yin Case Study Research: Design and Methods , 1984 .

[13]  Watts S. Humphrey,et al.  Introduction to the Personal Software Process , 1996 .

[14]  Michael E. Fagan Design and Code Inspections to Reduce Errors in Program Development , 1976, IBM Syst. J..

[15]  Robert L. Glass,et al.  Science and substance: a challenge to software engineers , 1994, IEEE Software.

[16]  Shari Lawrence Pfleeger,et al.  Software Metrics : A Rigorous and Practical Approach , 1998 .

[17]  Will Hayes,et al.  Research synthesis in software engineering: a case for meta-analysis , 1999, Proceedings Sixth International Software Metrics Symposium (Cat. No.PR00403).

[18]  Victor R. Basili,et al.  Comparing the Effectiveness of Software Testing Strategies , 1987, IEEE Transactions on Software Engineering.

[19]  Robert B. Grady,et al.  Software Metrics: Establishing a Company-Wide Program , 1987 .

[20]  Sam Kash Kachigan Multivariate statistical analysis: A conceptual introduction , 1982 .

[21]  Victor R. Basili,et al.  Evolving and packaging reading technologies , 1997, J. Syst. Softw..

[22]  Forrest Shull,et al.  Building Knowledge through Families of Experiments , 1999, IEEE Trans. Software Eng..

[23]  David E. Avison,et al.  Controlling action research projects , 2001, Inf. Technol. People.

[24]  Jeffrey C. Carver,et al.  Replicating software engineering experiments: addressing the tacit knowledge problem , 2002, Proceedings International Symposium on Empirical Software Engineering.

[25]  Martin Shepperd,et al.  Foundations of software measurement , 1995 .

[26]  K. Eisenhardt Building theories from case study research , 1989, STUDI ORGANIZZATIVI.

[27]  Per Runeson,et al.  Combining Scenario-based Requirements with Static Verification and Dynamic Testing , 1998, REFSQ.

[28]  Victor R. Basili,et al.  The Experimental Paradigm in Software Engineering , 1992, Experimental Software Engineering Issues.

[29]  Marvin V. Zelkowitz,et al.  Experimental Models for Validating Technology , 1998, Computer.

[30]  Victor R. Basili,et al.  Software process evolution at the SEL , 1994, IEEE Software.

[31]  L. Toothaker Book Review : Nonparametric Statistics for the Behavioral Sciences (Second Edition) , 1989 .

[32]  Barbara A. Kitchenham,et al.  Combining empirical results in software engineering , 1998, Inf. Softw. Technol..

[33]  Tracy Hall,et al.  Ethical Issues in Software Engineering Research: A Survey of Current Practice , 2001, Empirical Software Engineering.

[34]  Victor R. Basili,et al.  Experimentation in software engineering , 1986, IEEE Transactions on Software Engineering.

[35]  B. Kitchenham,et al.  Case Studies for Method and Tool Evaluation , 1995, IEEE Softw..

[36]  Forrest Shull,et al.  Developing techniques for using software documents: a series of empirical studies , 1998 .

[37]  Sivert Sørumgård,et al.  Verification of Process Conformance in Empirical Studies of Software Development , 1997 .

[38]  Robert L. Glass,et al.  The software-research crisis , 1994, IEEE Software.

[39]  Ben Shneiderman,et al.  Experimental investigations of the utility of detailed flowcharts in programming , 1977, CACM.

[40]  Adam A. Porter,et al.  Comparing Detection Methods for Software Requirements Inspections: A Replicated Experiment , 1995, IEEE Trans. Software Eng..

[41]  Colin Potts,et al.  Software-engineering research revisited , 1993, IEEE Software.

[42]  Izak Benbasat,et al.  The Case Research Strategy in Studies of Information Systems , 1987, MIS Q..

[43]  Victor R. Basili,et al.  A Methodology for Collecting Valid Software Engineering Data , 1984, IEEE Transactions on Software Engineering.

[44]  Claes Wohlin,et al.  An evaluation of methods for prioritizing software requirements , 1998, Inf. Softw. Technol..

[45]  James Miller,et al.  Applying meta-analytical procedures to software engineering experiments , 2000, J. Syst. Softw..

[46]  Philip M. Johnson,et al.  Does Every Inspection Really Need a Meeting? , 1998, Empirical Software Engineering.

[47]  W. Edwards Deming,et al.  Out of the Crisis , 1982 .

[48]  Shari Lawrence Pfleeger,et al.  Experimental design and analysis in software engineering, part 4: choosing an experimental design , 1995, SOEN.

[49]  Claes Wohlin,et al.  Using Students as Subjects—A Comparative Study of Students and Professionals in Lead-Time Impact Assessment , 2000, Empirical Software Engineering.

[50]  Colin Robson,et al.  Real World Research: A Resource for Social Scientists and Practitioner-Researchers , 1993 .

[51]  Andreas Zendler,et al.  A Preliminary Software Engineering Theory as Investigated by Published Experiments , 2001, Empirical Software Engineering.

[52]  R. Larsson Case Survey Methodology: Quantitative Analysis of Patterns Across Case Studies , 1993 .

[53]  D. Campbell,et al.  EXPERIMENTAL AND QUASI-EXPERIMENT Al DESIGNS FOR RESEARCH , 2012 .

[54]  Lionel C. Briand,et al.  Practical guidelines for measurement-based process improvement , 1996, Softw. Process. Improv. Pract..

[55]  Henry C. Lucas,et al.  A Structured Programming Experiment , 1976, Comput. J..

[56]  N. Dalkey,et al.  An Experimental Application of the Delphi Method to the Use of Experts , 1963 .

[57]  M Joseph,et al.  Software metrics: Establishing a company-wide program: by R B Grady and D L Caswell. Published by Prentice-Hall. 288pp. £37.80 , 1988 .

[58]  T. Cook,et al.  Quasi-experimentation: Design & analysis issues for field settings , 1979 .

[59]  Claes Wohlin,et al.  Experimentation in software engineering: an introduction , 2000 .

[60]  Anneliese Amschler Andrews,et al.  Ethical Issues in Empirical Software Engineering: The Limits of Policy , 2001, Empirical Software Engineering.

[61]  Adam A. Porter,et al.  Comparing Detection Methods For Software Requirements Inspections: A Replication Using Professional Subjects , 1998, Empirical Software Engineering.

[62]  Christopher M. Lott,et al.  Repeatable software engineering experiments for comparing defect-detection techniques , 2004, Empirical Software Engineering.

[63]  A. Strauss,et al.  Basics of Qualitative Research , 1992 .

[64]  James Miller,et al.  Further Experiences with Scenarios and Checklists , 1998, Empirical Software Engineering.

[65]  Janice Singer,et al.  Ethical Issues in Empirical Studies of Software Engineering , 2002, IEEE Trans. Software Eng..

[66]  Joan E. Sieber,et al.  Protecting Research Subjects, Employees and Researchers: Implications for Software Engineering , 2001, Empirical Software Engineering.

[67]  Ola Blomkvist,et al.  An Extended Replication of an Experiment for Assessing Methods for Software Requirements Inspections , 1998, Empirical Software Engineering.

[68]  Forrest Shull,et al.  The empirical investigation of Perspective-Based Reading , 1995, Empirical Software Engineering.

[69]  David C. Hoaglin,et al.  Some Implementations of the Boxplot , 1989 .

[70]  Adam A. Porter,et al.  An experiment to assess different defect detection methods for software requirements inspections , 1994, Proceedings of 16th International Conference on Software Engineering.

[71]  Trevor Wood-Harper,et al.  A critical perspective on action research as a method for information systems research , 1996, J. Inf. Technol..

[72]  Bill Hetzel,et al.  Making Software Measurement Work: Building an Effective Measurement Program , 1993 .

[73]  G. Noblit,et al.  Meta-Ethnography: Synthesizing Qualitative Studies , 1988 .

[74]  Sam Kash Kachigan Statistical Analysis: An Interdisciplinary Introduction to Univariate & Multivariate Methods , 1986 .

[75]  H. Dieter Rombach,et al.  Experimentation as a vehicle for software technology transfer-A family of software reading techniques , 1997, Inf. Softw. Technol..

[76]  Matthias Jarke,et al.  Scenarios in System Development: Current Practice , 1998, IEEE Softw..

[77]  Victor R. Basili,et al.  The TAME Project: Towards Improvement-Oriented Software Environments , 1988, IEEE Trans. Software Eng..

[78]  Claes Wohlin,et al.  A project effort estimation study , 1998, Inf. Softw. Technol..

[79]  Watts S. Humphrey,et al.  A discipline for software engineering , 2012, Series in software engineering.

[80]  Watts S. Humphrey,et al.  Managing the software process , 1989, The SEI series in software engineering.

[81]  H. Sackman,et al.  An Exploratory Investigation of Programmer Performance Under On-Line and Off-Line Conditions , 1967 .

[82]  Sandro Morasca,et al.  On the application of measurement theory in software engineering , 2004, Empirical Software Engineering.

[83]  L. Delbeke Quasi-experimentation - design and analysis issues for field settings - cook,td, campbell,dt , 1980 .

[84]  Shari Lawrence Pfleeger,et al.  Preliminary Guidelines for Empirical Research in Software Engineering , 2002, IEEE Trans. Software Eng..

[85]  Tom DeMarco,et al.  Controlling Software Projects , 1982 .

[86]  Allen S. Lee A Scientific Methodology for MIS Case Studies , 1989, MIS Q..

[87]  B. Manly Multivariate Statistical Methods : A Primer , 1986 .

[88]  Margaret J. Robertson,et al.  Design and Analysis of Experiments , 2006, Handbook of statistics.

[89]  Giuseppe Visaggio,et al.  A Replicated Experiment to Assess Requirements Inspection Techniques , 2004, Empirical Software Engineering.

[90]  Victor R. Basili,et al.  Cleanroom Software Development: An Empirical Evaluation , 1987, IEEE Transactions on Software Engineering.

[91]  Shari Lawrence Pfleeger,et al.  Experimental design and analysis in software engineering , 1995, Ann. Softw. Eng..

[92]  Paul Lukowicz,et al.  Experimental evaluation in computer science: A quantitative study , 1995, J. Syst. Softw..

[93]  Meir M. Lehman Programs, life cycles, and laws of software evolution , 1980 .

[94]  Victor R. Basili,et al.  Software development: a paradigm for the future , 1989, [1989] Proceedings of the Thirteenth Annual International Computer Software & Applications Conference.

[95]  E. P. Doolan,et al.  Experience with Fagan's inspection method , 1992, Softw. Pract. Exp..

[96]  Stephen G. Eick,et al.  Estimating software fault content before coding , 1992, International Conference on Software Engineering.

[97]  Lionel C. Briand,et al.  A Controlled Experiment for Evaluating Quality Guidelines on the Maintainability of Object-Oriented Designs , 2001, IEEE Trans. Software Eng..

[98]  R.C. Linger,et al.  Cleanroom process model , 1994, IEEE Software.

[99]  Natalia Juristo Juzgado,et al.  Basics of Software Engineering Experimentation , 2010, Springer US.

[100]  Jacob Cohen,et al.  Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. , 1968 .

[101]  Lawrence G. Votta,et al.  Does every inspection need a meeting? , 1993, SIGSOFT '93.

[102]  Harvey P. Siy,et al.  An Experiment ot Assess the Cost-Benefits of Code Inspections in Large Scale Software Development , 1997, IEEE Trans. Software Eng..

[103]  Walter F. Tichy,et al.  Should Computer Scientists Experiment More? , 1998, Computer.

[104]  Claes Wohlin,et al.  A Framework for Technology Introduction in Software Organizations 1 , 1996 .

[105]  Glenford J. Myers,et al.  A controlled experiment in program testing and code walkthroughs/inspections , 1978, CACM.

[106]  Norman E. Fenton,et al.  Measurement : A Necessary Scientific Basis , 2004 .

[107]  James Miller Estimating the number of remaining defects after inspection , 1999 .

[108]  R. Stake The art of case study research , 1995 .

[109]  Jean-Pierre Serre,et al.  Fermat ’ s Last Theorem , 2017 .