Systematic review and aggregation of empirical studies on elicitation techniques

We have located the results of empirical studies on elicitation techniques and aggregated these results to gather empirically grounded evidence. Our chosen surveying methodology was systematic review, whereas we used an adaptation of comparative analysis for aggregation because meta-analysis techniques could not be applied. The review identified 564 publications from the SCOPUS, IEEEXPLORE, and ACM DL databases, as well as Google. We selected and extracted data from 26 of those publications. The selected publications contain 30 empirical studies. These studies were designed to test 43 elicitation techniques and 50 different response variables. We got 100 separate results from the experiments. The aggregation generated 17 pieces of knowledge about the interviewing, laddering, sorting, and protocol analysis elicitation techniques. We provide a set of guidelines based on the gathered pieces of knowledge.

[1]  A. Griffin,et al.  Voice of the Customer , 2010 .

[2]  J. Higgins,et al.  Cochrane Handbook for Systematic Reviews of Interventions , 2010, International Coaching Psychology Review.

[3]  Natalia Juristo Juzgado,et al.  Developing search strategies for detecting relevant experiments , 2009, Empirical Software Engineering.

[4]  Xiao-Hua Zhou,et al.  Statistical Methods for Meta‐Analysis , 2008 .

[5]  Natalia Juristo Juzgado,et al.  In Search of What We Experimentally Know about Unit Testing , 2006, IEEE Software.

[6]  Natalia Juristo Juzgado,et al.  Effectiveness of Requirements Elicitation Techniques: Empirical Results Derived from a Systematic Review , 2006, 14th IEEE International Requirements Engineering Conference (RE'06).

[7]  Tore Dybå,et al.  A systematic review of statistical power in software engineering experiments , 2006, Inf. Softw. Technol..

[8]  Glenn J. Browne,et al.  Stopping Behavior of Systems Analysts During Information Requirements Elicitation , 2004, J. Manag. Inf. Syst..

[9]  Natalia Juristo Juzgado,et al.  Reviewing 25 Years of Testing Technique Experiments , 2004, Empirical Software Engineering.

[10]  A. Maule,et al.  Causal Cognitive Mapping in the Organizational Strategy Field: A Comparison of Alternative Elicitation Procedures , 2004 .

[11]  Evan W. Duggan,et al.  Higher Quality Requirements: Supporting Joint Application Development with the Nominal Group Technique , 2003, Inf. Technol. Manag..

[12]  Paul G Shekelle,et al.  Efficacy and safety of ephedra and ephedrine for weight loss and athletic performance: a meta-analysis. , 2003, JAMA.

[13]  Einar Breivik,et al.  Elicitation of product attributes in an evaluation context: A comparison of three elicitation techniques , 2003 .

[14]  Mary Beth Rosson,et al.  Effectiveness of elicitation techniques in distributed requirements engineering , 2002, Proceedings IEEE Joint International Conference on Requirements Engineering.

[15]  Ann Blandford,et al.  A case study on integrating contextual information with analytical usability evaluation , 2002, Int. J. Hum. Comput. Stud..

[16]  Soren Lauesen,et al.  Software Requirements: Styles & Techniques , 2002 .

[17]  Malcolm Eva,et al.  Requirements acquisition for rapid applications development , 2001, Inf. Manag..

[18]  Suzanne Robertson,et al.  Requirements trawling: techniques for discovering requirements , 2001, Int. J. Hum. Comput. Stud..

[19]  Sally Thorne,et al.  Meta-Study of Qualitative Health Research: A Practical Guide to Meta-Analysis and Meta-Synthesis , 2001 .

[20]  Glenn J. Browne,et al.  An Empirical Investigation of User Requirements Elicitation: Comparing the Effectiveness of Prompting Techniques , 2001, J. Manag. Inf. Syst..

[21]  Frank M. Shipman,et al.  A comparison of questionnaire-based and GUI-based requirements gathering , 2000, Proceedings ASE 2000. Fifteenth IEEE International Conference on Automated Software Engineering.

[22]  Jennifer E. Fowlkes,et al.  The Utility of Event-Based Knowledge Elicitation , 2000, Hum. Factors.

[23]  Camille Salinesi,et al.  Guiding the process of requirements elicitation through scenario analysis: Results of an empirical study , 1999, Proceedings. Tenth International Workshop on Database and Expert Systems Applications. DEXA 99.

[24]  Niels Asger Nielsen,et al.  A comparison of five elicitation techniques for elicitation of attributes of low involvement products , 1999 .

[25]  I. Sommerville,et al.  Requirements Engineering: Processes and Techniques , 1998 .

[26]  Paul H. Cheney,et al.  A Theoretically Grounded Approach to Assist Memory Recall During Information Requirements Determination , 1998, J. Manag. Inf. Syst..

[27]  Stephen. Jones,et al.  The Use of a Prototype System for Evaluating Knowledge Elicitation Techniques , 1998 .

[28]  A. Sutcliffe,et al.  A technique combination approach to requirements engineering , 1997, Proceedings of ISRE '97: 3rd IEEE International Symposium on Requirements Engineering.

[29]  Neil A. M. Maiden,et al.  ACRE: selecting methods for requirements acquisition , 1996, Softw. Eng. J..

[30]  Eva Hudlicka,et al.  Requirements elicitation with indirect knowledge elicitation techniques: comparison of three methods , 1996, Proceedings of the Second International Conference on Requirements Engineering.

[31]  Nancy J. Cooke,et al.  Toward an on-line knowledge assessment methodology: Building on the relationship between knowing and doing , 1996 .

[32]  Starr Roxanne Hiltz,et al.  The Effects of Distributed Group Support and Process Structuring on Software Requirements Development Teams: Results on Creativity and Quality , 1995, J. Manag. Inf. Syst..

[33]  Gerald F. Smith,et al.  Belief assessment: an underdeveloped phase of probability elicitation , 1995 .

[34]  Alistair G. Sutcliffe,et al.  Requirements rationales: integrating approaches to requirement analysis , 1995, Symposium on Designing Interactive Systems.

[35]  N. Shadbolt,et al.  Eliciting Knowledge from Experts: A Methodological Analysis , 1995 .

[36]  Denise Silver,et al.  Joint Application Development , 1995 .

[37]  A R Jadad,et al.  Assessing the quality of randomized controlled trials: an annotated bibliography of scales and checklists. , 1995, Controlled clinical trials.

[38]  Nancy J. Cooke,et al.  Varieties of knowledge elicitation techniques , 1994, Int. J. Hum. Comput. Stud..

[39]  Nigel Shadbolt,et al.  Laddering: technique and tool use in knowledge acquisition , 1994 .

[40]  Clyde W. Holsapple,et al.  An exploratory study of two KA methods , 1994 .

[41]  Margaret Tan,et al.  Establishing Mutual Understanding in Systems Design: An Empirical Study , 1994, J. Manag. Inf. Syst..

[42]  Richard T. Gill,et al.  Conceptual Graph Analysis: Knowledge Acquisition for Instructional System Design , 1993 .

[43]  Robert W. Zmud,et al.  The Use of Mental Imagery to Facilitate Information Identification in Requirements Analysis , 1993, J. Manag. Inf. Syst..

[44]  Joseph A. Goguen,et al.  Techniques for requirements elicitation , 1993, [1993] Proceedings of the IEEE International Symposium on Requirements Engineering.

[45]  John H. Boose,et al.  A survey of knowledge acquisition techniques and tools , 1993, Knowl. Acquis..

[46]  Nancy J. Cooke,et al.  Eliciting Semantic Relations for Empirically Derived Networks , 1992, Int. J. Man Mach. Stud..

[47]  Nigel Shadbolt,et al.  A comparison of sorting techniques in knowledge acquisition , 1992 .

[48]  Robert W. Zmud,et al.  The use of mental imagery as a requirements analysis technique , 1992 .

[49]  Robert W. Zmud,et al.  A Synthesis of Research on Requirements Analysis and Knowledge Acquisition Techniques , 1992, MIS Q..

[50]  Anne P. Massey,et al.  Focus Groups as a Knowledge Elicitation Technique: An Exploratory Survey , 1991, IEEE Trans. Knowl. Data Eng..

[51]  Nigel Shadbolt,et al.  The efficacy of knowledge elicitation techniques: a comparison across domains and levels of expertise , 1990 .

[52]  Ritu Agarwal,et al.  Knowledge Acquisition Using Structured Interviewing: An Empirical Investigation , 1990, J. Manag. Inf. Syst..

[53]  John G. Gammack,et al.  Constructive interaction in knowledge engineering , 1990 .

[54]  V. Goodrich,et al.  An experimental evaluation of task and methodology variables for requirements definition phase success , 1990, Twenty-Third Annual Hawaii International Conference on System Sciences.

[55]  Baruch Fischhoff,et al.  Eliciting knowledge for analytical representation , 1989, IEEE Trans. Syst. Man Cybern..

[56]  Leonard Adelman,et al.  Measurement issues in knowledge engineering , 1989, IEEE Trans. Syst. Man Cybern..

[57]  Nigel Shadbolt,et al.  The empirical study of knowledge elicitation techniques , 1989, SGAR.

[58]  Beth Crandall,et al.  A comparative study of think-aloud and critical decision knowledge elicitation methods , 1989, SGAR.

[59]  Charles C. Ragin.  The Comparative Method: Moving Beyond Qualitative and Quantitative Strategies , 1989 .

[60]  Nigel Shadbolt,et al.  A formal evaluation of knowledge elicitation techniques for expert systems: domain 1 , 1988 .

[61]  Mo Adam Mahmood,et al.  System Development Methods - A Comparative Investigation , 1987, MIS Q..

[62]  Bob J. Wielinga,et al.  Knowledge Acquisition for Expert Systems , 1987, Advanced Topics in Artificial Intelligence.

[63]  Robert R. Hoffman,et al.  The Problem of Extracting the Knowledge of Experts from the Perspective of Experimental Psychology , 1987, AI Mag..

[64]  Marianne LaFrance,et al.  The Knowledge Acquisition Grid: A Method for Training Knowledge Engineers , 1987, Int. J. Man Mach. Stud..

[65]  Michael L. Donnell,et al.  An experimental investigation of the relative effectiveness of two techniques for structuring multiattributed hierarchies , 1986 .

[66]  K. A. Ericsson,et al.  Protocol Analysis: Verbal Reports as Data , 1984 .

[67]  S. Frankel,et al.  Methodological Observations on Applied Behavioral Science , 1983 .

[68]  J N Horton,et al.  Nominal group technique , 1980, Anaesthesia.

[69]  Gordon B. Davis,et al.  Determining Management Information Needs: A Comparison of Methods , 1977, MIS Q..

[70]  R. Yin,et al.  Using the Case Survey Method To Analyze Policy Studies. , 1975 .

[71]  Donald T. Campbell,et al.  Reforms as Experiments , 1969 .

[72]  Shari Lawrence Pfleeger,et al.  Soup or Art? The Role of Evidential Force in Empirical Software Engineering , 2005, IEEE Softw..

[73]  Mary Dixon-Woods,et al.  Synthesising qualitative and quantitative evidence: a review of possible methods. , 2005, Journal of health services research & policy.

[74]  Barbara Kitchenham,et al.  Procedures for Performing Systematic Reviews , 2004 .

[75]  Nigel Shadbolt,et al.  Comparing knowledge elicitation techniques: a case study , 2004, Artificial Intelligence Review.

[76]  L. Freeman.  The effects of concept maps on requirements elicitation and system models during information systems development , 2004 .

[77]  Alan M. Davis,et al.  A Tale of Two Ontologies: The Basis for Systems Analysis Technique Selection , 2003, AMCIS.

[78]  S. Lauesen.  Software Requirements Styles and Techniques , 2001 .

[79]  W. Shadish,et al.  Experimental and Quasi-Experimental Designs for Generalized Causal Inference , 2001 .

[80]  Sally Thorne,et al.  Meta-Study of Qualitative Health Research , 2001 .

[81]  Joyce J. Elam,et al.  Semantic Structuring in Analyst Acquisition and Representation of Facts in Requirements Analysis , 1998, Inf. Syst. Res..

[82]  Prodromos D. Chatzoglou,et al.  Factors affecting completion of the requirements capture stage of projects with different characteristics , 1997, Inf. Softw. Technol..

[83]  Glenn J. Browne,et al.  Evoking Information in Probability Assessment: Knowledge Maps and Reasoning-Based Directed Questions , 1997 .

[84]  H. Hricak,et al.  Evidence-based medicine. , 1997, Singapore medical journal.

[85]  Moody W. Janette.  Enhancing knowledge elicitation using the cognitive interview , 1996 .

[86]  F. Martin,et al.  Deriving formal parameters for comparing knowledge elicitation techniques based on mathematical functions , 1996 .

[87]  Justus D. Naumann,et al.  An experimental comparison of abstract and concrete representations in systems analysis , 1992, Inf. Manag..

[88]  B. Richards.  Pazzani and C. Brunk. Detecting and Correcting Errors in Rule-based Expert Systems: an Integration of Empirical and Explanation-based Learning. in Proceedings of the 5th Knowledge Acquisition for Knowledge-based Systems Workshop , 1992 .

[89]  Jonathan Alan Silver,et al.  Understanding customer needs--a systematic approach to the "voice of the customer" , 1991 .

[90]  G. Noblit,et al.  Meta-Ethnography: Synthesizing Qualitative Studies , 1988 .

[91]  Leslie Johnson,et al.  Knowledge Elicitation Involving Teachback Interviewing , 1987 .

[92]  Lawrence M. Fagan,et al.  Use of a Domain Model to Drive an Interactive Knowledge-Editing Tool , 1987, Int. J. Man Mach. Stud..

[93]  A. L. Kidd.  Knowledge acquisition for expert systems: a practical handbook , 1987 .

[94]  Jerry C. Olson,et al.  Toward Conceptualizing and Measuring Cognitive Structures , 1981 .