Stopping Behavior of Systems Analysts During Information Requirements Elicitation

Understanding the cognitive activities of analysts during information requirements determination (IRD) has been recognized as a key indicator of IRD success. The research presented here examines one such cognitive activity: analysts' determination of the sufficiency of information gathered during the elicitation of requirements. Research in behavioral decision-making has identified various heuristics, or stopping rules, that are used to gauge the sufficiency of the information obtained and to terminate information acquisition. Despite the fact that analysts undoubtedly employ such stopping rules in requirements elicitation, no research has studied this phenomenon. In the present research, we present a classification of stopping rules appropriate for information gathering problems. Stopping-rule use was identified for 54 practicing systems analysts participating in a requirements determination problem in a laboratory setting. Results indicated that analyst experience influences the application of specific cognitive stopping rules, and that the use of these stopping rules has an impact on requirements determination outcomes. In addition, the use of certain stopping rules resulted in greater quantity and completeness of requirements elicited from users. Theoretical implications for the elicitation of information and practical implications for the training of systems analysts are discussed.

[1]  Shawn P. Curley,et al.  Belief, knowledge, and uncertainty: A cognitive perspective on subjective probability , 1991 .

[2]  Elizabeth J. Davidson.  Joint application design (JAD) in practice , 1999, J. Syst. Softw..

[3]  Glenn J. Browne,et al.  Evoking Information in Probability Assessment: Knowledge Maps and Reasoning-Based Directed Questions , 1997 .

[4]  B. Fischhoff,et al.  Fault Trees: Sensitivity of Estimated Failure Probabilities to Problem Representation , 2005 .

[5]  Richard Leifer,et al.  Deep structures: Real information requirements determination , 1994, Inf. Manag..

[6]  Glenn J. Browne,et al.  An Empirical Investigation of User Requirements Elicitation: Comparing the Effectiveness of Prompting Techniques , 2001, J. Manag. Inf. Syst..

[7]  D. Norman,et al.  To Ask a Question, One Must Know Enough to Know What Is Not Known. Report No. 7802. , 1978 .

[8]  J. R. Landis,et al.  The measurement of observer agreement for categorical data. , 1977, Biometrics.

[9]  Vinod Goel,et al.  Motivating the Notion of Generic Design within Information-Processing Theory: The Design Problem Space , 1989, AI Mag..

[10]  Joey F. George,et al.  Modern Systems Analysis and Design , 1996 .

[11]  A. Tversky,et al.  Judgment under Uncertainty: Heuristics and Biases , 1974, Science.

[12]  S. Curley.  JUDGMENT-BASED AND REASONING-BASED STOPPING RULES IN DECISION MAKING UNDER UNCERTAINTY , 2002 .

[13]  Brian K. Thorn,et al.  Predecisional information acquisition: Effects of task variables on suboptimal search strategies , 1987 .

[14]  David J. Groggel,et al.  Practical Nonparametric Statistics , 2000, Technometrics.

[15]  Gerald L. Lohse,et al.  Cognitive evaluation of system representation diagrams , 1995, Inf. Manag..

[16]  Iris Vessey,et al.  Learning to Specify Information Requirements: The Relationship between Application and Methodology , 1993, J. Manag. Inf. Syst..

[17]  A. Chao,et al.  Reliability-estimation and stopping-rules for software testing, based on repeated appearances of bugs , 1995 .

[18]  Richard G. Swensson,et al.  Fixed and optional stopping models for two-choice discrimination times , 1974 .

[19]  Glenn J. Browne,et al.  An empirical evaluation of context-independent prompting tools for requirements determination , 1998 .

[20]  Joyce J. Elam,et al.  Semantic Structuring in Analyst Acquisition and Representation of Facts in Requirements Analysis , 1998, Inf. Syst. Res..

[21]  Eric J. Johnson,et al.  Behavioral decision research: A constructive processing perspective. , 1992 .

[22]  James Shanteau,et al.  Reducing the influence of irrelevant information on experienced decision makers , 1984 .

[23]  Gordon F. Pitz,et al.  Strategies of information seeking in deferred decision making , 1969 .

[24]  Russell S. Winer,et al.  Locally Rational Decision Making: The Distracting Effect of Information on Managerial Performance , 1992 .

[25]  Kenji Takahashi,et al.  Inquiry-based requirements analysis , 1994, IEEE Software.

[26]  Itzhak Shemer,et al.  Systems analysis: a systemic analysis of a conceptual model , 1987, CACM.

[27]  J. Edward Russo,et al.  Stopping criteria in sequential choice , 1996 .

[28]  Gordon B. Davis,et al.  Strategies for Information Requirements Determination , 1982, IBM Syst. J..

[29]  Minder Chen,et al.  Using Group Support Systems and Joint Application Development for Requirements Specification , 1993, J. Manag. Inf. Syst..

[30]  Darryl A. Seale,et al.  Sequential Decision Making with Relative Ranks: An Experimental Investigation of the "Secretary Problem"> , 1997 .

[31]  Joey F. George,et al.  Modern Systems Analysis and Design (4th Edition) , 2004 .

[32]  James C. Wetherbe.  Executive Information Requirements: Getting It Right , 1991, MIS Q..

[33]  Margaret Tan,et al.  Establishing Mutual Understanding in Systems Design: An Empirical Study , 1994, J. Manag. Inf. Syst..

[34]  R. Byrne.  Planning meals: Problem-solving on a real data-base , 1977, Cognition.

[35]  Amnon Rapoport,et al.  Choice behavior in an optional stopping task , 1970 .

[36]  J. Baron,et al.  Heuristics and biases in diagnostic reasoning: II. Congruence, information, and certainty☆ , 1988 .

[37]  D. Goldstein,et al.  Betting on one good reason : The Take the Best heuristic , 1999 .

[38]  B. Fischhoff,et al.  Cost benefit analysis and the art of motorcycle maintenance , 1977 .

[39]  Hugh J. Watson,et al.  Determining Information Requirements for an EIS , 1993, MIS Q..

[40]  Izak Benbasat,et al.  Process Tracing Methods in Decision Support Systems Research: Exploring the Black Box , 1987, MIS Q..

[41]  Franz Schmalhofer,et al.  STOCHASTIC CHOICE HEURISTICS , 1984 .

[42]  Glenn J. Browne,et al.  Representing User Requirements:An Empirical Investigation of Formality in Modeling Tools , 1997 .

[43]  A. Tversky,et al.  Thinking through uncertainty: Nonconsequential reasoning and choice , 1992, Cognitive Psychology.

[44]  Robert W. Zmud,et al.  The Use of Mental Imagery to Facilitate Information Identification in Requirements Analysis , 1993, J. Manag. Inf. Syst..

[45]  Roger C. Schank,et al.  Scripts, plans, goals and understanding: an inquiry into human knowledge structures , 1978 .

[46]  Robert P. Bostrom,et al.  Successful application of communication techniques to improve the systems development process , 1989, Inf. Manag..

[47]  John R. Valusek,et al.  Information requirements determination: obstacles within, among and between participants , 1985, SIGCPR '85.

[48]  Thomas R. Hinrichs,et al.  Problem solving in open worlds - a case study in design , 1992 .

[49]  M A Branch,et al.  Software maintenance management , 1986 .

[50]  Russell L. Purvis,et al.  An examination of designer and user perceptions of JAD and the traditional IS design methodology , 1997, Inf. Manag..

[51]  Barry W. Boehm,et al.  Software Engineering Economics , 1993, IEEE Transactions on Software Engineering.

[52]  Suzanne Robertson,et al.  Mastering the Requirements Process , 1999 .

[53]  Ritu Agarwal,et al.  Knowledge Acquisition Using Structured Interviewing: An Empirical Investigation , 1990, J. Manag. Inf. Syst..

[54]  Jeffrey L. Whitten,et al.  Systems Analysis and Design Methods , 1986 .

[55]  H. Meij.  Question asking: To know that you do not know is not enough. , 1990 .

[56]  Allen Newell,et al.  Human Problem Solving. , 1973 .

[57]  Jerome R. Busemeyer,et al.  Psychological models of deferred decision making , 1988 .

[58]  Herbert A. Simon,et al.  The Sciences of the Artificial , 1970 .

[59]  Carl A. Kogut.  Consumer search behavior and sunk costs , 1990 .

[60]  J. Andel.  Sequential Analysis , 2022, The SAGE Encyclopedia of Research Design.

[61]  V. R. Waldron.  Interviewing for knowledge , 1986, IEEE Transactions on Professional Communication.

[62]  Joyce J. Elam,et al.  Inside a software design team: knowledge acquisition, sharing, and integration , 1993, CACM.

[63]  Marinus J. Bouwman.  Human Diagnostic Reasoning by Computer: An Illustration from Financial Analysis , 1983 .

[64]  Gary W. Dickson,et al.  Problem solving for effective systems analysis: an experimental exploration , 1983, CACM.

[65]  Sara Jones,et al.  Formal methods and requirements engineering: Challenges and synergies , 1998, J. Syst. Softw..

[66]  Nicholas P. Vitalari,et al.  Structuring the requirements analysis process for information systems: a proposition viewpoint , 1992 .

[67]  Sonia Gatzanis.  Making sense of statistics in psychologyA second level course , 1997 .

[68]  Nicholas P. Vitalari,et al.  Differences Between Novice and Expert Systems Analysts: What Do We Know and What Do We Do? , 1998, J. Manag. Inf. Syst..

[69]  Enid Mumford,et al.  A Socio-Technical Approach to Systems Design , 2000, Requirements Engineering.

[70]  Robert W. Zmud,et al.  A Synthesis of Research on Requirements Analysis and Knowledge Acquisition Techniques , 1992, MIS Q..

[71]  C. Gettys,et al.  Hypothesis plausibility and hypothesis generation , 1979 .

[72]  W DicksonGary,et al.  Problem solving for effective systems analysis , 1983 .

[73]  Justus D. Naumann,et al.  An experimental comparison of abstract and concrete representations in systems analysis , 1992, Inf. Manag..

[74]  Naveed Gilani,et al.  Information search in judgment tasks: A regression model and some preliminary findings , 1982 .

[75]  Brian Everitt,et al.  Making Sense of Statistics in Psychology: A Second-Level Course , 1996 .

[76]  J. Busemeyer.  Choice behavior in a sequential decision-making task☆☆☆ , 1982 .

[77]  Amnon Rapoport,et al.  Search behavior with and without optional stopping , 1972 .

[78]  Atul Prakash,et al.  Software Engineering: Problems and Perspectives , 1984, Computer.

[79]  Edward J. Garrity,et al.  A Problem-solving Perspective on Systems Development , 1990, J. Manag. Inf. Syst..

[80]  Hal W. Stephenson,et al.  Quality Problem Solving , 2000, Technometrics.

[81]  P. Farquhar,et al.  Decision structuring with phantom alternatives , 1993 .