Enhancing Analysts' Mental Models for Improving Requirements Elicitation: A Two-stage Theoretical Framework and Empirical Results

: Research has extensively documented the importance of accurate system requirements in avoiding project delays, cost overruns, and system malfunctions. Requirement elicitation (RE) is a critical step in determining system requirements. While much research on RE has emerged, a deeper understanding of three aspects could help significantly improve RE: 1) insights about the role and impacts of support tools in the RE process, 2) the impact of using support tools in multiple stages of the RE process, and 3) a clear focus on the multiplicity of perspectives in assessing RE outcomes. To understand how using support tools could improve RE, we rely on the theoretical lens of mental models (MM) to develop a dynamic conceptual model and argue that analysts form mental models (MMs) of the system during RE and these MMs impact their outcome performance. We posit that one can enhance analysts’ MMs by using a knowledge-based repository (KBR) of components and services embodying domain knowledge specific to the target application during two key stages of RE, which results in improved RE outcomes. We measured the RE outcomes from user and analyst perspectives. The knowledge-based component repository we used in this research (which we developed in collaboration with a multi-national company) focused on insurance claim processing. The repository served as the support tool in RE in a multi-period lab experiment with multiple teams of analysts. The results supported the conceptualized model and showed the significant impacts of such tools in supporting analysts and their performance outcomes at two stages of RE. This work makes multiple contributions: it offers a theoretical framework for understanding and enhancing the RE process, develops measures for analysts’ mental models and RE performance outcomes, and shows the process by which one can improve analysts’ RE performance through access to a KBR of components at two key stages of the RE process.

[1]  D. Rumelhart.  Schemata: The Building Blocks of Cognition , 2017 .

[2]  David E. Rumelhart,et al.  The Representation of Knowledge in Memory 1 , 2017 .

[3]  Mark Lycett,et al.  The Development of Component-based Information Systems , 2016 .

[4]  Andreas Ritter,et al.  Structural Equations With Latent Variables , 2016 .

[5]  Alexander Maedche,et al.  Designing a Requirement Mining System , 2015, J. Assoc. Inf. Syst..

[6]  Venugopal Balijepally,et al.  Task Mental Model and Software Developers' Performance: An Experimental Investigation , 2015, Commun. Assoc. Inf. Syst..

[7]  Christoph Rosenkranz,et al.  Boundary Interactions and Motors of Change in Requirements Elicitation: A Dynamic Perspective on Knowledge Sharing , 2014, J. Assoc. Inf. Syst..

[8]  Robert D. Macredie,et al.  Effective Communication in Requirements Elicitation: A Comparison of Methodologies , 2002, Requirements Engineering.

[9]  Irit Hadar,et al.  The role of domain knowledge in requirements elicitation via interviews: an exploratory study , 2012, Requirements Engineering.

[10]  Veda C. Storey,et al.  Deriving knowledge representation guidelines by analyzing knowledge engineer behavior , 2012, Decis. Support Syst..

[11]  A Skarlatidou,et al.  What Do Lay People Want to Know About the Disposal of Nuclear Waste? A Mental Model Approach to the Design and Development of an Online Risk Communication , 2012, Risk analysis : an official publication of the Society for Risk Analysis.

[12]  Todd Kulesza,et al.  Tell me more?: the effects of mental model soundness on personalizing an intelligent agent , 2012, CHI.

[13]  David R. Peterson,et al.  Causal Analysis to Enhance Creative Problem-Solving: Performance and Effects on Mental Models , 2012 .

[14]  Fatemeh Zahedi,et al.  A knowledge based component/service repository to enhance analysts' domain knowledge for requirements analysis , 2012, Inf. Manag..

[15]  Lauren N. Harkrider,et al.  Leader Ethical Decision-Making in Organizations: Strategies for Sensemaking , 2012 .

[16]  Andrew H. Van de Ven,et al.  Breakdowns in Implementing Models of Organization Change , 2011 .

[17]  Saonee Sarker,et al.  An Exploration into the Process of Requirements Elicitation: A Grounded Approach , 2010, J. Assoc. Inf. Syst..

[18]  Glenn J. Browne,et al.  Investigating Retrieval-Induced Forgetting During Information Requirements Determination , 2010, J. Assoc. Inf. Syst..

[19]  Gerrit van Bruggen,et al.  How Incorporating Feedback Mechanisms in a DSS Affects DSS Evaluations , 2009, Inf. Syst. Res..

[20]  Ş. Purzer,et al.  Using students' previous experience and prior knowledge to facilitate conceptual change in an introductory materials course , 2009, 2009 39th IEEE Frontiers in Education Conference.

[21]  Ilan Oshri,et al.  Managing Component-Based Development in Global Teams , 2009 .

[22]  Nannette P. Napier,et al.  Combining Perceptions and Prescriptions in Requirements Engineering Process Assessment: An Industrial Case Study , 2009, IEEE Transactions on Software Engineering.

[23]  Sven Overhage,et al.  On Component Identification Approaches - Classification, State of the Art, and Comparison , 2009, CBSE.

[24]  Vanita Yadav,et al.  Flexible Global Software Development (GSD): Antecedents of Success in Requirements Analysis , 2009, J. Glob. Inf. Manag..

[25]  Yair Wand,et al.  Using Cognitive Principles to Guide Classification in Information Systems Modeling , 2008, MIS Q..

[26]  Thompson S. H. Teo,et al.  Knowledge portals in Chinese consulting firms: a task–technology fit perspective , 2008, Eur. J. Inf. Syst..

[27]  Wu He,et al.  The effects of conceptual description and search practice on users' mental models and information seeking in a case-based reasoning retrieval system , 2008, Inf. Process. Manag..

[28]  Tuure Tuunanen,et al.  A Contigency Model for Requirements Development , 2007, J. Assoc. Inf. Syst..

[29]  Ahmed Elfatatry,et al.  Dealing with change: components versus services , 2007, CACM.

[30]  Jeff Z. Pan,et al.  Towards semantically enhanced Web service repositories , 2007, J. Web Semant..

[31]  Marc Roper,et al.  Investigating the viability of mental models held by novice programmers , 2007, SIGCSE.

[32]  Xianggui Qu,et al.  Multivariate Data Analysis , 2007, Technometrics.

[33]  Kung-Kiu Lau,et al.  Software Component Models , 2006, IEEE Transactions on Software Engineering.

[34]  Natalia Juristo Juzgado,et al.  Effectiveness of Requirements Elicitation Techniques: Empirical Results Derived from a Systematic Review , 2006, 14th IEEE International Requirements Engineering Conference (RE'06).

[35]  Peter Meso,et al.  Conceptualizing Systems for Understanding: An Empirical Test of Decomposition Principles in Object-Oriented Analysis , 2006, Inf. Syst. Res..

[36]  Venkataraman Ramesh,et al.  Understanding Conceptual Schemas: Exploring the Role of Application and IS Domain Knowledge , 2006, Inf. Syst. Res..

[37]  Didar Zowghi,et al.  Requirements Elicitation: A Survey of Techniques, Approaches, and Tools , 2005 .

[38]  Shmuel Ellis,et al.  After-event reviews: drawing lessons from successful and failed experience. , 2005, The Journal of applied psychology.

[39]  C. Shea,et al.  Schema Theory: A Critical Appraisal and Reevaluation , 2005, Journal of motor behavior.

[40]  Rayford B. Vaughn,et al.  Software requirement understanding using Pathfinder networks: discovering and evaluating mental models , 2005, J. Syst. Softw..

[41]  Glenn J. Browne,et al.  Stopping rule use during information search in design problems , 2004 .

[42]  Gerrit van Bruggen,et al.  DSS Effectiveness in Marketing Resource Allocation Decisions: Reality vs. Perception , 2004, Inf. Syst. Res..

[43]  Susan Wiedenbeck,et al.  Self-efficacy and mental models in learning to program , 2004, ITiCSE '04.

[44]  Evan W. Duggan,et al.  Supporting the JAD Facilitator with the Nominal Group Technique , 2004, J. Organ. End User Comput..

[45]  Alan M. Davis,et al.  A Unified Model of Requirements Elicitation , 2004, J. Manag. Inf. Syst..

[46]  Glenn J. Browne,et al.  Stopping Behavior of Systems Analysts During Information Requirements Elicitation , 2004, J. Manag. Inf. Syst..

[47]  Mary Hegarty,et al.  Constructing and Revising Mental Models of a Mechanical System: The role of domain knowledge in understanding external visualizations , 2004 .

[48]  Fatemeh Zahedi,et al.  Design, retrieval, and assembly in component-based software development , 2003, CACM.

[49]  Evan W. Duggan,et al.  Higher Quality Requirements: Supporting Joint Application Development with the Nominal Group Technique , 2003, Inf. Technol. Manag..

[50]  Scott B. MacKenzie,et al.  Common method biases in behavioral research: a critical review of the literature and recommended remedies. , 2003, The Journal of applied psychology.

[51]  Kyung-Whan Lee,et al.  Object-Oriented Component Identification Method Using the Affinity Analysis Technique , 2003, OOIS.

[52]  Fatemeh Zahedi,et al.  Knowledge-Based Repository Scheme for Storing and Retrieving Business Components: A Theoretical Design and an Empirical Analysis , 2003, IEEE Trans. Software Eng..

[53]  Detmar W. Straub,et al.  Trust and TAM in Online Shopping: An Integrated Model , 2003, MIS Q..

[54]  Tuure Tuunanen,et al.  A New Perspective on Requirements Elicitation Methods , 2003 .

[55]  Ron Weber,et al.  Research Commentary: Information Systems and Conceptual Modeling - A Research Agenda , 2002, Inf. Syst. Res..

[56]  Ali Arsanjani,et al.  A goal-driven approach to enterprise component identification and specification , 2002, CACM.

[57]  Venkataraman Ramesh,et al.  Improving information requirements determination: a cognitive perspective , 2002, Inf. Manag..

[58]  Hossein Nassaji,et al.  Schema Theory and Knowledge-Based Processes in Second Language Reading Comprehension: A Need for Alternative Perspectives , 2002 .

[59]  Pamela A. Savage-Knepshield.  Mental models : issues in construction, congruency, and cognition , 2002 .

[60]  Douglas Havelka,et al.  Requirements Determination: An Information Systems Specialist Perspective of Process Quality , 2002, Requirements Engineering.

[61]  C Moore,et al.  Mental models change rapidly with implicitly acquired information about the local environment: a two-tone image study. , 2001, Journal of experimental psychology. Human perception and performance.

[62]  Barry W. Boehm,et al.  EasyWinWin: a groupware-supported methodology for requirements negotiation , 2001, ESEC/FSE-9.

[63]  Glenn J. Browne,et al.  An Empirical Investigation of User Requirements Elicitation: Comparing the Effectiveness of Prompting Techniques , 2001, J. Manag. Inf. Syst..

[64]  Viswanath Venkatesh,et al.  Determinants of Perceived Ease of Use: Integrating Control, Intrinsic Motivation, and Emotion into the Technology Acceptance Model , 2000, Inf. Syst. Res..

[65]  Detmar W. Straub,et al.  Structural Equation Modeling and Regression: Guidelines for Research Practice , 2000, Commun. Assoc. Inf. Syst..

[66]  Lorne Olfman,et al.  The Role of Training in Preparing End Users to Learn Related Software , 2000, J. Organ. End User Comput..

[67]  E. Deci,et al.  Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. , 2000, The American psychologist.

[68]  B. Frey,et al.  Motivation, Knowledge Transfer, and Organizational Forms , 1999 .

[69]  Linda Dawson,et al.  The use of object-oriented models in requirements engineering: a field study , 1999, ICIS.

[70]  Regine Freitag,et al.  Making Use of Scenarios for Validating Analysis and Design , 1998, IEEE Trans. Software Eng..

[71]  Paul H. Cheney,et al.  A Theoretically Grounded Approach to Assist Memory Recall During Information Requirements Determination , 1998, J. Manag. Inf. Syst..

[72]  Nicholas P. Vitalari,et al.  Differences Between Novice and Expert Systems Analysts: What Do We Know and What Do We Do? , 1998, J. Manag. Inf. Syst..

[73]  Lorne Olfman,et al.  User Interface Consistency Across End-User Applications: The Effects on Mental Models , 1998, J. Manag. Inf. Syst..

[74]  Patricia J. Guinan,et al.  Enabling Software Development Team Performance During Requirements Definition: A Behavioral Versus Technical Approach , 1998, Inf. Syst. Res..

[75]  Joyce J. Elam,et al.  Semantic Structuring in Analyst Acquisition and Representation of Facts in Requirements Analysis , 1998, Inf. Syst. Res..

[76]  Izak Benbasat,et al.  An Empirical Study of Computer System Learning: Comparison of Co-Discovery and Self-Discovery Methods , 1997, Inf. Syst. Res..

[77]  R. Vallerand.  Toward A Hierarchical Model of Intrinsic and Extrinsic Motivation , 1997 .

[78]  Chris Higgins,et al.  Information Acquisition and Mental Models: An Investigation into the Relationship Between Behaviour and Learning , 1996, Inf. Syst. Res..

[79]  Sharon J. Derry,et al.  Cognitive schema theory in the constructivist debate , 1996 .

[80]  K. Weick.  FROM SENSEMAKING IN ORGANIZATIONS , 2021, The New Economic Sociology.

[81]  Dale Goodhue,et al.  Task-Technology Fit and Individual Performance , 1995, MIS Q..

[82]  N. Shadbolt,et al.  Eliciting Knowledge from Experts: A Methodological Analysis , 1995 .

[83]  Radhika Santhanam,et al.  Improving End-User Proficiency: Effects of Conceptual Training and Nature of Interaction , 1994, Inf. Syst. Res..

[84]  M. Just,et al.  Constructing mental models of machines from text and diagrams. , 1993 .

[85]  Michael G. Christel,et al.  AMORE: The Advanced Multimedia Organizer for Requirements Elicitation , 1993 .

[86]  Nancy Staggers,et al.  Mental Models: Concepts for Human-Computer Interaction Research , 1993, Int. J. Man Mach. Stud..

[87]  Robert W. Zmud,et al.  The Use of Mental Imagery to Facilitate Information Identification in Requirements Analysis , 1993, J. Manag. Inf. Syst..

[88]  M. Browne,et al.  Alternative Ways of Assessing Model Fit , 1992 .

[89]  P. Bentler.  On the fit of models to covariances and methodology to the Bulletin. , 1992, Psychological bulletin.

[90]  Justus D. Naumann,et al.  An experimental comparison of abstract and concrete representations in systems analysis , 1992, Inf. Manag..

[91]  Patricia E. Blosser.  How to ask the right questions , 1991 .

[92]  Richard C. Waters,et al.  The Requirements Apprentice: Automated Assistance for Requirements Acquisition , 1991, IEEE Trans. Software Eng..

[93]  Joan K. Gallini,et al.  When Is an Illustration Worth Ten Thousand Words , 1990 .

[94]  Ritu Agarwal,et al.  Knowledge Acquisition Using Structured Interviewing: An Empirical Investigation , 1990, J. Manag. Inf. Syst..

[95]  C. Stubbart.  MANAGERIAL COGNITION: A MISSING LINK IN STRATEGIC MANAGEMENT RESEARCH , 1989 .

[96]  James P. Walsh,et al.  Selectivity and selective perception: An investigation of managers''belief structures and informatio , 1988 .

[97]  T. M. Rajkumar,et al.  Comparison of analysis techniques for information requirement determination , 1988, CACM.

[98]  E. Deci,et al.  The support of autonomy and the control of behavior. , 1987, Journal of personality and social psychology.

[99]  Neville Moray.  Intelligent Aids, Mental Models, and the Theory of Machines , 1987, Int. J. Man Mach. Stud..

[100]  Yvonne Wærn,et al.  On the dynamics of mental models , 1987, Informatics and Psychology Workshop.

[101]  N. M. Morris,et al.  On Looking into the Black Box: Prospects and Limits in the Search for Mental Models , 1986 .

[102]  Christine L. Borgman,et al.  The user's mental model of an information retrieval system , 1999, SIGIR '85.

[103]  B. Adelson,et al.  The Role of Domain Expenence in Software Design , 1985, IEEE Transactions on Software Engineering.

[104]  David E. Kieras,et al.  The Role of a Mental Model in Learning to Operate a Device , 1990, Cogn. Sci..

[105]  P. Johnson-Laird,et al.  Mental Models: Towards a Cognitive Science of Language, Inference, and Consciousness , 1985 .

[106]  D. Gentner,et al.  Flowing waters or teeming crowds: Mental models of electricity , 1982 .

[107]  M. McCloskey.  Naive Theories of Motion. , 1982 .

[108]  P. Bentler,et al.  Significance Tests and Goodness of Fit in the Analysis of Covariance Structures , 1980 .

[109]  Richard C. Anderson,et al.  Schooling and the Acquisition of Knowledge , 1978 .

[110]  J. Kelso,et al.  Motor schema formation in children. , 1978 .

[111]  Donald A. Norman,et al.  Accretion, tuning and restructuring: Three modes of learning , 1976 .

[112]  R. Schmidt.  A schema theory of discrete motor skill learning. , 1975 .

[113]  Barry M. Staw,et al.  Self-perception of intrinsic and extrinsic motivation. , 1975, Journal of personality and social psychology.

[114]  Marvin Minsky,et al.  A framework for representing knowledge" in the psychology of computer vision , 1975 .

[115]  Marvin Minsky,et al.  A framework for representing knowledge , 1974 .

[116]  R. L. Ebel,et al.  Estimation of the reliability of ratings , 1951 .

[117]  W. H. F. Barnes.  The Nature of Explanation , 1944, Nature.

[118]  F. Bartlett,et al.  Remembering: A Study in Experimental and Social Psychology , 1932 .