Departures from optimality: Understanding human analyst's information foraging in assisted requirements tracing

Studying human analyst's behavior in automated tracing is a new research thrust. Building on a growing body of work in this area, we offer a novel approach to understanding requirements analyst's information seeking and gathering. We model analysts as predators in pursuit of prey - the relevant traceability information, and leverage the optimality models to characterize a rational decision process. The behavior of real analysts with that of the optimal information forager is then compared and contrasted. The results show that the analysts' information diets are much wider than the theory's predictions, and their residing in low-profitability information patches is much longer than the optimal residence time. These uncovered discrepancies not only offer concrete insights into the obstacles faced by analysts, but also lead to principled ways to increase practical tool support for overcoming the obstacles.

[1]  Peter Pirolli,et al.  Computational models of information scent-following in a very large browsable text collection , 1997, CHI.

[2]  Jane Huffman Hayes,et al.  Automated Requirements Traceability: The Study of Human Analysts , 2010, 2010 18th IEEE International Requirements Engineering Conference.

[3]  Rachel K. E. Bellamy,et al.  Scents in Programs:Does Information Foraging Theory Apply to Program Maintenance? , 2007, IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC 2007).

[4]  Rachel K. E. Bellamy,et al.  How Programmers Debug, Revisited: An Information Foraging Theory Perspective , 2013, IEEE Transactions on Software Engineering.

[5]  Hoyt Lougee,et al.  SOFTWARE CONSIDERATIONS IN AIRBORNE SYSTEMS AND EQUIPMENT CERTIFICATION , 2001 .

[6]  Nan Niu,et al.  Source code indexing for automated tracing , 2011, TEFSE '11.

[7]  Jane Huffman Hayes,et al.  Advancing candidate link generation for requirements tracing: the study of methods , 2006, IEEE Transactions on Software Engineering.

[8]  Giuliano Antoniol,et al.  Recovering Traceability Links between Code and Documentation , 2002, IEEE Trans. Software Eng..

[9]  Jane Huffman Hayes,et al.  On human analyst performance in assisted requirements tracing: Statistical analysis , 2011, 2011 IEEE 19th International Requirements Engineering Conference.

[10]  Alexander Egyed,et al.  Effort and Quality of Recovering Requirements-to-Code Traces: Two Exploratory Experiments , 2010, 2010 18th IEEE International Requirements Engineering Conference.

[11]  Jane Cleland-Huang,et al.  Clustering support for automated tracing , 2007, ASE '07.

[12]  Nan Niu,et al.  Enhancing candidate link generation for requirements tracing: The cluster hypothesis revisited , 2012, 2012 20th IEEE International Requirements Engineering Conference (RE).

[13]  Jane Huffman Hayes,et al.  Humans in the traceability loop: can't live with 'em, can't live without 'em , 2005, TEFSE '05.

[14]  Rachel K. E. Bellamy,et al.  Reactive information foraging for evolving goals , 2010, CHI.

[15]  Genny Tortora,et al.  Recovering traceability links in software artifact management systems using information retrieval methods , 2007, TSEM.

[16]  Rachel K. E. Bellamy,et al.  Using information scent to model the dynamic foraging behavior of programmers in maintenance tasks , 2008, CHI.

[17]  Jane Huffman Hayes,et al.  A Framework for Comparing Requirements Tracing Experiments , 2005, Int. J. Softw. Eng. Knowl. Eng..

[18]  Stefan Biffl,et al.  Value-Based Requirements Traceability: Lessons Learned , 2007, 15th IEEE International Requirements Engineering Conference (RE 2007).

[19]  Genny Tortora,et al.  IR-Based Traceability Recovery Processes: An Empirical Comparison of "One-Shot" and Incremental Processes , 2008, 2008 23rd IEEE/ACM International Conference on Automated Software Engineering.

[20]  Jane Huffman Hayes,et al.  Application of swarm techniques to requirements tracing , 2011, Requirements Engineering.

[21]  Andrea De Lucia,et al.  On the Equivalence of Information Retrieval Methods for Automated Traceability Link Recovery , 2010, 2010 IEEE 18th International Conference on Program Comprehension.

[22]  John R. Anderson.  The Adaptive Character of Thought , 1990 .

[23]  Pattie Maes,et al.  Social information filtering: algorithms for automating “word of mouth” , 1995, CHI '95.

[24]  Brad A. Myers,et al.  An Exploratory Study of How Developers Seek, Relate, and Collect Relevant Information during Software Maintenance Tasks , 2006, IEEE Transactions on Software Engineering.

[25]  Jane Huffman Hayes,et al.  Towards overcoming human analyst fallibility in the requirements tracing process: NIER track , 2011, 2011 33rd International Conference on Software Engineering (ICSE).

[26]  Ed H. Chi,et al.  Using information scent to model user information needs and actions and the Web , 2001, CHI.

[27]  Jane Huffman Hayes,et al.  How do we trace requirements: an initial study of analyst behavior in trace validation tasks , 2011, CHASE.

[28]  Stephen Clark,et al.  Best Practices for Automated Traceability , 2007, Computer.

[29]  P. Pirolli,et al.  Spread of activation. , 1984 .

[30]  Amanda Spink,et al.  Real life information retrieval: a study of user queries on the Web , 1998, SIGF.

[31]  Nan Niu,et al.  Information foraging as a foundation for code navigation: NIER track , 2011, 2011 33rd International Conference on Software Engineering (ICSE).

[32]  Jane Cleland-Huang,et al.  Improving automated requirements trace retrieval: a study of term-based enhancement methods , 2010, Empirical Software Engineering.

[33]  Pamela Effrein Sandstrom,et al.  Information Foraging Theory: Adaptive Interaction with Information , 2010, J. Assoc. Inf. Sci. Technol..

[34]  Marti A. Hearst.  Search User Interfaces , 2009 .

[35]  H. Schütze,et al.  Dimensions of meaning , 1992, Supercomputing '92.

[36]  David J. Groggel,et al.  Practical Nonparametric Statistics , 2000, Technometrics.

[37]  Juha Savolainen,et al.  A Framework for Examining Topical Locality in Object-Oriented Software , 2012, 2012 IEEE 36th Annual Computer Software and Applications Conference.

[38]  Christopher D. Manning,et al.  Introduction to Information Retrieval , 2010, J. Assoc. Inf. Sci. Technol..

[39]  John C. Grundy,et al.  Improving automated documentation to code traceability by combining retrieval techniques , 2011, 2011 26th IEEE/ACM International Conference on Automated Software Engineering (ASE 2011).

[40]  Jane Cleland-Huang,et al.  Poirot: A Distributed Tool Supporting Enterprise-Wide Automated Traceability , 2006, 14th IEEE International Requirements Engineering Conference (RE'06).

[41]  Nan Niu,et al.  TraCter: A tool for candidate traceability link clustering , 2011, 2011 IEEE 19th International Requirements Engineering Conference.

[42]  Rachel K. E. Bellamy,et al.  Modeling programmer navigation: A head-to-head empirical evaluation of predictive models , 2011, 2011 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC).