A testing methodology and architecture for computer supported cooperative work software

Despite enormous potential, CSCW software is still immature. In particular, leading researchers in both the CSCW and testing fields have noted CSCW testing tools are nonexistent This thesis contributes a methodology and architecture for execution based testing of CSCW software. The CbSCW Abpplication MEthodoLOgy for Tbesting (CAMELOT) provides an organized set of specific techniques that can be used for technological evaluation. The evaluation is organized into two phases: single user and multi-user. Single user evaluation is subdivided further into general computing and human computer interaction. General computing examines software components that provide basic application capabilities. Human computer interaction focuses on the interface between the user and the software application. Multi-user evaluation examines distributed computing and human-human interaction. Distributed computing scrutinizes components responsible for multitasking and multiprocessing in the application at the thread, process, processor and machine level. Human-human interaction focuses on how the software facilitates interaction between users during application use. Rebecca, our testing architecture contributes to both general and multiuser testing systems. In the area of general testing Rebecca: (1) Provides an extensible component and event model that allows the record/playback of non-GUI events; (2) Allows selective event recording through record filtration; (3) Promotes the integration of the test system into the development environment Outputs test scripts in the developer's native language; (4) Reduces re-recording using component-centric events and runtime component resolution; (5) Simplifies the test process using a simple VCR-like interface. In the area of multiuser testing Rebecca: (1) Integrates live users into a test session with triggers that playback virtual user behavior based on user interface, state change, timer, or user customized events; (2) Provides runtime configuration of triggers via the threshold models; (3) Simplifies virtual user synchronization with deadlock detection and recovery; (4) Maintains IPC independence, but allows IPC to be recorded; (5) Scales well with a resource conserving architecture. Our architecture was implemented in Java as a working system called Rebecca-J. The methodology, architecture, and working system were evaluated by testing a mature CSCW application.

[1]  Roger L. Mellgren,et al.  Foundations of psychology , 1976 .

[2]  Lori L. Pollock,et al.  All-du-path coverage for parallel programs , 1998, ISSTA '98.

[3]  Siddhartha R. Dalal,et al.  The Automatic Efficient Test Generator (AETG) system , 1994, Proceedings of 1994 IEEE International Symposium on Software Reliability Engineering.

[4]  Mary Lou Soffa,et al.  Using a goal-driven approach to generate test cases for GUIs , 1999, Proceedings of the 1999 International Conference on Software Engineering (IEEE Cat. No.99CB37002).

[5]  Marc Snir.  Scalable parallel computing: the IBM 9076 scalable POWERparallel 1 , 1993, SPAA '93.

[6]  David A. Carrington,et al.  Test template framework: a specification-based testing case study , 1993, ISSTA '93.

[7]  Constance L. Heitmeyer,et al.  The SCR Method for Formally Specifying, Verifying, and Validating Requirements: Tool Support , 1997, Proceedings of the (19th) International Conference on Software Engineering.

[8]  David Kaufer,et al.  Computer support for distributed collaborative writing: defining parameters of interaction , 1994, CSCW '94.

[9]  Amy Bruckman,et al.  MOOSE goes to school: a comparison of three classrooms using a CSCL environment , 1997, CSCL.

[10]  Jay F. Nunamaker,et al.  Electronic Meeting Support: The GroupSystems Concept , 1991, Int. J. Man Mach. Stud..

[11]  Cheryl Geisler,et al.  Team learning through computer supported collaborative design , 1995, CSCL.

[12]  Judith S. Olson,et al.  Groupwork close up: a comparison of the group design process with and without a simple group editor , 1993, TOIS.

[13]  Cliff B. Jones,et al.  Systematic software development using VDM (2. ed.) , 1990, Prentice Hall International Series in Computer Science.

[14]  Pavel Curtis,et al.  MUDs grow up: social virtual reality in the real world , 1994, Proceedings of COMPCON '94.

[15]  Sunil K. Sarin,et al.  Experiences with workflow management: issues for the next generation , 1994, CSCW '94.

[16]  B. Miller,et al.  The Paradyn Parallel Performance Measurement Tools , 1995 .

[17]  John Gannon,et al.  State-based model checking of event-driven system requirements , 1991, SIGSOFT '91.

[18]  J. Michael Spivey,et al.  The Z notation - a reference manual , 1992, Prentice Hall International Series in Computer Science.

[19]  Debra J. Richardson,et al.  Specification-based test oracles for reactive systems , 1992, International Conference on Software Engineering.

[20]  Steven L. Rohall,et al.  The Rendezvous architecture and language for constructing multiuser applications , 1994, TCHI.

[21]  G. A. Geist,et al.  PICL. Portable Instrumented Communication Library , 1990 .

[22]  Allen D. Malony,et al.  Capturing and automating performance diagnosis: the Poirot approach , 1995, Proceedings of 9th International Parallel Processing Symposium.

[23]  Annemarie Ferraro.  An examination of the collaborative design process using multiple media resources and sharing protocols , 1998 .

[24]  Barry L. Kurtz,et al.  An introduction to computer science with modula-2 , 1988 .

[25]  Sudhir R. Ahuja,et al.  The Rapport multimedia conferencing system-a software overview , 1988, [1988] Proceedings. 2nd IEEE Conference on Computer Workstations.

[26]  Dirk E. Mahling,et al.  A collaborative environment for semi-structured medical problem based learning , 1995, CSCL.

[27]  Brian R. Gaines,et al.  Collaboration through concept maps , 1995, CSCL.

[28]  Jacob Buur,et al.  Replacing usability testing with user dialogue , 1999, CACM.

[29]  J. D. Fletcher,et al.  Beyond DoD: non-defense training and education applications of DIS , 1995 .

[30]  Marc J. Balcer,et al.  The category-partition method for specifying and generating fuctional tests , 1988, CACM.

[31]  W. W. Royce,et al.  Managing the development of large software systems: concepts and techniques , 1987, ICSE '87.

[32]  Andrew S. Tanenbaum,et al.  Computer Networks, Second Edition , 1981 .

[33]  Carl Gutwin,et al.  Supporting awareness of others in groupware , 1996, CHI Conference Companion.

[34]  Daniel C. Edelson,et al.  A design for effective support of inquiry and collaboration , 1995, CSCL.

[35]  Simeon C. Ntafos,et al.  An Evaluation of Random Testing , 1984, IEEE Transactions on Software Engineering.

[36]  Saul Greenberg,et al.  GroupWare for Real-Time Drawing: A Designer's Guide , 1995 .

[37]  Thomas W. Malone,et al.  Intelligent Information Sharing Systems , 1986 .

[38]  Prasun Dewan,et al.  Access control for collaborative environments , 1992, CSCW '92.

[39]  Ellen Christiansen,et al.  Making distance learning collaborative , 1995, CSCL.

[40]  Saul Greenberg,et al.  GROUPWARE TOOLKITS FOR SYNCHRONOUS WORK , 1996 .

[41]  C. A. R. HOARE,et al.  An axiomatic basis for computer programming , 1969, CACM.

[42]  Antonia Bertolino,et al.  Automatic Generation of Path Covers Based on the Control Flow Analysis of Computer Programs , 1994, IEEE Trans. Software Eng..

[43]  Andreas Ulrich,et al.  International workshop on testing distributed component-based systems , 1999, Proceedings of the 1999 International Conference on Software Engineering (IEEE Cat. No.99CB37002).

[44]  Judith S. Olson,et al.  Concurrent editing: the group's interface , 1990, INTERACT.

[45]  Joe John Duran.  Mainstream Videoconferencing: A Developer's Guide to Distance Multimedia , 1997 .

[46]  Rémy Evard,et al.  Collaborative Networked Communication: MUDs as Systems Tools , 1993, LISA.

[47]  James J. Horning,et al.  The Larch Family of Specification Languages , 1985, IEEE Software.

[48]  Zohar Manna,et al.  The Logic of Computer Programming , 1978, IEEE Transactions on Software Engineering.

[49]  D. J. Robson,et al.  Applying formal specification and functional testing to graphical user interfaces , 1991, [1991] Proceedings, Advanced Computer Technology, Reliable Systems and Applications.

[50]  Michal Young.  ISSTA 98 : Proceedings of the ACM SIGSOFT International Symposium on Software Testing and Analysis, Clearwater Beach, Florida, USA, March 2-5, 1998 , 1998 .

[51]  Leonard Krasner,et al.  Behavior influence and personality;: The social matrix of human action , 1973 .

[52]  B. Schneirdeman,et al.  Designing the User Interface: Strategies for Effective Human-Computer Interaction , 1998 .

[53]  William W. Gaver,et al.  Effective sounds in complex systems: the ARKOLA simulation , 1991, CHI.

[54]  R. Kemmerer,et al.  Hoare ’ s Axiomatic Semantics , 2002 .

[55]  Clarence A. Ellis,et al.  Groupware: some issues and experiences , 1991, CACM.

[56]  Prasun Dewan,et al.  A high-level and flexible framework for implementing multiuser user interfaces , 1992, TOIS.

[57]  Janet Bowers.  An alternative perspective for developing a mathematical microworld , 1995, CSCL.

[58]  Jeff Sokolov.  Methodologies for evaluation of collaborative systems , 1999, SIGG.

[59]  Barton P. Miller,et al.  The Paradyn Parallel Performance Measurement Tool , 1995, Computer.

[60]  Hidehiko Okada,et al.  GUITESTER: A Log-Based Usability Testing Tool for Graphical Usser Interfaces , 1999 .

[61]  Lynette Hirschman,et al.  Scenario-based evaluation of loosely-integrated collaborative systems , 2000, CHI Extended Abstracts.

[62]  M. Hauben,et al.  Netizens: On the History and Impact of Usenet and the Internet , 1998, First Monday.

[63]  David Kaufer,et al.  Issues in the design of computer support for co-authoring and commenting , 1990, CSCW '90.

[64]  Glenford J. Myers,et al.  Art of Software Testing , 1979 .

[65]  Stephen Travis Pope,et al.  A cookbook for using the model-view controller user interface paradigm in Smalltalk-80 , 1988 .

[66]  Keith A. Lantz,et al.  An experiment in integrated multimedia conferencing , 1986, CSCW '86.

[67]  Yvonne Rogers,et al.  PETRA: Participatory Evaluation Through Redesign and Analysis , 1995, Interact. Comput..

[68]  Marlene Scardamalia,et al.  Computer Support for Knowledge-Building Communities , 1994 .

[69]  Annita Fjuk.  Towards and analytical framework for CSCdistanceL , 1995, CSCL.

[70]  S. Louis Hakimi,et al.  On Path Cover Problems in Digraphs and Applications to Program Testing , 1979, IEEE Transactions on Software Engineering.

[71]  Gregg Rothermel,et al.  What you see is what you test: a methodology for testing form-based visual programs , 1998, Proceedings of the 20th International Conference on Software Engineering.

[72]  Thomas J. Ostrand,et al.  A visual test development environment for GUI systems , 1998, ISSTA.

[73]  Norbert A. Streitz,et al.  DOLPHIN: integrated meeting support across local and remote desktop environments and LiveBoards , 1994, CSCW '94.

[74]  George S. Avrunin,et al.  A practical technique for bounding the time between events in concurrent real-time systems , 1993, ISSTA '93.

[75]  Allen Van Gelder,et al.  Computer Algorithms: Introduction to Design and Analysis , 1978 .

[76]  Cheryl Geisler,et al.  A collaborative learning environment for intellectual teamwork across the curriculum , 1997, Proceedings Frontiers in Education 1997 27th Annual Conference. Teaching and Learning in an Era of Change.

[77]  Douglas C. Engelbart,et al.  A research center for augmenting human intellect , 1968, AFIPS Fall Joint Computing Conference.

[78]  Jakob Nielsen,et al.  Heuristic evaluation of user interfaces , 1990, CHI '90.

[79]  Robert E. Kraut,et al.  Collaborative document production using quilt , 1988, CSCW '88.

[80]  George S. Avrunin,et al.  Automated Derivation of Time Bounds in Uniprocessor Concurrent Systems , 1994, IEEE Trans. Software Eng..

[81]  Ben Shneiderman,et al.  Designing the User Interface: Strategies for Effective Human-Computer Interaction , 1998 .

[82]  Tore Urnes,et al.  Tools for Implementing Groupware: Survey and Evaluation , 1994 .

[83]  Mary Campione,et al.  The Java Tutorial: Object-Oriented Programming for the Internet , 1996 .

[84]  Jeffrey J. Hendrickson,et al.  Integrated data capture and analysis tools for research and testing on graphical user interfaces , 1992, CHI.

[85]  Amy Bruckman.  Programming for Fun: MUDs as a Context for Collaborative Learning. , 1994 .

[86]  Prasun Dewan.  Tools for implementing multiuser user interfaces , 1993 .

[87]  Mark S. Ackerman,et al.  Social activity indicators: interface components for CSCW systems , 1995, UIST '95.

[88]  James D. Hollan,et al.  Strategic directions in human-computer interaction , 1996, CSUR.

[89]  Elaine J. Weyuker,et al.  Selecting Software Test Data Using Data Flow Information , 1985, IEEE Transactions on Software Engineering.

[90]  David Luckham,et al.  Two Dimensional Pinpointing: An Application of Formal Specification to Debugging Packages , 1989 .

[91]  Leslie G. Valiant,et al.  A bridging model for parallel computation , 1990, CACM.

[92]  Saul Greenberg,et al.  GROUPKIT: a groupware toolkit for building real-time conferencing applications , 1992, CSCW '92.

[93]  Saul Greenberg,et al.  TurboTurtle: a collaborative microworld for exploring Newtonian physics , 1995 .

[94]  Daniel G. Bobrow,et al.  Beyond the chalkboard: computer support for collaboration and problem solving in meetings , 1988, CACM.

[95]  Lee J. White.  Regression testing of GUI event interactions , 1996, 1996 Proceedings of International Conference on Software Maintenance.

[96]  Brad A. Myers,et al.  Why are Human-Computer interfaces Difficult to Design and Implement? , 1993 .

[97]  Clare-Marie Karat,et al.  Comparison of empirical testing and walkthrough methods in user interface evaluation , 1992, CHI.

[98]  Louise E. Moser,et al.  A graphical interval logic for specifying concurrent systems , 1994, TSEM.

[99]  Thomas W. Mastaglio,et al.  A Large-Scale Complex Virtual Environment for Team Training , 1995, Computer.

[100]  David Crow,et al.  The evolution of CSCW: past, present and future development , 1997, SGCH.

[101]  Mary Campione,et al.  The Java Tutorial, Second Edition: Object-Oriented Programming for the Internet (Book/CD) , 1998 .

[102]  E. Glinert.  An Architecture for Testing Synchronous Multiuser Software , 1998 .

[103]  Raj Jain,et al.  The art of computer systems performance analysis - techniques for experimental design, measurement, simulation, and modeling , 1991, Wiley professional computing.

[104]  Ian J. Hayes,et al.  Specification directed module testing , 1986, IEEE Transactions on Software Engineering.

[105]  Ramez Elmasri,et al.  Fundamentals of Database Systems , 1989 .

[106]  Harlan D. Mills,et al.  Certifying the reliability of software , 1986, IEEE Transactions on Software Engineering.

[107]  Ephraim P. Glinert,et al.  Exploring Collaborative Learning in Rensselaer's Classroom-in-the-Round , 1998 .