Evaluation of a Scientific Collaboratory System: Investigating Utility before Deployment

The evaluation of scientific collaboratories has lagged behind their development, and fundamental questions have yet to be answered: Can distributed scientific research produce high quality results? Do the capabilities afforded by collaboratories outweigh their disadvantages from scientists’ perspectives? Are there system features and performance characteristics that are common to successful collaboratory systems? Our goal is to help answer such fundamental questions by evaluating a specific scientific collaboratory system called the nanoManipulator Collaboratory System. The system is a set of tools that provide collaborative interactive access to a specialized scientific instrument and office applications. To evaluate the system, we conducted a repeated-measures controlled experiment that compared the outcomes and process of scientific work completed by 20 pairs of participants (upper level undergraduate science students) working face-to-face and remotely. We collected scientific outcomes (graded lab reports) to investigate the quality of scientific work, post-questionnaire data to measure intentions to adopt the system, and post-interviews to understand the participants’ views of doing science under both conditions. We hypothesized that study participants would be less effective, report more difficulty, and be less favorably inclined to adopt the system when collaborating remotely. However, the quantitative data showed no statistically significant differences with respect to effectiveness and adoption. Furthermore, in post-interviews participants reported advantages and disadvantages working under both conditions but developed work-arounds to cope with the perceived disadvantages of collaborating remotely. A theoretical explanation for the results can be found in the theory of the life-world (Schutz & Luckman, 1973, 1989). Considered as a whole, the analysis leads us to conclude there is positive potential for the development and adoption of scientific collaboratory systems.

[1]  Diane H. Sonnenwald,et al.  Expectations for a scientific collaboratory: a case study , 2003, GROUP.

[2]  G. Olson,et al.  From Laboratories to Collaboratories: A New Organizational Form for Scientific Collaboration , 1997 .

[3]  E. Rogers,et al.  Diffusion of innovations , 1964, Encyclopedia of Sport Management.

[4]  Mary C. Whitton,et al.  Designing to support situation awareness across distances: an example from a scientific collaboratory , 2004, Inf. Process. Manag..

[5]  Wanda J. Orlikowski,et al.  Learning from Notes: organizational issues in groupware implementation , 1992, CSCW '92.

[6]  Carl Gutwin,et al.  A usability study of workspace awareness widgets , 1996, CHI Conference Companion.

[7]  W. G. Matthews,et al.  Controlled manipulation of molecular samples with the nanoManipulator , 2000 .

[8]  J. C. Flanagan.  Psychological Bulletin THE CRITICAL INCIDENT TECHNIQUE , 2022 .

[9]  Mary C. Whitton,et al.  Managing collaboration in the distributed nano Manipulator , 2003 .

[10]  Vernon L. Chi,et al.  Surface modification tools in a virtual environment interface to a scanning probe microscope , 1995, I3D '95.

[11]  Telecommunications Board,et al.  National Collaboratories: Applying Information Technology for Scientific Research , 1993 .

[12]  Robert E. Kraut,et al.  Collaboration in performance of physical tasks: effects on outcomes and communication , 1996, CSCW '96.

[13]  E. List.  The Structures of the Life-World , 1977 .

[14]  Frederick P. Brooks,et al.  Quantitative manipulation of DNA and viruses with the nanomanipulator scanning force microscope , 1999 .

[15]  Mary C. Whitton,et al.  Enabling distributed collaborative science , 2000, CSCW '00.

[16]  Jonathan J. Cadiz,et al.  Coordination, overload and team performance: effects of team communication strategies , 1998, CSCW '98.

[17]  W A Wulf,et al.  The collaboratory opportunity. , 1993, Science.

[18]  Jonathan Grudin,et al.  EIGHT CHALLENGES FOR DEVELOPERS , 2010 .

[19]  Carl Gutwin,et al.  A review of groupware evaluations , 2000, Proceedings IEEE 9th International Workshops on Enabling Technologies: Infrastructure for Collaborative Enterprises (WET ICE 2000).

[20]  Mary C. Whitton,et al.  Designing to Support Collaborative Scientific Research Across Distances: The nanoManipulator Environment , 2001, Collaborative Virtual Environments.

[21]  B. Berg.  Qualitative Research Methods for the Social Sciences , 1989 .

[22]  Thomas A. Finholt,et al.  Collaboratories , 2002, Annu. Rev. Inf. Sci. Technol..

[23]  Caroline Gale.  The effects of gaze awareness on dialogue in a video-based collaborative manipulative task , 1998, CHI Conference Summary.

[24]  James D. Myers,et al.  Collaboratories: Doing Science on the Internet , 1996, Computer.

[25]  Thomas Luckmann,et al.  The Structures of the Life World V1 Op , 1973 .

[26]  Stephanie D. Teasley,et al.  Groupware in the wild: lessons learned from a year of virtual collocation , 1996, CSCW '96.

[27]  Mike Phillips,et al.  Media space , 1998, Digit. Creativity.

[28]  J. Knottnerus,et al.  Real world research. , 2010, Journal of clinical epidemiology.

[29]  Karen Ruhleder,et al.  Steps toward an ecology of infrastructure , 1996 .

[30]  Wanda J. Orlikowski,et al.  Learning from Notes: organizational issues in groupware implementation , 1992, CSCW '92.

[31]  Hari Singh Nalwa,et al.  Handbook of nanostructured materials and nanotechnology , 2000 .

[32]  Mary C. Whitton,et al.  Using innovation diffusion theory to guide collaboration technology evaluation: work in progress , 2001, Proceedings Tenth IEEE International Workshop on Enabling Technologies: Infrastructure for Collaborative Enterprises. WET ICE 2001.

[33]  Nicholas S. Vonortas.  The Process of Technological Innovation , 1997 .

[34]  William W. Gaver,et al.  Sound Support for Collaboration , 1991, ECSCW.

[35]  Ben Shneiderman,et al.  Designing The User Interface , 2013 .

[36]  Bruce R. Schatz,et al.  Building an Electronic Community System , 1991, J. Manag. Inf. Syst..

[37]  Prasun Dewan,et al.  An Integrated Approach to Designing and Evaluating Collaborative Applications and Infrastructures , 2004, Computer Supported Cooperative Work (CSCW).

[38]  Judith S. Olson,et al.  Distance Matters , 2000, Hum. Comput. Interact..

[39]  Austin Henderson,et al.  Your place or mine? Learning from long-term use of Audio-Video communication , 1996, Computer Supported Cooperative Work (CSCW).