Cognitive Evaluation of Human-Robot Systems: A Method for Analyzing Cognitive Change in Human-Robot Systems

To help answer questions about the behavior of participants in human-robot systems, we propose the cognitive evaluation of human-robot systems (CEHRS) method based on our work with the personal exploration rover (PER). The CEHRS method consists of six steps: (1) identify all system participants, (2) collect data from all participant groups, including the system's creators, (3) analyze participant data in light of system-wide goals, (4) answer targeted questions about each participant group to determine the flow of knowledge, information, and influence throughout the system, (5) look for inconsistencies in the knowledge and beliefs of different participant groups, and (6) make recommendations for improvement. We offer this comprehensive, human-centered evaluation method as a starting point for future work in understanding cognitive change in human-robot interactions

[1]  Edwin Hutchins,et al.  The technology of team navigation , 1990 .

[2]  Illah R. Nourbakhsh,et al.  Long-Term Human-Robot Interaction: The Personal Exploration Rover and Museum Docents , 2005, AIED.

[3]  K. Dautenhahn,et al.  A Survey of Socially Interactive Robots : Concepts , Design , and Applications , 1992 .

[4]  Jean Scholtz,et al.  Common metrics for human-robot interaction , 2006, HRI '06.

[5]  Illah R. Nourbakhsh,et al.  A survey of socially interactive robots , 2003, Robotics Auton. Syst..

[6]  Sachiko Komagata.  Comparing expert and novice understanding of a complex system from the perspective of structures, behaviors, and functions. , 2004 .

[7]  Takayuki Kanda,et al.  Interaction debugging: an integral approach to analyze human-robot interaction , 2006, HRI '06.

[8]  Philip N. Johnson-Laird,et al.  Mental Models in Cognitive Science , 1980, Cogn. Sci..

[9]  Jean Scholtz,et al.  Beyond usability evaluation: analysis of human-robot interaction at a major robotics competition , 2004 .

[10]  Anders Green,et al.  Social and collaborative aspects of interaction with a service robot , 2003, Robotics Auton. Syst..

[11]  M. Scaife,et al.  “Because a robot's brain hasn't got a brain, it just controls itself” — Children's attributions of brain related behaviour to intelligent artefacts , 1996 .

[12]  J. Lave.  Understanding practice: The practice of learning , 1993 .

[13]  W. Brewer,et al.  Mental models of the earth: A study of conceptual change in childhood , 1992, Cognitive Psychology.

[14]  Jeffrey A. Hoffman,et al.  Towards a formal representation for heterogeneous teams of humans and robots in space exploration , 2005 .

[15]  Sara B. Kiesler,et al.  Eliciting information from people with a gendered humanoid robot , 2005, ROMAN 2005. IEEE International Workshop on Robot and Human Interactive Communication, 2005..

[16]  Robin R. Murphy,et al.  Moonlight in Miami: a field study of human-robot interaction in the context of an urban search and rescue disaster response training exercise , 2004 .

[17]  Lucy A. Suchman,et al.  Making work visible , 1995, CACM.

[18]  Susan R. Fussell,et al.  Constructing shared communicative environments , 1991, Perspectives on socially shared cognition.

[19]  Jean Scholtz,et al.  Development of a test bed for evaluating human-robot performance for explosive ordnance disposal robots , 2006, HRI '06.

[20]  Kendall Scott,et al.  UML distilled - a brief guide to the Standard Object Modeling Language (2. ed.) , 2000, notThenot Addison-Wesley object technology series.

[21]  M. Chi.  Quantifying Qualitative Analyses of Verbal Data: A Practical Guide , 1997 .

[22]  Holly A. Yanco,et al.  A Taxonomy for Human-Robot Interaction , 2002 .

[23]  Kangde Yao,et al.  A Brief Guide to the Standard Object Modelling Language , 2000 .

[24]  Robert E. Kraut,et al.  Intellectual Teamwork: Social and Technological Foundations of Cooperative Work , 1990 .

[25]  Kevin Crowley,et al.  The Personal Exploration Rover: The Ground-up Design, Deployment and Educational Evaluation of an Educational Robot for Unmediated Informal Learning Sites , 2004 .

[26]  Jean Scholtz,et al.  Theory and evaluation of human robot interactions , 2003, 36th Annual Hawaii International Conference on System Sciences, 2003. Proceedings of the.