2012 Proceedings of the Performance Metrics for Intelligent Systems (PerMI'12) Workshop

System evaluations have been conducted in robotics and human-robot interaction for many years. These evaluations usually take place after a robot system has been designed and built as a way to validate the completed system. However, by performing evaluation only at the end of the development cycle, we lose opportunities to create systems with even better performance. Taking inspiration from human-computer interaction, we can design more effective robot systems for human-robot interaction by incorporating user feedback in the initial design phase. This talk will present a number of such formative evaluations from a variety of robotics domains, including assistive robotics and telepresence robot systems. BIOGRAPHY Dr. Holly Yanco is Professor and Associate Chair of Computer Science at the University of Massachusetts Lowell. Her research interests include human-robot interaction, multi-touch computing, interface design, robot autonomy, fostering trust of autonomous systems, evaluation methods for human-robot interaction, and the use of robots in K-12 education to broaden participation in computer science. Her research has been funded by the National Science Foundation, including a Career Award, the Army Research Office, Microsoft, and the National Institute of Standards and Technology. Dr. Yanco is the General Chair of the 2012 ACM/IEEE International Conference on Human-Robot Interaction. She served on the Executive Council of the Association for the Advancement of Artificial Intelligence (AAAI) from 2006-2009 and was the Symposium Chair for AAAI from 2002-2005. She was awarded senior membership in AAAI in 2011. Dr. Yanco has a PhD and MS in Computer Science from the Massachusetts Institute of Technology (MIT) and a BA in Computer Science and Philosophy from Wellesley College. Mark Rice, Maritime Applied Physics Corporation Geographic Information Systems (GIS) as an Environment for Intelligent Systems Performance Measurement

[1]  Deb Roy,et al.  Grounded Situation Models for Robots: Bridging Language, Perception, and Action , 2005 .

[2]  Roel Vertegaal,et al.  Effects of Gaze on Multiparty Mediated Communication , 2000, Graphics Interface.

[3]  Alexander C. Berg,et al.  Who's In the Picture , 2004, NIPS 2004.

[4]  Cyrus Rashtchian,et al.  Every Picture Tells a Story: Generating Sentences from Images , 2010, ECCV.

[5]  Cordelia Schmid,et al.  Learning realistic human actions from movies , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[6]  G. A. Miller,et al.  The Intelligibility of Interrupted Speech , 1948 .

[7]  Eduard Hovy,et al.  NL Generation for Virtual Humans in a Complex Social Environment , 2003 .

[8]  Ellen Riloff,et al.  A Bootstrapping Method for Learning Semantic Lexicons using Extraction Pattern Contexts , 2002, EMNLP.

[9]  John F. Canny,et al.  More than face-to-face: empathy effects of video framing , 2009, CHI.

[10]  Panos Markopoulos,et al.  Measuring affective benefits and costs of awareness systems supporting intimate social networks , 2004 .

[11]  John C. Tang,et al.  Embodied social proxy: mediating interpersonal connection in hub-and-satellite teams , 2010, CHI.

[12]  George A. Miller,et al.  WordNet: A Lexical Database for English , 1995, HLT.

[13]  Juyang Weng,et al.  Developmental Robotics: Theory and Experiments , 2004, Int. J. Humanoid Robotics.

[14]  James H. Martin,et al.  Speech and Language Processing, 2nd Edition , 2008 .

[15]  David A. Forsyth,et al.  Object Recognition as Machine Translation: Learning a Lexicon for a Fixed Image Vocabulary , 2002, ECCV.

[16]  Daniel J. Brooks,et al.  Designing Telepresence Robot Systems for Use by People with Special Needs , 2011 .

[17]  Lawrence O'Gorman,et al.  Latency in Speech Feature Analysis for Telepresence Event Coding , 2010, 2010 20th International Conference on Pattern Recognition.

[18]  John C. Tang,et al.  What video can and cannot do for collaboration: A case study , 2005, Multimedia Systems.

[19]  Abigail Sellen,et al.  Remote Conversations: The Effects of Mediating Talk With Technology , 1995, Hum. Comput. Interact..

[20]  Silvia Coradeschi,et al.  Sense of Presence in a Robotic Telepresence Domain , 2011, HCI.

[21]  Refractor.  Vision , 2000, The Lancet.

[22]  Dan Klein,et al.  A Game-Theoretic Approach to Generating Spatial Descriptions , 2010, EMNLP.

[23]  Wolfram Burgard,et al.  Improving Grid-based SLAM with Rao-Blackwellized Particle Filters by Adaptive Proposals and Selective Resampling , 2005, Proceedings of the 2005 IEEE International Conference on Robotics and Automation.

[24]  Ben Taskar,et al.  Movie/Script: Alignment and Parsing of Video and Text Transcription , 2008, ECCV.

[25]  Holly A. Yanco,et al.  Telepresence Robots Roam the Halls of My Office Building , 2011 .

[26]  Hal Daumé,et al.  Approximate Scalable Bounded Space Sketch for Large Data NLP , 2011, EMNLP.

[27]  Leila Takayama,et al.  "Now, i have a body": uses and social norms for mobile remote presence in the workplace , 2011, CHI.

[28]  Susan R. Fussell,et al.  Anthropomorphic Interactions with a Robot and Robot–like Agent , 2008 .

[29]  Ekaterini Pastra.  Vision - language integration : a double-grounding case , 2004 .

[30]  Anton Nijholt,et al.  Eye gaze patterns in conversations: there is more to conversational agents than meets the eyes , 2001, CHI.

[31]  Paul J. Besl,et al.  A Method for Registration of 3-D Shapes , 1992, IEEE Trans. Pattern Anal. Mach. Intell..

[32]  M. Lombard,et al.  Measuring Presence: The Temple Presence Inventory , 2009 .

[33]  Xiang Cao,et al.  Home video communication: mediating 'closeness' , 2010, CSCW '10.

[34]  Leila Takayama,et al.  Assisted driving of a mobile remote presence system: System design and controlled user evaluation , 2011, 2011 IEEE International Conference on Robotics and Automation.

[35]  François Michaud,et al.  Exploratory design and evaluation of a homecare teleassistive mobile robotic system , 2010 .

[36]  A. Monk,et al.  A Look Is Worth a Thousand Words: Full Gaze Awareness in Video-Mediated Conversation , 2002 .

[37]  Cyrill Stachniss,et al.  On measuring the accuracy of SLAM algorithms , 2009, Auton. Robots.

[38]  Yeh-Liang Hsu,et al.  Developing a telepresence robot for interpersonal communication with the elderly in a home environment. , 2007, Telemedicine journal and e-health : the official journal of the American Telemedicine Association.

[39]  Rosanna E. Guadagno,et al.  Getting to know you: Face-to-face versus online interactions , 2011, Comput. Hum. Behav..

[40]  Larry S. Davis,et al.  Beyond Nouns: Exploiting Prepositions and Comparative Adjectives for Learning Visual Classifiers , 2008, ECCV.

[41]  Hiroshi Murase,et al.  Quantifying interpersonal influence in face-to-face conversations based on visual attention patterns , 2006, CHI Extended Abstracts.

[42]  Steven J. Karau,et al.  The effects of performance and team cohesion on attribution: a longitudinal simulation , 2004 .

[43]  Kieron Sheehy,et al.  Beaming children where they cannot go: telepresence robots and inclusive education: an exploratory study , 2011 .

[44]  Kathleen R. McKeown.  Query-focused Summarization Using Text-to-Text Generation: When Information Comes from Multilingual Sources , 2009 .

[45]  Katherine M. Tsui,et al.  Exploring use cases for telepresence robots , 2011, 2011 6th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[46]  John C. Tang,et al.  Motion and Attention in a Kinetic Videoconferencing Proxy , 2011, INTERACT.

[47]  Peter Werkhoven,et al.  How Conversations Change Over Time in Face-to-Face and Video-Mediated Communication , 2009 .

[48]  Danica Kragic,et al.  Learning Actions from Observations , 2010, IEEE Robotics & Automation Magazine.

[49]  Larry S. Davis,et al.  Observing Human-Object Interactions: Using Spatial and Functional Compatibility for Recognition , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[50]  Hadas Kress-Gazit,et al.  From structured english to robot motion , 2007, 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[51]  Barbara Caputo,et al.  Who's Doing What: Joint Modeling of Names and Verbs for Simultaneous Face and Pose Annotation , 2009, NIPS.

[52]  Dieter Fox,et al.  Following directions using statistical machine translation , 2010, HRI 2010.

[53]  Deb Roy,et al.  Towards situated speech understanding: visual context priming of language models , 2005, Comput. Speech Lang..

[54]  Steve Whittaker,et al.  Conversations Over Video Conferences: An Evaluation of the Spoken Aspects of Video-Mediated Communication , 1993, Hum. Comput. Interact..

[55]  Jean-Michel Morel,et al.  From Gestalt Theory to Image Analysis , 2008 .

[56]  Michael J. Singer,et al.  The Factor Structure of the Presence Questionnaire , 2005, Presence: Teleoperators & Virtual Environments.

[57]  Peter Ford Dominey,et al.  A cognitive neuroscience perspective on embodied language for human–robot cooperation , 2010, Brain and Language.

[58]  Masafumi Nishida,et al.  Eye-gaze experiments for conversation monitoring , 2009, IUCS.

[59]  Viswanath Venkatesh,et al.  A Longitudinal Investigation of Personal Computers in Homes: Adoption Determinants and Emerging Challenges , 2001, MIS Q..