Models and Experiments in Robotics

This chapter surveys the practices that are being employed in experimentally assessing the special class of computational models embedded in robots. The assessment of these models is particularly challenging mainly due to the difficulty of accurately estimating and modeling the interactions between the robots and their environments, especially in the case of autonomous robots, which make decisions without continuous human supervision. The field of autonomous robotics has recognized this difficulty and launched a number of initiatives to deal with it. This chapter, after a conceptual premise and a broad introduction to the experimental issues of robotics, critically reviews these initiatives that range from taking inspiration from traditional experimental practices, to simulations, benchmarking, standards, and competitions.

[1]  Gourab Sen Gupta,et al.  Robotics competitions in engineering eduction , 2000, 2009 4th International Conference on Autonomous Robots and Agents.

[2]  Stephan Hartmann,et al.  The World as a Process , 1996 .

[3]  Jerome A. Feldman,et al.  Rejuvenating experimental computer science: a report to the National Science Foundation and others , 1979, CACM.

[4]  Susanne E. Hambrusch,et al.  Gender demographics trends and changes in U.S. CS departments , 2011, CACM.

[5]  José Hernández-Orallo,et al.  AI Evaluation: past, present and future , 2014, ArXiv.

[6]  Peter J. Denning,et al.  Is computer science science? , 2005, CACM.

[7]  Clayton T. Morrison,et al.  Computer science can use more science , 2011, CACM.

[8]  Robin R. Murphy,et al.  A Case Study of How Mobile Robot Competitions Promote Future Research , 2001, RoboCup.

[9]  Natalia Juristo Juzgado,et al.  Replications of software engineering experiments , 2013, Empirical Software Engineering.

[10]  John Anderson,et al.  Robotics competitions as benchmarks for AI research , 2011, Knowl. Eng. Rev..

[11]  Alexei A. Efros,et al.  Unbiased look at dataset bias , 2011, CVPR 2011.

[12]  Raj Madhavan,et al.  Guest editorial: Special issue on characterizing mobile robot localization and mapping , 2009, Auton. Robots.

[13]  Raj Madhavan,et al.  The role of competitions in advancing intelligent systems: a practitioner's perspective , 2009, PerMIS.

[14]  Stefano Carpin,et al.  Quantitative Assessments of USARSim Accuracy , 2006 .

[15]  M. Barni,et al.  Pushing science into signal processing [my turn] , 2005, IEEE Signal Processing Magazine.

[16]  Sven Ove Hansson,et al.  The Role of Technology in Science: Philosophical Perspectives , 2015 .

[17]  Olivier Michel,et al.  Rat's Life: A Cognitive Robotics Benchmark , 2008, EUROS.

[18]  Jacky Baltes,et al.  A benchmark suite for mobile robots , 2000, Proceedings. 2000 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2000) (Cat. No.00CH37113).

[19]  Francesco Amigoni,et al.  Good Experimental Methodologies for Autonomous Robotics: From Theory to Practice , 2014 .

[20]  Thomas Bräunl,et al.  Research relevance of mobile robot competitions , 1999, IEEE Robotics Autom. Mag..

[21]  Susan Stepney,et al.  Engineering simulations for cancer systems biology. , 2012, Current drug targets.

[22]  Erwin Prassler,et al.  Comparative evaluation of robotic software integration systems: A case study , 2007, 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[23]  Javier Ruiz-del-Solar,et al.  RoboCup@Home: Analysis and results of evolving competitions for domestic and service robots , 2015, Artif. Intell..

[24]  Brandon Rohrer.  Accelerating progress in Artificial General Intelligence: Choosing a benchmark for natural world interaction , 2010, J. Artif. Gen. Intell..

[25]  Raj Madhavan,et al.  Draft Standards Development of Two Working Groups [Industrial Activities] , 2014, IEEE Robotics Autom. Mag..

[26]  Thomas Dean,et al.  A Retrospective of the AAAI Robot Competitions , 1997, AI Mag..

[27]  Raj Madhavan,et al.  Benchmarking and standardization of intelligent robotic systems , 2009, 2009 International Conference on Advanced Robotics.

[28]  Sven Wachsmuth,et al.  The Cognitive Interaction Toolkit - Improving Reproducibility of Robotic Systems Experiments , 2014, SIMPAR.

[29]  Peter J. Denning,et al.  The profession of ITComputing's paradigm , 2009, Commun. ACM.

[30]  Fernando Pérez-González,et al.  Putting Reproducible Signal Processing into Practice: A Case Study in Watermarking , 2007, 2007 IEEE International Conference on Acoustics, Speech and Signal Processing - ICASSP '07.

[31]  Peter Freeman,et al.  Back to experimentation , 2008, CACM.

[32]  Marco Bibuli,et al.  Towards Good Experimental Methodologies for Unmanned Marine Vehicles , 2013, EUROCAST.

[33]  Monica Reggiani,et al.  An insightful comparison between experiments in mobile robotics and in science , 2009, Auton. Robots.

[34]  Odest Chadwicke Jenkins,et al.  Recasting Robotics Challenges as Experiments [Competitions] , 2011 .

[35]  Peter J. Denning,et al.  ACM President's Letter: What is experimental computer science? , 1980, CACM.

[36]  Siddhartha S. Srinivasa,et al.  The YCB object and Model set: Towards common benchmarks for manipulation research , 2015, 2015 International Conference on Advanced Robotics (ICAR).

[37]  Alistair Isaac,et al.  Modeling without representation , 2013, Synthese.

[38]  M. Sridharan,et al.  Hydrochemical Facies and Ionic Exchange in Coastal Aquifers of Puducherry Region, India: Implications for Seawater Intrusion , 2017, Earth Systems and Environment.

[39]  Marvin V. Zelkowitz,et al.  Experimental Models for Validating Technology , 1998, Computer.

[40]  Elena R. Messina,et al.  Proceedings of the Performance Metrics for Intelligent Systems (PerMIS)2007 Workshop, August 28-30, 2007 , 2007 .

[41]  Ulrich Rückert,et al.  Teletesting: Path planning experimentation and benchmarking in the Teleworkbench , 2013, 2013 European Conference on Mobile Robots.

[42]  C. Drummond.  Replicability is not Reproducibility:Nor is it Good Science , 2009 .

[43]  Allen Newell,et al.  Computer science as empirical inquiry: symbols and search , 1976, CACM.

[44]  Jelena Kovacevic,et al.  Reproducible research in signal processing , 2009, IEEE Signal Process. Mag..

[45]  Walter F. Tichy,et al.  Should Computer Scientists Experiment More? , 1998, Computer.

[46]  Marvin V. Zelkowitz,et al.  Experimental validation in software engineering , 1997, Inf. Softw. Technol..

[47]  Paul R. Cohen,et al.  Benchmarks, Test Beds, Controlled Experimentation, and the Design of Agent Architectures , 1993, AI Mag..

[48]  Robin R. Murphy,et al.  Using Robot Competitions to Promote Intellectual Development , 2000, AI Mag..

[49]  Stephan Hartmann,et al.  The World as a Process: Simulations in the Natural and Social Sciences , 1996 .