Multi-Relationship Evaluation Design (MRED): An Interactive Test Plan Designer for Advanced and Emerging Technologies

[1]  Jian Ma,et al.  An approach to multiple attribute decision making based on incomplete information on alternatives , 1999, Proceedings of the 32nd Annual Hawaii International Conference on Systems Sciences. 1999. HICSS-32. Abstracts and CD-ROM of Full Papers.

[2]  J. Harsanyi.  Cardinal Welfare, Individualistic Ethics, and Interpersonal Comparisons of Utility , 1955 .

[3]  Brian A. Weiss,et al.  Intelligent systems for urban search and rescue: challenges and lessons learned , 2003, SPIE Defense + Commercial Sensing.

[4]  Donald G. Saari,et al.  Which is better: the Condorcet or Borda winner? , 2006, Soc. Choice Welf..

[5]  A. Sen,et al.  Social Choice Theory: A Re-Examination , 1977 .

[6]  Desheng Dash Wu,et al.  Performance evaluation: An integrated method using data envelopment analysis and fuzzy preference relations , 2009, Eur. J. Oper. Res..

[7]  M. Fleming,et al.  A Cardinal Concept of Welfare , 1952 .

[8]  Mark Morrison,et al.  Aggregation Biases in Stated Preference Studies , 2000 .

[9]  Lars Peter Østerdal,et al.  Cardinal Scales for Health Evaluation , 2010, Decis. Anal..

[10]  E.R. Messina,et al.  Measuring the Performance of Urban Search and Rescue Robots , 2007, 2007 IEEE Conference on Technologies for Homeland Security.

[11]  Brian A. Weiss,et al.  Multi-Relationship Evaluation Design: Formalizing Test Plan Input and Output Elements for Evaluating Developing Intelligent Systems , 2011 .

[12]  Adele E. Howe,et al.  How evaluation guides AI research , 1988 .

[13]  Robert Bialczak,et al.  Comparison Methodology for Robotic Operator Control Units , 2002 .

[14]  Rida Laraki,et al.  A theory of measuring, electing, and ranking , 2007, Proceedings of the National Academy of Sciences.

[15]  Jizhong Xiao,et al.  Design and Performance Analysis of Retractable-claw wheels for Field Robots , 2010, Int. J. Robotics Autom..

[16]  Brian A. Weiss,et al.  Evolution of a Performance Metric for Urban Search and Rescue Robots (2003) , 2003 .

[17]  Jean Scholtz,et al.  A field study of two techniques for situation awareness for robot navigation in urban search and rescue , 2005, ROMAN 2005. IEEE International Workshop on Robot and Human Interactive Communication, 2005..

[18]  Frank Lopez,et al.  Unmanned and autonomous systems mission based test and evaluation , 2009, PerMIS.

[19]  John C. Mankins,et al.  Technology readiness assessments: A retrospective , 2009 .

[20]  Jean Scholtz,et al.  Implementation of a situation awareness assessment tool for evaluation of human-robot interfaces , 2005, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[21]  Daniele Nardi,et al.  Performance evaluation of pure-motion tasks for mobile robots with respect to world models , 2009, Auton. Robots.

[22]  M Forrest,et al.  Ordinal scale and statistics in medical research. , 1986, British medical journal.

[23]  Lefteri H. Tsoukalas,et al.  Performance Metrics for Intelligent Systems An Engineering Perspective , 2002 .

[24]  Stefano Carpin,et al.  USARSim: Providing a Framework for Multi-Robot Performance Evaluation | NIST , 2006 .

[25]  Brian A. Weiss,et al.  Multi-relationship evaluation design: modeling an automatic test plan generator , 2012, PerMIS.

[26]  J. Geanakoplos.  Three brief proofs of Arrow’s Impossibility Theorem , 2001 .

[27]  Frederick Mosteller,et al.  Data Analysis and Regression , 1978 .

[28]  Arnoud Visser,et al.  Evaluating maps produced by urban search and rescue robots: lessons learned from RoboCup , 2009, Auton. Robots.

[29]  Brian A. Weiss,et al.  The impact of evaluation scenario development on the quantitative performance of speech translation systems prescribed by the SCORE framework , 2009, PerMIS.

[30]  Brian A. Weiss,et al.  Performance Assessments of Two-Way, Free-Form, Speech-to-Speech Translation Systems for Tactical Use , 2011 .

[31]  Jean Scholtz,et al.  Development of a test bed for evaluating human-robot performance for explosive ordnance disposal robots , 2006, HRI '06.

[32]  Jean Scholtz,et al.  A Framework for Evaluating Collaborative Systems in the Real World , 2005, Proceedings of the 38th Annual Hawaii International Conference on System Sciences.

[33]  Chen Pei-you,et al.  Negotiation model based on uncertainty multi-attribute decision making , 2009, 2009 Chinese Control and Decision Conference.

[34]  Brian A. Weiss,et al.  Evaluating speech translation systems: applying SCORE to TRANSTAC technologies , 2009, PerMIS.

[35]  A. I. Ölçer,et al.  A new fuzzy multiple attributive group decision making methodology and its application to propulsion/manoeuvring system selection problem , 2005, Eur. J. Oper. Res..

[36]  Jean Scholtz,et al.  Metrics and Methodologies for Evaluating Technologies for Intelligence Analysts | NIST , 2006 .

[37]  Manfred Tscheligi,et al.  The USUS Evaluation Framework for Human-Robot Interaction , 2009 .

[38]  Jianwei Zhang,et al.  A Novel Reconfigurable Robot for Urban Search and Rescue , 2006 .

[39]  Brian A. Weiss,et al.  Evolution of the SCORE framework to enhance field-based performance evaluations of emerging technologies , 2008, PerMIS.

[40]  Stephen F. Conley.  Test and Evaluation Strategies for Network-Enabled Systems , 2009 .

[41]  Zeshui Xu,et al.  Multiple-Attribute Group Decision Making With Different Formats of Preference Information on Attributes , 2007, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[42]  Brian A. Weiss,et al.  The multi-relationship evaluation design framework: creating evaluation blueprints to assess advanced and intelligent technologies , 2010, PerMIS.

[43]  Adam Jacoff,et al.  Urban search and rescue robot performance standards: progress update , 2007, SPIE Defense + Commercial Sensing.

[44]  Michael Sipser,et al.  Introduction to the Theory of Computation , 1996, SIGA.

[45]  Jie Lu,et al.  Decider: A fuzzy multi-criteria group decision support system , 2010, Knowl. Based Syst..

[46]  Claude Hillinger,et al.  Voting and the Cardinal Aggregation of Judgments , 2004 .

[47]  Adam Jacoff,et al.  A Standard Test Course for Urban Search and Rescue Robots , 2000 .

[48]  Leonard R. Sussman,et al.  Nominal, Ordinal, Interval, and Ratio Typologies are Misleading , 1993 .

[49]  Gaurav S. Sukhatme,et al.  An Evaluation Methodology for Autonomous Mobile Robots for Planetary Exploration , 1995 .

[50]  Jean Scholtz,et al.  Development of an Evaluation Method for Acceptable Usability , 2007 .

[51]  Brian A. Weiss,et al.  Technology Evaluations and Performance Metrics for Soldier-Worn Sensors for ASSIST | NIST , 2007 .

[52]  Yang Ru-qing,et al.  Research on Semi-Automatic Bomb Fetching for an EOD Robot , 2007 .

[53]  Ying Zhang,et al.  A Platform for Studying Locomotion Systems: Modular Reconfigurable Robots , 2002 .

[54]  Robert M. O'Brien,et al.  The Use of Pearson's with Ordinal Data , 1979 .

[55]  Dennis K. Leedom.  Advancing the State-of-the-Art in Intelligent Systems: Scientific Rigor in Our Methods of Experimentation , 2003 .

[56]  Miles Thompson,et al.  Testing the Intelligence of Unmanned Autonomous Systems , 2008 .

[57]  B. Wright,et al.  Observations are always ordinal; measurements, however, must be interval. , 1989, Archives of physical medicine and rehabilitation.

[58]  Jean C. Scholtz,et al.  Evaluation Methods for Human-System Performance of Intelligent Systems , 2002 .

[59]  Abideen Tetlay,et al.  Determining the Lines of System Maturity, System Readiness and Capability Readiness in the System Development Lifecycle. , 2009 .

[60]  Brian A. Weiss,et al.  Overview of the First Advanced Technology Evaluations for ASSIST | NIST , 2006 .

[61]  Adam Jacoff,et al.  Stepfield pallets: repeatable terrain for evaluating robot mobility , 2008, PerMIS.

[62]  Jon A. Krosnick,et al.  The Measurement of Values in Surveys: A Comparison of Ratings and Rankings , 1985 .

[63]  Catherine A. Remley,et al.  Standards Development for Wireless Communications for Urban Search and Rescue Robots | NIST , 2007 .

[64]  J. Geoffrey Chase,et al.  Human-Robot Collaboration: A Literature Review and Augmented Reality Approach in Design , 2008 .

[65]  Clifford A. Whitcomb,et al.  A prescriptive production-distribution approach for decision making in new product design , 1999, IEEE Trans. Syst. Man Cybern. Part C.

[66]  Deborah L Thurston,et al.  Real and Misconceived Limitations to Decision Based Design With Utility Analysis , 2001 .

[67]  R. Ramanathan,et al.  Group preference aggregation methods employed in AHP: An evaluation and an intrinsic process for deriving members' weightages , 1994 .

[68]  S S Stevens,et al.  On the Theory of Scales of Measurement. , 1946, Science.

[69]  Kathleen Richardson.  Robots to the rescue , 2011 .

[70]  Dan R. Olsen,et al.  Metrics for Evaluating Human-Robot Interactions , 2003 .

[71]  Marc Steinberg,et al.  Human system performance metrics for evaluation of mixed-initiative heterogeneous autonomous systems , 2007 .

[72]  Illah R. Nourbakhsh,et al.  Human-robot teaming for search and rescue , 2005, IEEE Pervasive Computing.

[73]  B. V. Praag,et al.  Ordinal and cardinal utility , 1991 .

[74]  Brian A. Weiss,et al.  Performance Evaluation of Speech Translation Systems , 2008, LREC.

[75]  Sergei Vasiljev.  1 A Cardinal Voting : the Way to Escape the Social Choice Impossibility , 2005 .

[76]  Jean Scholtz,et al.  Evaluation of Human-Robot Interaction in the NIST Reference Search and Rescue Test Arenas | NIST , 2004 .

[77]  Fiorenzo Franceschini,et al.  The conceptual link between measurements, evaluations, preferences and indicators, according to the representational theory , 2007, Eur. J. Oper. Res..

[78]  Tom Frost,et al.  Derived Performance Metrics and Measurements Compared to Field Experience for the PackBot , 2002 .

[79]  Brian A. Weiss,et al.  Applying SCORE to field‐based performance evaluations of soldier worn sensor technologies , 2007, J. Field Robotics.

[80]  Brian A. Weiss,et al.  Lessons learned in evaluating DARPA advanced military technologies , 2010, PerMIS.

[81]  Kenneth J. Arrow,et al.  Extended sympathy and the possibility of social choice , 1978 .

[82]  Brian A. Weiss,et al.  Test arenas and performance metrics for urban search and rescue robots , 2003, Proceedings 2003 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2003) (Cat. No.03CH37453).

[83]  Ferenc Szidarovszky,et al.  Multi-attribute decision making: dominance with respect to an importance order of the attributes , 1993 .

[84]  Kerstin Dautenhahn,et al.  Methodology & Themes of Human-Robot Interaction: A Growing Research Field , 2007 .

[85]  Raj Madhavan,et al.  Performance analysis of unmanned vehicle positioning and obstacle mapping , 2006, SPIE Defense + Commercial Sensing.

[86]  Kevin Forsberg,et al.  Visualizing Project Management , 1996 .

[87]  Bilal M. Ayyub,et al.  Elicitation of expert opinions for uncertainty and risks: Answer to the Book Review by Roger M. Cooke , 2003, Fuzzy Sets Syst..

[88]  Donald G. Saari,et al.  The Borda dictionary , 1990 .

[89]  Michael Dummett,et al.  The Borda count and agenda manipulation , 1998 .

[90]  Adam Jacoff,et al.  RoboCup 2004: Rescue Robot League , 2005 .

[91]  Brian A. Weiss,et al.  Development of Domain-Specific Scenarios for Training and Evaluation of Two-Way, Free Form, Spoken Language Translation Devices , 2011 .

[92]  James S. Albus,et al.  Metrics and Performance Measures for Intelligent Unmanned Ground Vehicles , 2002 .

[93]  Holly A. Yanco.  Designing metrics for comparing the performance of robotic systems in robot competitions , 2002 .

[94]  C. Dym,et al.  Rank ordering engineering designs: pairwise comparison charts and Borda counts , 2002 .

[95]  Robert F. Erlandson,et al.  System Evaluation Methodologies: Combined Multidimensional Scaling and Ordering Techniques , 1978, IEEE Transactions on Systems, Man, and Cybernetics.

[96]  Dan S. Felsenthal,et al.  The Majority Judgement Voting Procedure : A Critical Evaluation 1 , 2009 .

[97]  Kotaro Suzumura,et al.  Introduction, Handbook of Social Choice and Welfare, Edited by Kenneth Arrow, Amartya Sen and Kotaro Suzumura, Amsterdam: Elsevier/North-Holland , 2001 .

[98]  Luca Iocchi,et al.  A unified benchmark framework for autonomous Mobile robots and Vehicles Motion Algorithms (MoVeMA benchmarks) , 2008 .

[99]  Adam Jacoff,et al.  Quantitative Assessment of Robot-Generated Maps , 2009 .

[100]  Karl Murphy,et al.  Autonomous Mobility for the Demo III Experimental Unmanned Vehicles , 2002 .

[101]  James S. Albus,et al.  Collaborative tactical behaviors for autonomous ground and air vehicles , 2005, SPIE Defense + Commercial Sensing.

[102]  Adam Jacoff,et al.  DHS/NIST Response Robot Evaluation Exercises | NIST , 2007 .

[103]  Quan Zhang,et al.  Multiple Attribute Decision Making Based on Fuzzy Selected Subset and Linguistic Variables , 2009, 2009 International Conference on Research Challenges in Computer Science.

[104]  Brian A. Weiss,et al.  Multi-Relationship Evaluation Design: Formalizing Evaluation-Design Input and Output Blueprint Elements for Testing Developing Intelligent Systems | NIST , 2011 .

[105]  Robert L. Wade,et al.  Robotic systems technical and operational metrics correlation , 2008, PerMIS.