Assessment of the State-of-the-Art of System-Wide Safety and Assurance Technologies

Since its initiation, the System-wide Safety & Assurance Technologies (SSAT) Project has been focused on developing multidisciplinary tools and techniques that are verified and validated to ensure prevention of loss of property and life in NextGen and enable proactive risk management through predictive methods. To this end, four technical challenges have been listed to help realize the goals of SSAT, namely (i) assurance of flight critical systems, (ii) discovery of precursors to safety incidents, (iii) assuring safe human-systems integration, and (iv) prognostic algorithm design for safety assurance. The objective of this report is to provide an extensive survey of SSAT-related research accomplishments by researchers within and outside NASA to get an understanding of what the state-of-the-art is for technologies enabling each of the four technical challenges. We hope that this report will serve as a good resource for anyone interested in gaining an understanding of the SSAT technical challenges, and also be useful in the future for project planning and resource allocation for related research. NASA/TM—2017-219515 1 Assessment of the State-of-the-Art of System-Wide Safety and Assurance Technologies Indranil Roychoudhury Stinger Ghaffarian Technologies, Inc. Moffett Field, California 94035 Mary S. Reveley National Aeronautics and Space Administration Glenn Research Center Cleveland, Ohio 44135 Nipa Phojanamongkolkij National Aeronautics and Space Administration Langley Research Center Hampton, Virginia 23681 Karen M. Leone Vantage Partners, LLC Brook Park, Ohio 44142

[1]  Karen M. Feigh,et al.  Modeling the Work of Humans and Automation in Complex Operations , 2013 .

[2]  Karen M. Feigh,et al.  Using neural networks to assess human-automation interaction , 2011, 2011 IEEE/AIAA 30th Digital Avionics Systems Conference.

[3]  Jean-Louis Boulanger Static Analysis of Software: The Abstract Interpretation , 2011 .

[4]  Sridhar Ramaswamy,et al.  Efficient algorithms for mining outliers from large data sets , 2000, SIGMOD '00.

[5]  Robert Murphey,et al.  A Control Architecture for Human-Machine Interaction in the Presence of Unreliable Automation and Operator Cognitive Limitations ∗ , 2012 .

[6]  Vijay Manikandan Janakiraman,et al.  Identifying Precursors to Anomalies Using Inverse Reinforcement Learning ∗ , 2014 .

[7]  Christine M. Mitchell,et al.  A Discrete Control Model of Operator Function: A Methodology for Information Display Design , 1986, IEEE Transactions on Systems, Man, and Cybernetics.

[8]  Liang Tang,et al.  Simulation-based Design and Validation of Automated Contingency Management for Propulsion Systems , 2007, 2007 IEEE Aerospace Conference.

[9]  Lawrence Z. Markosian,et al.  Towards Certification of a Space System Application of Fault Detection and Isolation , 2008 .

[10]  Michael I. Jordan,et al.  Multiple kernel learning, conic duality, and the SMO algorithm , 2004, ICML.

[11]  Alastair K. Cooke,et al.  Review of pilot models used in aircraft flight dynamics , 2014 .

[12]  Kai Goebel,et al.  Distilling the Verification Process for Prognostics Algorithms , 2013 .

[13]  George J. Vachtsevanos,et al.  Impact of Input Uncertainty on Failure Prognostic Algorithms: Extending the Remaining Useful Life of Nonlinear Systems , 2010 .

[14]  Karen M. Feigh,et al.  Pilot mental models and loss of control , 2014 .

[15]  Jean Souyris,et al.  Astrée: From Research to Industry , 2007, SAS.

[16]  Sankalita Saha,et al.  Requirements Specifications for Prognostics: An Overview , 2010 .

[17]  Kenneth L. McMillan,et al.  A methodology for hardware verification using compositional model checking , 2000, Sci. Comput. Program..

[18]  Jim Woodcock,et al.  Industrial Practice in Formal Methods: A Review , 2009, FM.

[19]  T. Yoneyama,et al.  Prognostics performance metrics and their relation to requirements, design, verification and cost-benefit , 2008, 2008 International Conference on Prognostics and Health Management.

[20]  Jan Kofron,et al.  Behavior Protocols Verification: Fighting State Explosion , 2004 .

[21]  Jan-Patrick Osterloh,et al.  Modeling approach to multi-agent system of human and machine agents: Application in design of early experiments for novel aeronautics systems , 2013, 2013 11th IEEE International Conference on Industrial Informatics (INDIN).

[22]  Xiaojin Zhu,et al.  Semi-Supervised Learning , 2010, Encyclopedia of Machine Learning.

[23]  S. Tremblay,et al.  Using near infrared spectroscopy and heart rate variability to detect mental overload , 2014, Behavioural Brain Research.

[24]  Sang-Hwan Kim,et al.  Multidimensional measure of display clutter and pilot performance for advanced head-up display. , 2011, Aviation, space, and environmental medicine.

[25]  Karen M. Feigh,et al.  Measuring Human-Automation Function Allocation , 2014 .

[26]  U. Rieder,et al.  Markov Decision Processes , 2010 .

[27]  Carl E. Rasmussen,et al.  Infinite Mixtures of Gaussian Process Experts , 2001, NIPS.

[28]  Kamalika Das,et al.  Block-GP: Scalable Gaussian Process Regression for Multimodal Data , 2010, 2010 IEEE International Conference on Data Mining.

[29]  Mickaël Baron,et al.  Formal Validation of HCI User Tasks , 2003, Software Engineering Research and Practice.

[30]  Michelle Yeh,et al.  Human factors considerations in the design and evaluation of flight deck displays and controls , 2013 .

[31]  Kenneth H. Goodrich,et al.  Towards an Improved Pilot-Vehicle Interface for Highly Automated Aircraft: Evaluation of the Haptic Flight Control System , 2012 .

[32]  Andreas Christmann,et al.  Support vector machines , 2008, Data Mining and Knowledge Discovery Handbook.

[33]  Wen-Chin Li,et al.  The evaluation of pilots performance and mental workload by eye movement , 2012 .

[34]  Daniel G. Morrow,et al.  Using neural networks to assess flight deck human-automation interaction , 2013, Reliab. Eng. Syst. Saf..

[35]  Brian F. Gore,et al.  A Methodical Approach for Developing Valid Human Performance Models of Flight Deck Operations , 2011, HCI.

[36]  Patrick Behm,et al.  MÉTÉOR : An Industrial Success in Formal Development , 1998, B.

[37]  Aoying Zhou,et al.  Density-Based Clustering over an Evolving Data Stream with Noise , 2006, SDM.

[38]  Daniel W. Repperger,et al.  Evaluation of Eye Metrics as a Detector of Fatigue , 2011, Hum. Factors.

[39]  Keith Warfield,et al.  Technology readiness levels , 2016 .

[40]  Kanishka Bhaduri,et al.  Algorithms for speeding up distance-based outlier detection , 2011, KDD.

[41]  Abhinav Saxena,et al.  Developing IVHM Requirements for Aerospace Systems , 2013 .

[42]  N. Iyer,et al.  Framework for post-prognostic decision support , 2006, 2006 IEEE Aerospace Conference.

[43]  George Vachtsevanos,et al.  Prognostics Enhanced Reconfigurable Control of Electro-Mechanical Actuators , 2009 .

[44]  Philippe A. Palanque,et al.  Formal modelling of incidents and accidents as a means for enriching training material for satellite control operations , 2008 .

[45]  Paul M. Anderson The Use and Limitations of Static-Analysis Tools to Improve Software Quality , 2008 .

[46]  David L. Iverson Inductive System Health Monitoring , 2004, IC-AI.

[47]  B. Saha,et al.  Designing Data-Driven Battery Prognostic Approaches for Variable Loading Profiles : Some Lessons Learned , 2012 .

[48]  Gerard J. Holzmann,et al.  The Model Checker SPIN , 1997, IEEE Trans. Software Eng..

[49]  Jeffrey S. Foster,et al.  A comparison of bug finding tools for Java , 2004, 15th International Symposium on Software Reliability Engineering.

[50]  David A. Landgrebe,et al.  A survey of decision tree classifier methodology , 1991, IEEE Trans. Syst. Man Cybern..

[51]  Takashi Yoneyama,et al.  Model Predictive Control using Prognosis and Health Monitoring of actuators , 2010, 2010 IEEE International Symposium on Industrial Electronics.

[52]  Jaideep Srivastava,et al.  Detection of Precursors to Aviation Safety Incidents Due to Human Factors , 2013, 2013 IEEE 13th International Conference on Data Mining Workshops.

[53]  Ellen J. Bass,et al.  A Systematic Approach to Model Checking Human–Automation Interaction Using Task Analytic Models , 2011, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[54]  Qun Wang,et al.  An experimental analysis of situation awareness for cockpit display interface evaluation based on flight simulation , 2013 .

[55]  Barry Kirwan,et al.  A Guide To Task Analysis: The Task Analysis Working Group , 1992 .

[56]  Touch screen technology in flight deck, how far is it helpful? , 2012, 2012 IEEE/AIAA 31st Digital Avionics Systems Conference (DASC).

[57]  G. Borghini,et al.  Neuroscience and Biobehavioral Reviews , 2022 .

[58]  Jeffery A. Schroeder,et al.  An Evaluation of Several Stall Models for Commercial Transport Training , 2014 .

[59]  Wen-Chin Li,et al.  The Investigation of Visual Attention and Workload by Experts and Novices in the Cockpit , 2013, HCI.

[60]  Thomas K. Ferris,et al.  “Cognitive Efficiency” in Display Media: A First Investigation of Basic Signal Dimensions , 2012 .

[61]  R. Conrad Rorie,et al.  A Preliminary Investigation of Training Order for Introducing NextGen Tools , 2011, HCI.

[62]  Brian F. Gore,et al.  Evaluating NextGen Closely Spaced Parallel Operations Concepts with Validated Human Performance Models: Scenario Development and Results , 2013 .

[63]  Virginie Wiels,et al.  Formal Verification of Avionics Software Products , 2009, FM.

[64]  VARUN CHANDOLA,et al.  Anomaly detection: A survey , 2009, CSUR.

[65]  Yannick Moy,et al.  Testing or Formal Verification: DO-178C Alternatives and Industrial Experience , 2013, IEEE Software.

[66]  George Vachtsevanos,et al.  Adaptive Load-Allocation for Prognosis-Based Risk Management , 2011 .

[67]  William Perrizo,et al.  RDF: a density-based outlier detection method using vertical data representation , 2004, Fourth IEEE International Conference on Data Mining (ICDM'04).

[68]  Rebecca R. Goldstein,et al.  Searching in clutter: Visual attention strategies of expert pilots , 2012 .

[69]  Jeannette M. Wing A specifier's introduction to formal methods , 1990, Computer.

[70]  Philippe A. Palanque,et al.  Validating interactive system design through the verification of formal task and system models , 1995, EHCI.

[71]  Wei Lin,et al.  Towards Requirements in Systems Engineering for Aerospace IVHM Design , 2013 .

[72]  Philippe A. Palanque,et al.  Formal socio-technical barrier modelling for safety-critical interactive systems design , 2007 .

[73]  Xuefei Guan,et al.  Comparison of Two Probabilistic Fatigue Damage Assessment Approaches Using Prognostic Performance Metrics , 2011, International Journal of Prognostics and Health Management.

[74]  Mario G. Perhinschi,et al.  Fuzzy logic‐based detection scheme for pilot fatigue , 2010 .

[75]  Michael C. Dorneich,et al.  Analysis of the Risks and Benefits of Flight Deck Adaptive Systems , 2012 .

[76]  Kanishka Bhaduri,et al.  Fast and Flexible Multivariate Time Series Subsequence Search , 2010, 2010 IEEE International Conference on Data Mining.

[77]  Liang Tang,et al.  Prognostics in the Control Loop , 2007, AAAI Fall Symposium: Artificial Intelligence for Prognostics.

[78]  Fabio Paternò,et al.  Formal Models for Cooperative Tasks: Concepts and an Application for En-Route Air-Traffic Control , 1998, DSV-IS.

[79]  Richard W. Buskens,et al.  Verification of Adaptive Systems , 2012, Infotech@Aerospace.

[80]  J. Alonso,et al.  A Modeling Framework for Prognostic Decision Making and its Application to UAV Mission Planning , 2013 .

[81]  Stephen D. Bay,et al.  Mining distance-based outliers in near linear time with randomization and a simple pruning rule , 2003, KDD '03.

[82]  Michael C. Dorneich,et al.  Innovative flight deck function allocation concepts for NextGen , 2012 .

[83]  Stephan Merz,et al.  Model Checking , 2000 .

[84]  Matthew G. Karlaftis,et al.  An Analysis of Controller-Pilot Miscommunication in the NextGen Environment , 2011 .

[85]  Edward Balaban,et al.  An Approach to Prognostic Decision Making in the Aerospace Domain , 2012 .

[86]  Stephen Grossberg,et al.  Fuzzy ARTMAP: A neural network architecture for incremental supervised learning of analog multidimensional maps , 1992, IEEE Trans. Neural Networks.

[87]  Antonio C. Siochi,et al.  The UAN: a user-oriented representation for direct manipulation interface designs , 1990, TOIS.

[88]  Ö. Eker,et al.  Major challenges in prognostics: study on benchmarking prognostic datasets , 2012 .

[89]  Mickaël Baron,et al.  Formal and experimental validation approaches in HCI systems design based on a shared event B model , 2006, International Journal on Software Tools for Technology Transfer.

[90]  Kanishka Bhaduri,et al.  ν-Anomica: A Fast Support Vector Based Novelty Detection Technique , 2009, 2009 Ninth IEEE International Conference on Data Mining.

[91]  Robert E. Fields,et al.  Analysis of erroneous actions in the design of critical systems , 2001 .

[92]  Yamira Santiago-Espada,et al.  Predicting Operator Execution Times Using CogTool , 2013 .

[93]  Kanishka Bhaduri,et al.  Discovering Anomalous Aviation Safety Events Using Scalable Data Mining Algorithms , 2013, J. Aerosp. Inf. Syst..

[94]  Ashok N. Srivastava,et al.  Anomaly Detection and Diagnosis Algorithms for Discrete Symbol Sequences with Applications to Airline Safety , 2009, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[95]  T. Shane Sowers,et al.  Real-Time Simulation for Verification and Validation of Diagnostic and Prognostic Algorithms , 2005 .

[96]  W. D. Ledegang,et al.  Pilot performance in centrifuge-based simulation of unusual attitude recovery , 2012 .

[97]  D.R. Wallace,et al.  Software verification and validation: an overview , 1989, IEEE Software.

[98]  Michael Pecht,et al.  Using real options to manage condition-based maintenance enabled by PHM , 2011, 2011 IEEE Conference on Prognostics and Health Management.

[99]  Brian F. Gore The Use of Behavior Models for Predicting Complex Operations , 2010 .

[100]  Lian Duan,et al.  A Local Density Based Spatial Clustering Algorithm with Noise , 2006, 2006 IEEE International Conference on Systems, Man and Cybernetics.

[101]  Edmund M. Clarke,et al.  Formal Methods: State of the Art and Future Directions Working Group Members , 1996 .

[102]  Ronald A. Hess,et al.  Visual Perception and Manual Control , 2011 .

[103]  Fabio Paternò,et al.  Integrating Model Checking and HCI Tools to Help Designers Verify User Interface Properties , 2000, DSV-IS.

[104]  Ashok N. Srivastava,et al.  Multiple kernel learning for heterogeneous anomaly detection: algorithm and aviation safety case study , 2010, KDD.

[105]  Sankalita Saha,et al.  Metrics for Offline Evaluation of Prognostic Performance , 2021, International Journal of Prognostics and Health Management.

[106]  Stephen Whitlow,et al.  The Crew Workload Manager , 2011 .

[107]  Takafumi Kanamori,et al.  Inlier-Based Outlier Detection via Direct Density Ratio Estimation , 2008, 2008 Eighth IEEE International Conference on Data Mining.

[108]  Stephen M. Casner,et al.  The Effectiveness of Airline Pilot Training for Abnormal Events , 2013, Hum. Factors.

[109]  Santanu Das,et al.  Sparse Solutions for Single Class SVMs: A Bi-Criterion Approach , 2011, SDM.

[110]  Fabio Paternò,et al.  ConcurTaskTrees: A Diagrammatic Notation for Specifying Task Models , 1997, INTERACT.

[111]  David A. Mindell,et al.  An Ethnographic Approach to Human-Machine Relationships in Commercial Aviation: Heads-Up Guidance and Enhanced Vision , 2011 .

[112]  Jonathan W Schooler,et al.  Thoughts in Flight , 2014, Hum. Factors.

[113]  Divya C Chandra,et al.  Evaluation of a Technique to Simplify Depictions of Visually Complex Aeronautical Procedures for NextGen , 2013 .

[114]  Elisabeth A. Strunk,et al.  A Survey of Tools for Model Checking and Model-Based Development , 2006 .

[115]  Hoyt Lougee,et al.  SOFTWARE CONSIDERATIONS IN AIRBORNE SYSTEMS AND EQUIPMENT CERTIFICATION , 2001 .

[116]  Paul S. Miner,et al.  Baseline Assessment and Prioritization Framework for Ivhm Integrity Assurance Enabling Capabilities , 2013 .

[117]  Carl A. Gunter,et al.  Specifying and Analyzing Workflows for Automated Identification and Data Capture , 2009, 2009 42nd Hawaii International Conference on System Sciences.

[118]  Albert Y. Zomaya,et al.  SVM-enabled prognostic method for clinical decision making: The use of CD4 T-cell level and HIV-1 viral load for guiding treatment initiation and alteration , 2010, 2010 IEEE 23rd International Symposium on Computer-Based Medical Systems (CBMS).

[119]  Sang-Hwan Kim,et al.  Influence of Flight Domain and Cockpit Display Dynamics on Pilot Perceived Clutter , 2013, J. Aerosp. Inf. Syst..

[120]  Aarti Gupta,et al.  Formal hardware verification methods: A survey , 1992, Formal Methods Syst. Des..