Run Time Assurance for Safety-Critical Systems: An Introduction to Safety Filtering Approaches for Complex Control Systems

More than three miles above the Arizona desert, an F-16 student pilot experienced a gravityinduced loss of consciousness (GLOC), passing out while turning at nearly 9Gs (nine times the force of gravity) flying over 400 knots (over 460 miles per hour). With its pilot unconscious, the aircraft turn devolved into a dive, dropping from over 17,000 feet to less than 8,000 feet in altitude in less than 10 seconds. An auditory warning in the cockpit called out to the pilot “altitude, altitude” just before he crossed through 11,000 feet, switching to a command to “pull up” around 8,000 feet. Meanwhile, the student’s instructor was watching the event unfold from his own aircraft. As the student’s aircraft passed through 12,500 feet, the instructor called over the radio “two recover,” commanding the student (“two”) to end the dive. As the student’s aircraft passed through 11,000 feet the instructor’s “two recover!” came with increased urgency. At 9,000 feet, and with terror rising in his voice the instructor yelled “TWO RECOVER!” Fortunately, at the same time as the instructor’s third panicked radio call, a new Run Time Assurance (RTA) system kicked in to automatically recover the aircraft. The Automatic Ground Collision Avoidance System (Auto GCAS), an RTA system integrated on the jets less than two years earlier in the Fall of 2014, detected that the aircraft was about to collide, commanded a roll to wings level and pull up maneuver, and recovered the aircraft less than 3,000 feet above the

[1]  Paulo Tabuada,et al.  Control Barrier Functions: Theory and Applications , 2019, 2019 18th European Control Conference (ECC).

[2]  Ufuk Topcu,et al.  Safe Reinforcement Learning via Shielding , 2017, AAAI.

[3]  Richard Lehmann,et al.  Automatic Aircraft Collision Avoidance Algorithm Design for Fighter Aircraft , 2012 .

[4]  Mark A. Skoog,et al.  Leveraging ASTM Industry Standard F3269-17 for Providing Safe Operations of a Highly Autonomous Aircraft , 2020, 2020 IEEE Aerospace Conference.

[5]  Lauretta O. Osho,et al.  Axiomatic Basis for Computer Programming , 2013 .

[6]  John D. Schierman,et al.  Runtime Assurance Framework Development for Highly Adaptive Flight Control Systems , 2015 .

[7]  David Angeli,et al.  Monotone control systems , 2003, IEEE Trans. Autom. Control..

[8]  Alexandre M. Bayen,et al.  A time-dependent Hamilton-Jacobi formulation of reachable sets for continuous dynamic games , 2005, IEEE Transactions on Automatic Control.

[9]  Samuel Coogan,et al.  Safety from in-the-loop reachability for cyber-physical systems , 2021, CAADCPS@CPSIoTWeek.

[10]  Nancy A. Lynch,et al.  The Theory of Timed I/o Automata , 2003 .

[11]  Sayan Mitra,et al.  A verification framework for hybrid systems , 2007 .

[12]  John D. Lee,et al.  Trust in Automation: Designing for Appropriate Reliance , 2004, Hum. Factors.

[13]  Rajeev Alur,et al.  Principles of Cyber-Physical Systems , 2015 .

[14]  E. D. Sontagc,et al.  Nonmonotone systems decomposable into monotone systems with negative feedback , 2005 .

[15]  Lui Sha,et al.  Analytic Redundancy for Software Fault-Tolerance In Hard Real-Time Systems , 1994 .

[16]  R. Sanfelice,et al.  Hybrid dynamical systems , 2009, IEEE Control Systems.

[17]  Weiming Xiang,et al.  NNV: The Neural Network Verification Tool for Deep Neural Networks and Learning-Enabled Cyber-Physical Systems , 2020, CAV.

[18]  Nathan Fulton,et al.  Safe Reinforcement Learning via Formal Methods: Toward Safe Control Through Proof and Learning , 2018, AAAI.

[19]  Ufuk Topcu,et al.  Shield synthesis , 2017, Formal Methods Syst. Des..

[20]  Koushil Sreenath,et al.  Feedback Control of an Exoskeleton for Paraplegics: Toward Robustly Stable, Hands-Free Dynamic Walking , 2018, IEEE Control Systems.

[21]  Guang Yang,et al.  Continuous-time Signal Temporal Logic Planning with Control Barrier Functions , 2019, 2020 American Control Conference (ACC).

[22]  Nancy G. Leveson,et al.  Engineering a Safer World: Systems Thinking Applied to Safety , 2012 .

[23]  Paulo Tabuada,et al.  Verification and Control of Hybrid Systems - A Symbolic Approach , 2009 .

[24]  Christel Baier,et al.  Principles of model checking , 2008 .

[25]  Hoyt Lougee,et al.  SOFTWARE CONSIDERATIONS IN AIRBORNE SYSTEMS AND EQUIPMENT CERTIFICATION , 2001 .

[26]  Eric Feron,et al.  Formal Verification of System States for Spacecraft Automatic Maneuvering , 2019 .

[27]  Michael W. Whalen,et al.  Evaluation of formal methods tools applied to a 6U CubeSat attitude control system , 2015 .

[28]  Lui Sha,et al.  An Architectural Description of the Simplex Architecture. , 1996 .

[29]  M. Valenti,et al.  Implementation and Flight Test Results of MILP-based UAV Guidance , 2005, 2005 IEEE Aerospace Conference.

[30]  Stanley Bak,et al.  Efficient n-to-n Collision Detection for Space Debris using 4D AABB Trees (Extended Report) , 2019, ARCH@CPSIoTWeek.

[31]  Owain Evans,et al.  Trial without Error: Towards Safe Reinforcement Learning via Human Intervention , 2017, AAMAS.

[32]  Thomas Gurriet Applied Safety Critical Control , 2020 .

[33]  Magnus Egerstedt,et al.  Safety With Limited Range Sensing Constraints For Fixed Wing Aircraft , 2020, ArXiv.

[34]  Zeroing Control Barrier Functions for Safe Volitional Pedaling in a Motorized Cycle , 2020 .

[35]  Thomas A. Henzinger,et al.  The Algorithmic Analysis of Hybrid Systems , 1995, Theor. Comput. Sci..

[36]  H L Smith,et al.  The Discrete Dynamics of Monotonically Decomposable Maps , 2006, Journal of mathematical biology.

[37]  Chris Manzie,et al.  Control Barrier Functions for Mechanical Systems: Theory and Application to Robotic Grasping , 2019, IEEE Transactions on Control Systems Technology.

[38]  Xiaodong Zhang,et al.  Nonlinear Adaptive Control of Quadrotor UAVs with Run-Time Safety Assurance , 2017 .

[39]  Petter Nilsson,et al.  A Scalable Safety Critical Control Framework for Nonlinear Systems , 2020, IEEE Access.

[40]  Jyh-Ching Juang,et al.  Active collision avoidance system for steering control of autonomous vehicles , 2014 .

[41]  Sanjai Rayadurgam,et al.  Your "What" Is My "How": Iteration and Hierarchy in System Design , 2013, IEEE Software.

[42]  Stanley Bak,et al.  Hybrid Cyberphysical System Verification with Simplex Using Discrete Abstractions , 2010, 2010 16th IEEE Real-Time and Embedded Technology and Applications Symposium.

[43]  Murat Arcak,et al.  TIRA: toolbox for interval reachability analysis , 2019, HSCC.

[44]  Tom Schouwenaars,et al.  Safe Trajectory Planning of Autonomous Vehicles , 2006 .

[45]  Wojciech M. Czarnecki,et al.  Grandmaster level in StarCraft II using multi-agent reinforcement learning , 2019, Nature.

[46]  Taylor T. Johnson,et al.  Improved Geometric Path Enumeration for Verifying ReLU Neural Networks , 2020, CAV.

[47]  J. How,et al.  Receding horizon path planning with implicit safety guarantees , 2004, Proceedings of the 2004 American Control Conference.

[48]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[49]  Edmund M. Clarke,et al.  Design and Synthesis of Synchronization Skeletons Using Branching Time Temporal Logic , 2008, 25 Years of Model Checking.

[50]  V.A. Carreno,et al.  A case-study application of RTCA DO-254: design assurance guidance for airborne electronic hardware , 2000, 19th DASC. 19th Digital Avionics Systems Conference. Proceedings (Cat. No.00CH37126).

[51]  Alex A. Kurzhanskiy,et al.  Mixed monotonicity of partial first-in-first-out traffic flow models , 2016, 2016 IEEE 55th Conference on Decision and Control (CDC).

[52]  Claire J. Tomlin,et al.  Decentralized cooperative collision avoidance for acceleration constrained vehicles , 2008, 2008 47th IEEE Conference on Decision and Control.

[53]  S. Shankar Sastry,et al.  Conflict resolution for air traffic management: a study in multiagent hybrid systems , 1998, IEEE Trans. Autom. Control..

[54]  Donald W. Loveland,et al.  A machine program for theorem-proving , 2011, CACM.

[55]  Eric Feron,et al.  Monitor-Based Runtime Assurance for Temporal Logic Specifications , 2019, 2019 IEEE 58th Conference on Decision and Control (CDC).

[56]  Mahesh Viswanathan,et al.  Runtime Assurance Based On Formal Specifications , 1999, PDPTA.

[57]  Lui Sha,et al.  The System-Level Simplex Architecture for Improved Real-Time Embedded System Safety , 2009, 2009 15th IEEE Real-Time and Embedded Technology and Applications Symposium.

[58]  Yanjun Huang,et al.  Lane Keeping Control of Autonomous Vehicles With Prescribed Performance Considering the Rollover Prevention and Input Saturation , 2020, IEEE Transactions on Intelligent Transportation Systems.

[59]  Angela W Suplisson,et al.  Optimal Recovery Trajectories for Automatic Ground Collision Avoidance Systems (Auto GCAS) , 2015 .

[60]  Demis Hassabis,et al.  Mastering the game of Go without human knowledge , 2017, Nature.

[61]  Gábor Orosz,et al.  End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Tasks , 2019, AAAI.

[62]  Javier García,et al.  A comprehensive survey on safe reinforcement learning , 2015, J. Mach. Learn. Res..

[63]  Magnus Egerstedt,et al.  Nonsmooth Barrier Functions With Applications to Multi-Robot Systems , 2017, IEEE Control Systems Letters.

[64]  Lui Sha,et al.  Using Simplicity to Control Complexity , 2001, IEEE Softw..

[65]  Sergey Levine,et al.  How to train your robot with deep reinforcement learning: lessons we have learned , 2021, Int. J. Robotics Res..

[66]  Joseph B. Lyons,et al.  Trust of a Military Automated System in an Operational Context , 2017 .

[67]  Donald E. Swihart,et al.  Design, integration and flight test of an autonomous ground collision avoidance system , 2011 .

[68]  James D. Carpenter Simulation and Piloted Simulator Study of an Automatic Ground Collision Avoidance System for Performance Limited Aircraft , 2019 .

[69]  Robert E. Voros Small Airplane Considerations for the Guidelines for Development of Civil Aircraft and Systems , 2013 .

[70]  Florian-Michael Adolf,et al.  Formal Requirements and Model-Checking for V&V Automation of a RPAS Mission Management System , 2015 .

[71]  Ron Koymans,et al.  Specifying real-time properties with metric temporal logic , 1990, Real-Time Systems.

[72]  Andrew G. Barto,et al.  Lyapunov Design for Safe Reinforcement Learning , 2003, J. Mach. Learn. Res..

[73]  Koushil Sreenath,et al.  Discrete Control Barrier Functions for Safety-Critical Control of Discrete Systems with Application to Bipedal Robot Navigation , 2017, Robotics: Science and Systems.

[74]  D E Swihart,et al.  Automatic Ground Collision Avoidance System design, integration, & flight test , 2011, IEEE Aerospace and Electronic Systems Magazine.

[75]  Koushil Sreenath,et al.  3D dynamic walking on stepping stones with control barrier functions , 2016, 2016 IEEE 55th Conference on Decision and Control (CDC).

[76]  John D. Schierman,et al.  Runtime Assurance Protection for Advanced Turbofan Engine Control , 2018 .

[77]  John D. Schierman,et al.  Run-Time Assurance for Advanced Flight-Critical Control Systems * , 2010 .

[78]  David Q. Mayne,et al.  An apologia for stabilising terminal conditions in model predictive control , 2013, Int. J. Control.

[79]  Ian M. Mitchell A Summary of Recent Progress on Efficient Parametric Approximations of Viability and Discriminating Kernels , 2015, SNR@CAV.

[80]  Eric Feron,et al.  Verification and runtime assurance for dynamical systems with uncertainty , 2021, HSCC.

[81]  Curt Hanson,et al.  Capability Description for NASA's F/A-18 TN 853 as a Testbed for the Integrated Resilient Aircraft Control Project , 2009 .

[82]  Claire J. Tomlin,et al.  Sampling-based approximation of the viability kernel for high-dimensional linear sampled-data systems , 2014, HSCC.

[83]  Paulo Tabuada,et al.  Control Barrier Function Based Quadratic Programs for Safety Critical Systems , 2016, IEEE Transactions on Automatic Control.

[84]  Paulo Tabuada,et al.  Correctness Guarantees for the Composition of Lane Keeping and Adaptive Cruise Control , 2016, IEEE Transactions on Automation Science and Engineering.

[85]  Aaron D. Ames,et al.  Backup Control Barrier Functions: Formulation and Comparative Study , 2021, 2021 60th IEEE Conference on Decision and Control (CDC).

[86]  Moshe Y. Vardi,et al.  Verification , 1917, Handbook of Automata Theory.

[87]  Samuel Coogan,et al.  Tight Decomposition Functions for Continuous-Time Mixed-Monotone Systems With Disturbances , 2021, IEEE Control Systems Letters.

[88]  Samuel Coogan,et al.  Computing Robustly Forward Invariant Sets for Mixed-Monotone Systems , 2020, 2020 59th IEEE Conference on Decision and Control (CDC).

[89]  Jonathan P. How,et al.  Aggressive 3-D collision avoidance for high-speed navigation , 2017, 2017 IEEE International Conference on Robotics and Automation (ICRA).

[90]  Thomas A. Henzinger,et al.  Assume-Guarantee Reasoning for Hierarchical Hybrid Systems , 2001, HSCC.

[91]  John C. Knight,et al.  Safety critical systems: challenges and directions , 2002, Proceedings of the 24th International Conference on Software Engineering. ICSE 2002.

[92]  Howard Barringer,et al.  Learning to divide and conquer: applying the L* algorithm to automate assume-guarantee reasoning , 2008, Formal Methods Syst. Des..

[93]  Eric Feron,et al.  Formal Specification and Analysis of Spacecraft Collision Avoidance Run Time Assurance Requirements , 2021, 2021 IEEE Aerospace Conference (50100).

[94]  W. H. Clohessy,et al.  Terminal Guidance System for Satellite Rendezvous , 2012 .

[95]  Kristin Yvonne Rozier,et al.  Specification: The Biggest Bottleneck in Formal Methods and Autonomy , 2016, VSTTE.

[96]  J. Doyle,et al.  Essentials of Robust Control , 1997 .

[97]  Radu Grosu,et al.  Collision avoidance for mobile robots with limited sensing and limited information about moving obstacles , 2017, Formal Methods Syst. Des..

[98]  Dimos V. Dimarogonas,et al.  Control Barrier Functions for Signal Temporal Logic Tasks , 2019, IEEE Control Systems Letters.

[99]  Aaron D. Ames,et al.  Towards Variable Assistance for Lower Body Exoskeletons , 2020, IEEE Robotics and Automation Letters.

[100]  Mark Skoog,et al.  Small UAV Automatic Ground Collision Avoidance System Design Considerations and Flight Test Results , 2015 .

[101]  Steve Cook Anna Dietrich Lloyd Hook Practice for Methods to Safely Bound Flight Behavior of Unmanned Aircraft Systems Containing Complex Functions , 2019 .

[102]  Saïd Mammar,et al.  Driver Steering Assistance for Lane-Departure Avoidance Based on Hybrid Automata and Composite Lyapunov Function , 2010, IEEE Transactions on Intelligent Transportation Systems.

[103]  Lui Sha,et al.  The Simplex architecture for safe online control system upgrades , 1998, Proceedings of the 1998 American Control Conference. ACC (IEEE Cat. No.98CH36207).

[104]  Wesley A Olson Airborne Collision Avoidance System X , 2015 .

[105]  Dimos V. Dimarogonas,et al.  High-Order Barrier Functions: Robustness, Safety, and Performance-Critical Control , 2021, IEEE Transactions on Automatic Control.

[106]  W. J. DeCoursey,et al.  Introduction: Probability and Statistics , 2003 .

[107]  Siddharth Mayya,et al.  The Robotarium: Automation of a Remotely Accessible, Multi-Robot Testbed , 2021, IEEE Robotics and Automation Letters.

[108]  Koushil Sreenath,et al.  Exponential Control Barrier Functions for enforcing high relative-degree safety-critical constraints , 2016, 2016 American Control Conference (ACC).

[109]  Siddharth Mayya,et al.  The Robotarium: Globally Impactful Opportunities, Challenges, and Lessons Learned in Remote-Access, Distributed Control of Multirobot Systems , 2020, IEEE Control Systems.

[110]  Aaron D. Ames,et al.  Towards Restoring Locomotion for Paraplegics: Realizing Dynamically Stable Walking on Exoskeletons , 2018, 2018 IEEE International Conference on Robotics and Automation (ICRA).

[111]  G. Mahr Validation , 2019, Academic Psychiatry.

[112]  Magnus Egerstedt,et al.  Constructive Barrier Certificates with Applications to Fixed-Wing Aircraft Collision Avoidance , 2018, 2018 IEEE Conference on Control Technology and Applications (CCTA).

[113]  Vijay Kumar,et al.  High speed navigation for quadrotors with limited onboard sensing , 2016, 2016 IEEE International Conference on Robotics and Automation (ICRA).

[114]  Ofir Nachum,et al.  A Lyapunov-based Approach to Safe Reinforcement Learning , 2018, NeurIPS.

[115]  Wes Ryan,et al.  Initial considerations of a multi-layered run time assurance approach to enable unpiloted aircraft , 2018, 2018 IEEE Aerospace Conference.

[116]  Patrick Cousot,et al.  Abstract Interpretation Frameworks , 1992, J. Log. Comput..

[117]  Samuel Coogan,et al.  A Barrier Function Approach to Finite-Time Stochastic System Verification and Control , 2019, Autom..

[118]  Clark W. Barrett,et al.  Simplifying Neural Networks Using Formal Verification , 2020, NFM.

[119]  J. Lygeros,et al.  A game theoretic approach to controller design for hybrid systems , 2000, Proceedings of the IEEE.

[120]  Petter Nilsson,et al.  Online Active Safety for Robotic Manipulators , 2019, 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[121]  James Brady,et al.  Certification strategies using run-time safety assurance for part 23 autopilot systems , 2016, 2016 IEEE Aerospace Conference.

[122]  John D. Schierman,et al.  Runtime Assurance for Autonomous Aerospace Systems , 2020 .

[123]  Magnus Egerstedt,et al.  Control of Multi-Agent Systems with Finite Time Control Barrier Certificates and Temporal Logic , 2018, 2018 IEEE Conference on Decision and Control (CDC).

[124]  Rocci G. Pisano,et al.  Burrows and Feeding of the Norway Rat , 1948 .

[125]  Masooda N. Bashir,et al.  Trust in Automation: Integrating Empirical Evidence on Factors That Influence Trust , 2015, Hum. Factors.

[126]  Kostas E. Bekris,et al.  Sim2Real in Robotics and Automation: Applications and Challenges , 2021, IEEE Trans Autom. Sci. Eng..

[127]  Claire J. Tomlin,et al.  Applications of hybrid reachability analysis to robotic aerial vehicles , 2011, Int. J. Robotics Res..

[128]  Ian M. Mitchell,et al.  Safety verification of conflict resolution manoeuvres , 2001, IEEE Trans. Intell. Transp. Syst..

[129]  Aaron D. Ames,et al.  A Scalable Controlled Set Invariance Framework with Practical Safety Guarantees , 2019, 2019 IEEE 58th Conference on Decision and Control (CDC).

[130]  Mitio Nagumo Über die Lage der Integralkurven gewöhnlicher Differentialgleichungen , 1942 .

[131]  Kerianne Hobbs,et al.  Elicitation and Formal Specification of Run Time Assurance Requirements for Aerospace Collision Avoidance Systems , 2020 .

[132]  Anitha Murugesan,et al.  Formal Methods Assisted Training of Safe Reinforcement Learning Agents , 2019, NFM.

[133]  Joseph B. Lyons,et al.  Trust of an Automatic Ground Collision Avoidance Technology: A Fighter Pilot Perspective , 2016 .

[134]  Maria Consiglio,et al.  DAIDALUS: Detect and avoid alerting logic for unmanned systems , 2015, 2015 IEEE/AIAA 34th Digital Avionics Systems Conference (DASC).

[135]  Kuldip S. Rattan,et al.  Formally Verified Run Time Assurance Architecture of a 6U CubeSat Attitude Control System , 2016 .

[136]  Ariadne: A common-sense thread for enabling provable safety in air mobility systems with unreliable components , 2022, AIAA SCITECH 2022 Forum.

[137]  Amy Chicatelli,et al.  Towards Run-time Assurance of Advanced Propulsion Algorithms , 2014 .

[138]  Calin Belta,et al.  Control Barrier Functions for Systems with High Relative Degree , 2019, 2019 IEEE 58th Conference on Decision and Control (CDC).

[139]  Stanley Bak,et al.  Space Debris Collision Detection using Reachability , 2018, ARCH@ADHS.

[140]  Mykel J. Kochenderfer,et al.  Reluplex: An Efficient SMT Solver for Verifying Deep Neural Networks , 2017, CAV.

[141]  Amy Burns,et al.  Auto GCAS for analog flight control system , 2011, 2011 IEEE/AIAA 30th Digital Avionics Systems Conference.

[142]  William R. Nichols,et al.  DoD Developer’s Guidebook for Software Assurance , 2020 .

[143]  Aaron D. Ames,et al.  Towards a Framework for Realizable Safety Critical Control through Active Set Invariance , 2018, 2018 ACM/IEEE 9th International Conference on Cyber-Physical Systems (ICCPS).

[144]  Li Wang,et al.  Safe, Remote-Access Swarm Robotics Research on the Robotarium , 2016, ArXiv.

[145]  Stephen Prajna Barrier certificates for nonlinear model validation , 2006, Autom..

[146]  Ali Jadbabaie,et al.  Safety Verification of Hybrid Systems Using Barrier Certificates , 2004, HSCC.

[147]  Donald E. Swihart,et al.  Automatic Ground Collision Avoidance System Design for Pre-Block 40 F-16 Configurations , 2012 .

[148]  Donald E. Swihart,et al.  Test and Evaluation of a Modified F-16 Analog Flight Control Computer , 2013 .

[149]  Koushil Sreenath,et al.  Safety-Critical Control for Dynamical Bipedal Walking with Precise Footstep Placement , 2015, ADHS.

[150]  I. Opie,et al.  The classic fairy tales , 1980 .

[151]  Petter Nilsson,et al.  Safety-Critical Rapid Aerial Exploration of Unknown Environments , 2020, 2020 IEEE International Conference on Robotics and Automation (ICRA).

[152]  Scott A. Smolka,et al.  Safe CPS from unsafe controllers , 2021, CAADCPS@CPSIoTWeek.

[153]  Lui Sha,et al.  A Software Architecture for Dependable and Evolvable Industrial Computing Systems. , 1995 .

[154]  MichaelWooldridge QueenMary andWestfieldCollege,et al.  AGENTS AND SOFTWARE ENGINEERING , 2002 .

[155]  Dejan Nickovic,et al.  Monitoring Temporal Properties of Continuous Signals , 2004, FORMATS/FTRTFT.

[156]  Murat Arcak,et al.  Efficient finite abstraction of mixed monotone systems , 2015, HSCC.

[157]  Ricardo G. Sanfelice,et al.  A Hybrid Control Strategy for Autonomous Navigation while Avoiding Multiple Obstacles at Unknown Locations , 2019, 2019 IEEE 15th International Conference on Automation Science and Engineering (CASE).

[158]  Li Wang,et al.  The Robotarium: A remotely accessible swarm robotics research testbed , 2016, 2017 IEEE International Conference on Robotics and Automation (ICRA).

[159]  Donald E. Swihart,et al.  Development of an Automatic Aircraft Collision Avoidance System for Fighter Aircraft , 2013 .

[160]  Murat Arcak,et al.  Stability of traffic flow networks with a polytree topology , 2016, Autom..

[161]  R. Sapolsky,et al.  Stress in the wild. , 1990, Scientific American.

[162]  Eduardo Valido-Cabrera Software reliability methods , 2006 .

[163]  Hanêne Ben-Abdallah,et al.  A Monitoring and Checking Framework for Run-time Correctness Assurance , 1998 .

[164]  Kerianne L. Hobbs,et al.  Natural Motion-based Trajectories for Automatic Spacecraft Collision Avoidance During Proximity Operations , 2021, 2021 IEEE Aerospace Conference (50100).

[165]  Demis Hassabis,et al.  Mastering the game of Go with deep neural networks and tree search , 2016, Nature.

[166]  Magnus Egerstedt,et al.  Safe certificate-based maneuvers for teams of quadrotors using differential flatness , 2017, 2017 IEEE International Conference on Robotics and Automation (ICRA).

[167]  Samuel Coogan,et al.  Enforcing Safety at Runtime for Systems with Disturbances , 2020, 2020 59th IEEE Conference on Decision and Control (CDC).

[168]  D. Wiegmann,et al.  Similarities and differences between human–human and human–automation trust: an integrative review , 2007 .

[169]  Aaron D. Ames,et al.  Guaranteed Obstacle Avoidance for Multi-Robot Operations With Limited Actuation: A Control Barrier Function Approach , 2021, IEEE Control Systems Letters.

[170]  Paulo Tabuada,et al.  Control barrier function based quadratic programs with application to adaptive cruise control , 2014, 53rd IEEE Conference on Decision and Control.

[171]  Paulo Tabuada,et al.  Robustness of Control Barrier Functions for Safety Critical Control , 2016, ADHS.

[172]  Ricardo G. Sanfelice,et al.  Multiple Barrier Function Certificates for Weak Forward Invariance in Hybrid Inclusions , 2019, 2019 IEEE 58th Conference on Decision and Control (CDC).

[173]  Magnus Egerstedt,et al.  Composition of Safety Constraints With Applications to Decentralized Fixed-Wing Collision Avoidance , 2019, ArXiv.

[174]  Marco Pavone,et al.  Collision-Inclusive Trajectory Optimization for Free-Flying Spacecraft , 2020 .

[175]  Insup Lee,et al.  A Study on Run Time Assurance for Complex Cyber Physical Systems , 2013 .

[176]  Kerianne H. Gross,et al.  Incremental Formal Methods Based Design Approach Demonstrated on a Coupled Tanks Control System , 2016, 2016 IEEE 17th International Symposium on High Assurance Systems Engineering (HASE).

[177]  Lui Sha,et al.  Evolving dependable real-time systems , 1996, 1996 IEEE Aerospace Applications Conference. Proceedings.

[178]  Russell Turner,et al.  Automatic integrated collision avoidance system , 2017 .

[179]  Samuel Coogan,et al.  Mixed Monotonicity for Reachability and Safety in Dynamical Systems , 2020, 2020 59th IEEE Conference on Decision and Control (CDC).