Evaluation of Neural Network Verification Methods for Air-to-Air Collision Avoidance
- 1 January 2023
- journal article
- research article
- Published by American Institute of Aeronautics and Astronautics (AIAA) in Journal of Air Transportation
- Vol. 31 (1), 1-17
- https://doi.org/10.2514/1.d0255
Abstract
Neural network approximations have become attractive to compress data for automation and autonomy algorithms for use on storage-limited and processing-limited aerospace hardware. However, unless these neural network approximations can be exhaustively verified to be safe, they cannot be certified for use on aircraft. An example of such systems is the unmanned Airborne Collision Avoidance System (ACAS) Xu, which is a very popular benchmark for open-loop neural network control system verification tools. This paper proposes a new closed-loop extension of this benchmark, which consists of a set of 10 closed-loop properties selected to evaluate the safety of an ownship aircraft in the presence of a co-altitude intruder aircraft. These closed-loop safety properties are used to evaluate five of the 45 neural networks that comprise the ACAS Xu benchmark (corresponding to co-altitude cases) as well as the switching logic between the five neural networks. The combination of nonlinear dynamics and switching between five neural networks is a challenging verification task accomplished with star-set reachability methods in two verification tools. The safety of the ownship aircraft under initial position uncertainty is guaranteed in every scenario proposed.Keywords
Funding Information
- ONR (N00014-22-1-2156)
- Defense Advanced Research Projects Agency (FA8750-18-C-0089)
- Air Force Office of Scientific Research (FA9550-18-1-0122, FA9550-19-1-0288, FA9550-21-1-0121, FA9550-22-1-0019, FA9550-22-1-0450)
- Division of Computing and Communication Foundations (1910017, 2028001, EPSCoR First Award, FMitF 1918450)
This publication has 16 references indexed in Scilit:
- Deep Neural Network Compression for Aircraft Collision Avoidance SystemsJournal of Guidance, Control, and Dynamics, 2019
- Advanced Deep-Learning Techniques for Salient and Category-Specific Object Detection: A SurveyIEEE Signal Processing Magazine, 2018
- Mastering the game of Go without human knowledgeNature, 2017
- Mastering the game of Go with deep neural networks and tree searchNature, 2016
- A Tutorial on Latin Hypercube Design of ExperimentsQuality and Reliability Engineering International, 2015
- Optimizing the Next Generation Collision Avoidance System for Safe, Suitable, and Acceptable Operational PerformanceAir Traffic Control Quarterly, 2013
- A Survey of Active Learning Algorithms for Supervised Remote Sensing Image ClassificationIEEE Journal of Selected Topics in Signal Processing, 2011
- Handbook of Natural Language ProcessingPublished by Taylor & Francis Ltd ,2010
- Reachability in Linear Dynamical SystemsPublished by Springer Science and Business Media LLC ,2008
- Unscented Filtering and Nonlinear EstimationProceedings of the IEEE, 2004