Autonomous Separation Assurance with Deep Multi-Agent Reinforcement Learning

Abstract

A novel deep multi-agent reinforcement learning framework is proposed to identify and resolve conflicts among a variable number of aircraft in a high-density, stochastic, and dynamic en route sector. The concept of using distributed vehicle autonomy to ensure separation is proposed, instead of a centralized sector air traffic controller. Our proposed framework uses proximal policy optimization that is customized to incorporate an attention network. This allows the agents to have access to variable aircraft information in the sector in a scalable, efficient approach to achieve high traffic throughput under uncertainty. Agents are trained using a centralized learning, decentralized execution scheme where one neural network is learned and shared by all agents. The proposed framework is validated on three case studies in the BlueSky air traffic simulator. Several baselines are introduced, and the numerical results show that the proposed framework significantly reduces offline training time, increases safe separation performance, and results in a more efficient policy.