Application of reinforcement learning with continuous state space to ramp metering in real-world conditions
- 1 September 2012
- conference paper
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE) in 2012 15th International IEEE Conference on Intelligent Transportation Systems
- p. 1590-1595
- https://doi.org/10.1109/itsc.2012.6338837
Abstract
In this paper we introduce a new approach to Freeway Ramp Metering (RM) based on Reinforcement Learning (RL) with focus on real-life experiments in a case study in the City of Toronto. Typical RL methods consider discrete state representation that lead to slow convergence in complex problems. Continuous representation of state space has the potential to significantly improve the learning speed and therefore enables tackling large-scale complex problems. A robust approach based on local regression, named k nearest neighbors temporal difference (kNN-TD), is employed to represent state space continuously in the RL environment. The performance of the new algorithm is compared against the ALINEA controller and typical RL methods using a micro-simulation testbed in Paramics. The results show that RM using the kNN-TD method can reduce total network travel time by 44% compared to the do-nothing case (without RM) and by 17% compared to ALINEA.Keywords
This publication has 16 references indexed in Scilit:
- Simulation of Exclusive Truck Facilities on Urban FreewaysJournal of Transportation Engineering, 2011
- Coordinated ramp metering for freeway networks – A model-predictive hierarchical control approachTransportation Research Part C: Emerging Technologies, 2010
- Machine learning for multi-jurisdictional optimal traffic corridor controlTransportation Research Part A: Policy and Practice, 2010
- Reinforcement Learning: A Tutorial Survey and Recent AdvancesINFORMS Journal on Computing, 2009
- Overview of Traffic Signal Operation Policies for Ramp MeteringTransportation Research Record: Journal of the Transportation Research Board, 2008
- Review of road traffic control strategiesProceedings of the IEEE, 2003
- Reinforcement learning: Introduction to theory and potential for transport applicationsCanadian Journal of Civil Engineering, 2003
- Reinforcement Learning for True Adaptive Traffic Signal ControlJournal of Transportation Engineering, 2003
- Freeway ramp metering: an overviewIEEE Transactions on Intelligent Transportation Systems, 2002
- Q-learningMachine Learning, 1992