Reinforcement Learning for RoboCup Soccer Keepaway
- 1 September 2005
- journal article
- Published by SAGE Publications in Adaptive Behavior
- Vol. 13 (3), 165-188
- https://doi.org/10.1177/105971230501300301
Abstract
RoboCup simulated soccer presents many challenges to reinforcement learning methods, including a large state space, hidden and uncertain state, multiple independent agents learning simultaneously, and long and variable delays in the effects of actions. We describe our application of episodic SMDP Sarsa(λ) with linear tile-coding function approximation and variable λ to learning higher-level decisions in a keepaway subtask of RoboCup soccer. In keepaway, one team, “the keepers,” tries to keep control of the ball for as long as possible despite the efforts of “the takers.” The keepers learn individually when to hold the ball and when to pass to a teammate. Our agents learned policies that significantly outperform a range of benchmark policies. We demonstrate the generality of our approach by applying it to a number of task variations including different field sizes and different numbers of players on each team.Keywords
This publication has 17 references indexed in Scilit:
- Karlsruhe Brainstormers - A Reinforcement Learning approach to robotic soccerLecture Notes in Computer Science, 2001
- Keeping the Ball from CMUnited-99Lecture Notes in Computer Science, 2001
- Experiences Acquired in the Design of RoboCup Teams: A Comparison of Two Fielded TeamsAutonomous Agents and Multi-Agent Systems, 2001
- Hierarchical Reinforcement Learning with the MAXQ Value Function DecompositionJournal of Artificial Intelligence Research, 2000
- Decision-Theoretic Planning: Structural Assumptions and Computational LeverageJournal of Artificial Intelligence Research, 1999
- Evolving Team Darwin UnitedLecture Notes in Computer Science, 1999
- Soccer server: A tool for research on multiagent systemsApplied Artificial Intelligence, 1998
- Co-evolving Soccer Softbot team coordination with genetic programmingLecture Notes in Computer Science, 1998
- Refinement of soccer agents' positions using reinforcement learningLecture Notes in Computer Science, 1998
- Markov Decision ProcessesWiley Series in Probability and Statistics, 1994