Learning to Forget: Continual Prediction with LSTM
Top Cited Papers
- 1 October 2000
- journal article
- Published by MIT Press in Neural Computation
- Vol. 12 (10), 2451-2471
- https://doi.org/10.1162/089976600300015015
Abstract
Long short-term memory (LSTM; Hochreiter & Schmidhuber, 1997) can solve numerous tasks not solvable by previous learning algorithms for recurrent neural networks (RNNs). We identify a weakness of LSTM networks processing continual input streams that are not a priori segmented into subsequences with explicitly marked ends at which the network's internal state could be reset. Without resets, the state may grow indefinitely and eventually cause the network to break down. Our remedy is a novel, adaptive "forget gate" that enables an LSTM cell to learn to reset itself at appropriate times, thus releasing internal resources. We review illustrative benchmark problems on which standard LSTM outperforms other RNN algorithms. All algorithms (including LSTM) fail to solve continual versions of these problems. LSTM with forget gates, however, easily solves them, and in an elegant way.Keywords
This publication has 13 references indexed in Scilit:
- Long Short-Term MemoryNeural Computation, 1997
- Learning long-term dependencies in NARX recurrent neural networksIEEE Transactions on Neural Networks, 1996
- Gradient calculations for dynamic recurrent neural networks: a surveyIEEE Transactions on Neural Networks, 1995
- Learning long-term dependencies with gradient descent is difficultIEEE Transactions on Neural Networks, 1994
- A Fixed Size Storage O(n3) Time Complexity Learning Algorithm for Fully Recurrent Continually Running NetworksNeural Computation, 1992
- Adaptive neural oscillator using continuous-time back-propagation learningNeural Networks, 1989
- Finite State Automata and Simple Recurrent NetworksNeural Computation, 1989
- Modular Construction of Time-Delay Neural Networks for Speech RecognitionNeural Computation, 1989
- LEARNING SEQUENTIAL STRUCTURE WITH THE REAL-TIME RECURRENT LEARNING ALGORITHMInternational Journal of Neural Systems, 1989
- A Local Learning Algorithm for Dynamic Feedforward and Recurrent NetworksConnection Science, 1989