Information storage in neural networks with low levels of activity
- 1 March 1987
- journal article
- research article
- Published by American Physical Society (APS) in Physical Review A
- Vol. 35 (5), 2293-2303
- https://doi.org/10.1103/physreva.35.2293
Abstract
The Hopfield model of a neural network is extended to allow for the storage and retrieval of biased patterns, {}, where =a is arbitrary. Such patterns represent levels of activity (i.e., percentage of firing neurons) equal to (1/2)(1+a), -1<a<1. If the coupling constants (synaptic efficacies) are constructed as in the original Hopfield model, the system can retrieve at most a very small number of patterns (p). This is due to the finite correlations (overlaps) between the patterns. The model is modified by subtracting the bias a from each pattern as it enters into the couplings. This modification restores the ability of the model to store a macroscopic number of patterns. Yet spurious states are found to plague the dynamics. It is then argued that the dynamics of the network should be consistent with the levels of activity of the stored patterns. This is implemented by adding a global constraint, which restricts the configuration space to states whose mean activity is in the neighborhood of (1/2)(1+a). The consequences of the restricted dynamics are analyzed in the replica symmetric mean-field theory. The global constraint suppresses spurious states and leads to the unexpected result that the storage capacity is higher than that of the unbiased network, up to very high values of the bias (‖a‖≃0.99). However, the information content of such networks is shown to be a monotonically decreasing function of a.
Keywords
This publication has 15 references indexed in Scilit:
- Associative recall of memory without errorsPhysical Review A, 1987
- Neural networks with nonlinear synapses and a static noisePhysical Review A, 1986
- Saturation Level of the Hopfield Model for Neural NetworkEurophysics Letters, 1986
- Storing Infinite Numbers of Patterns in a Spin-Glass Model of Neural NetworksPhysical Review Letters, 1985
- Spin-glass models of neural networksPhysical Review A, 1985
- Neurons with graded response have collective computational properties like those of two-state neurons.Proceedings of the National Academy of Sciences of the United States of America, 1984
- ‘Unlearning’ has a stabilizing effect in collective memoriesNature, 1983
- Neural networks and physical systems with emergent collective computational abilities.Proceedings of the National Academy of Sciences of the United States of America, 1982
- Analytic study of the memory storage capacity of a neural networkMathematical Biosciences, 1978
- The existence of persistent states in the brainMathematical Biosciences, 1974