We investigate quantitatively the so-called leverage effect, which corresponds to a negative correlation between past returns and future volatility. For individual stocks, this correlation is moderate and decays exponentially over 50 days, while for stock indices, it is much stronger but decays faster. For individual stocks, the magnitude of this correlation has a universal value that can be rationalized in terms of a new `retarded' model which interpolates between a purely additive and a purely multiplicative stochastic process. For stock indices a specific market panic phenomenon seems to be necessary to account for the observed amplitude of the effect.