Visually Indicated Sounds
Open Access
- 1 June 2016
- conference paper
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 2405-2413
- https://doi.org/10.1109/cvpr.2016.264
Abstract
Objects make distinctive sounds when they are hit or scratched. These sounds reveal aspects of an object's material properties, as well as the actions that produced them. In this paper, we propose the task of predicting what sound an object makes when struck as a way of studying physical interactions within a visual scene. We present an algorithm that synthesizes sound from silent videos of people hitting and scratching objects with a drumstick. This algorithm uses a recurrent neural network to predict sound features from videos and then produces a waveform from these features with an example-based synthesis procedure. We show that the sounds predicted by our model are realistic enough to fool participants in a "real or fake" psychophysical experiment, and that they convey significant information about material properties and physical interactions.This publication has 29 references indexed in Scilit:
- Recognizing Materials Using Perceptually Inspired FeaturesInternational Journal of Computer Vision, 2013
- The origins of inquiry: inductive inference and exploration in early childhoodTrends in Cognitive Sciences, 2012
- Spatial pattern of BOLD fMRI activation reveals cross-modal information in auditory cortexJournal of Neurophysiology, 2012
- Sound Texture Perception via Statistics of the Auditory Periphery: Evidence from Sound SynthesisNeuron, 2011
- The Development of Embodied Cognition: Six Lessons from BabiesArtificial Life, 2005
- Speech Enhancement Based onWavelet Thresholding the Multitaper SpectrumIEEE Transactions on Speech and Audio Processing, 2004
- Long Short-Term MemoryNeural Computation, 1997
- Derivation of auditory filter shapes from notched-noise dataHearing Research, 1990
- The estimation of the gradient of a density function, with applications in pattern recognitionIEEE Transactions on Information Theory, 1975
- Can One Hear the Shape of a Drum?The American Mathematical Monthly, 1966