Efficient physics based simulation of spatial audio for virtual and augmented reality

Abstract

Many applications in spatial hearing are being enabled by the virtual and augmented reality revolution. Because of these there has been a tremendous improvement in the quality of head mounted displays, and the availability of tracking (3 DOF and 6 DOF) in an integrated environment with headphones suitable for binaural rendering. While these devices were initially focused on creating visual representations of the 3D world (in the case of VR) or a visual representation of artificial 3D objects overlaid on the real world (in the case of augmented reality), it was soon realized that unless the audio provided a degree of spatial realism that complemented the visual and was consistent with it, the sense of immersion crucial to maintaining the illusion was lost. In this talk, I will describe the engine for creating virtual simulations in real time that incorporates head motion, environmental reverberation, room materials, as well as generic or individual HRTFs. Trade-offs that allow the engine to work on architectures ranging from single core CPUs, to mobile devices to high end multicore devices will be discussed. In contrast to these “object-based” representations of spatial auditory objects, there has also been a revival in approaches based on expanding the sound-field in low order ambisonics representations, and their subsequent binaural rendering. After providing a description of the optimizations made in our engine, we discuss future work.