Mutual disambiguation of 3D multimodal interaction in augmented and virtual reality
- 5 November 2003
- conference paper
- conference paper
- Published by Association for Computing Machinery (ACM)
Abstract
We describe an approach to 3D multimodal interaction in immersive augmented and virtual reality environments that accounts for the uncertain nature of the information sources. The resulting multimodal system fuses symbolic and statistical information from a set of 3D gesture, spoken language, and referential agents. The referential agents employ visible or invisible volumes that can be attached to 3D trackers in the environment, and which use a time-stamped history of the objects that intersect them to derive statistics for ranking potential referents. We discuss the means by which the system supports mutual disambiguation of these modalities and information sources, and show through a user study how mutual disambiguation accounts for over 45% of the successful 3D multimodal interpretations. An accompanying video demonstrates the system in action.Keywords
This publication has 18 references indexed in Scilit:
- XWandPublished by Association for Computing Machinery (ACM) ,2003
- Gesture + playPublished by Association for Computing Machinery (ACM) ,2003
- Cubic-Mouse-based interaction in virtual environmentsIEEE Computer Graphics and Applications, 2000
- Multimodal integration-a statistical viewIEEE Transactions on Multimedia, 1999
- Multimodal interaction for 2D and 3D environments [virtual reality]IEEE Computer Graphics and Applications, 1999
- Confirmation in multimodal systemsPublished by Association for Computational Linguistics (ACL) ,1998
- Unification-based multimodal integrationPublished by Association for Computational Linguistics (ACL) ,1997
- JDCAD: A highly interactive 3D modeling systemComputers & Graphics, 1994
- A method of interactive visualization of CAD surface models on a color video displayPublished by Association for Computing Machinery (ACM) ,1981
- “Put-that-there”ACM SIGGRAPH Computer Graphics, 1980