Mudra
- 14 November 2011
- conference paper
- conference paper
- Published by Association for Computing Machinery (ACM) in Proceedings of the 13th international conference on multimodal interfaces - ICMI '11
Abstract
In recent years, multimodal interfaces have gained momentum as an alternative to traditional WIMP interaction styles. Existing multimodal fusion engines and frameworks range from low-level data stream-oriented approaches to high-level semantic inference-based solutions. However, there is a lack of multimodal interaction engines offering native fusion support across different levels of abstractions to fully exploit the power of multimodal interactions. We present Mudra, a unified multimodal interaction framework supporting the integrated processing of low-level data streams as well as high-level semantic inferences. Our solution is based on a central fact base in combination with a declarative rule-based language to derive new facts at different abstraction levels. Our innovative architecture for multimodal interaction encourages the use of software engineering principles such as modularisation and composition to support a growing set of input modalities as well as to enable the integration of existing or novel multimodal fusion engines.Keywords
This publication has 15 references indexed in Scilit:
- Multimodal Interfaces: A Survey of Principles, Models and FrameworksLecture Notes in Computer Science, 2009
- A probabilistic approach to reference resolution in multimodal user interfacesPublished by Association for Computing Machinery (ACM) ,2004
- Advances in robust multimodal interface designIEEE Computer Graphics and Applications, 2003
- From members to teams to committee-a robust approach to gestural and multimodal recognitionIEEE Transactions on Neural Networks, 2002
- DiamondTouchPublished by Association for Computing Machinery (ACM) ,2001
- Unification-based multimodal integrationPublished by Association for Computational Linguistics (ACL) ,1997
- Four Easy Pieces for Assessing the Usability of Multimodal Interaction: The Care PropertiesPublished by Springer Science and Business Media LLC ,1995
- An improved automatic lipreading system to enhance speech recognitionPublished by Association for Computing Machinery (ACM) ,1988
- Rete: A fast algorithm for the many pattern/many object pattern match problemArtificial Intelligence, 1982
- “Put-that-there”Published by Association for Computing Machinery (ACM) ,1980