Hands-Free User Interface for AR/VR Devices Exploiting Wearer’s Facial Gestures Using Unsupervised Deep Learning
Open Access
- 14 October 2019
- Vol. 19 (20), 4441
- https://doi.org/10.3390/s19204441
Abstract
Developing a user interface (UI) suitable for headset environments is one of the challenges in the field of augmented reality (AR) technologies. This study proposes a hands-free UI for an AR headset that exploits facial gestures of the wearer to recognize user intentions. The facial gestures of the headset wearer are detected by a custom-designed sensor that detects skin deformation based on infrared diffusion characteristics of human skin. We designed a deep neural network classifier to determine the user’s intended gestures from skin-deformation data, which are exploited as user input commands for the proposed UI system. The proposed classifier is composed of a spatiotemporal autoencoder and deep embedded clustering algorithm, trained in an unsupervised manner. The UI device was embedded in a commercial AR headset, and several experiments were performed on the online sensor data to verify operation of the device. We achieved implementation of a hands-free UI for an AR headset with average accuracy of 95.4% user-command recognition, as determined through tests by participants.Keywords
Funding Information
- Institute for Information and Communications Technology Promotion (2017-0-00244)
This publication has 31 references indexed in Scilit:
- CyberTouch - Touch and Cursor Interface for VR HMDPublished by Springer Science and Business Media LLC ,2015
- Novel wrinkled periodic mesoporous organosilica nanoparticles for hydrophobic anticancer drug deliveryJournal of Porous Materials, 2014
- Brain-computer interfaces for communication and controlCommunications of the ACM, 2011
- Anomaly detectionACM Computing Surveys, 2009
- Determination of the optical properties of anisotropic biological media using an isotropic diffusion modelJournal of Biomedical Optics, 2007
- Active appearance modelsIEEE Transactions on Pattern Analysis and Machine Intelligence, 2001
- Long Short-Term MemoryNeural Computation, 1997
- A diffusion theory model of spatially resolved, steady-state diffuse reflectance for the noninvasive determination of tissue optical properties in vivoMedical Physics, 1992
- On the anatomy and physiology of the skinBritish Journal of Plastic Surgery, 1978
- A Mathematical Theory of CommunicationBell System Technical Journal, 1948