4 + 4 Fingers Manipulating Virtual Objects in Mixed-Reality Environment

Abstract

This paper describes the construction of a prototype system that allows the user to manipulate virtual objects in a mixed-reality environment. The user can perceive haptic sensations at his or her fingertips and see a virtual world with a live image of his or her hands manipulating the virtual objects. The difficulty of constructing such a visual presentation is the problem of merging the real image with correct geometrical occlusion with images of virtual objects in the simulated virtual world. In this paper, a method of solving registration of real and virtual images to produce mixedreality scenes is proposed. We have combined this method with our haptic interface device, SPIDAR-8, to construct a system that integrates these realities by using the sense of touch and vision of real images in the virtual world that can be realized during the manipulation of virtual objects. The implemented results are shown and remaining problems are discussed.