A stereo-vision approach for a natural 3D hand interaction with an AR object

Providing natural hand interaction between a virtual object and a user on Augmented Reality is a major issue to manipulate a rendered object in a convenient way. Conventional 2D image-based recognition and interaction technique in AR has a limitation to perform a natural interaction between the user and the virtual object. In this paper, we present a stereo-vision based natural 3D hand interaction with the augmented object. In the proposed 3D hand interaction approach, 3D hand location and finger direction can be easily obtained by using stereo-vision technique while user hand is approaching to the virtual object. Two types of hand manipulation for the augmented object such as the hand pointing and hand pinching are defined. The collision detection between user hand and the virtual object is determined by using a simple ray casting emitted from the user´s finger-point against the virtual object in the case of hand pointing. In the hand pinching, the collision is occurred when the thumb and the index finger are approaching to the object and the degree of angle between two fingers becomes a predetermined value. From the experiments, the proposed 3D hand interaction method can control the virtual object in a natural way rather than using a vision-based 2D hand interaction since the stereo-vision technique can obtain the depth information from the AR environments.