Marker-less feature and gesture detection for an interactive mixed reality avatar

Subject-object interaction in a museum exhibit requires capture of the rigid museum objects, gestures and tasks performed by visitors as well as the subject-object interactions. Our interactive system takes a real object belonging to the avatar of a Tlingit warrior. The chosen object is a Tlingit war helmet replica which had to be detected by a RGB camera without the help of fiducial markers and used to display information on the screen. Our contribution is to integrate cascade classifiers, normally used to detect faces, to detect the helmet and to inform the mixed reality application flow. Classifiers adapted better to the museums ambient conditions. Additionally, we used skeletal data to detect when the user puts the helmet on and to allow an avatar on screen to be controlled by the user.