Gesture recognition by neural networks and the expression of emotions

This paper summarizes our work in progress on the mapping of hand gestures to musical parameters in an interactive music performance and virtual reality environment. A neural network architecture for subgestures is introduced to provide a mechanism in order to achieve meaningful control parameters. We use data obtained from a sensor glove. These data are processed in order to extract meaningful data to control musical structures. This is done by an extended neural network architecture for sub-gestures combined with the extraction of parametric values. We focus on the mapping of gestural variations onto equivalent musical parameters. Motivated by a creative situation like a performance, we set up a dictionary of symbolic and parametric subgestures. Different hand gestures of this dictionary and characteristic variations are evaluated with respect to their applicability to intuitive and musical control of musical structures. The system is complemented with a 3D animated hand model and behaving representations of musical structures