Synchronization in Social Tasks: Robotic Drumming

Music performance is an important, well-structured setting for evaluating a robot´s ability to detect, understand and respond appropriately to complex human activity. Social tasks such as cooperative performance require participants to detect, interpret and attune to the actions of their partners quickly and accurately. The synthesis of multiple sensory perceptions may be a fruitful approach to this problem. In order to evaluate this approach, we programmed a humanoid robot, Nico, to play a drum in concert with human drummers and at the direction of a human conductor. Our results show that sensory integration can enable precise synchronization in social tasks even when perceptual data is imperfect, misleading and subject to extensive processing delay. By integrating several streams of information - visual, auditory, and proprioceptive - Nico can attune to a tempo that is set by a human conductor, in concert with human performers. Nico continuously evaluates its perceptions of its own actions and those of the humans around it, dealing with unforeseen changes in tempo and affect in real time