Using visual and auditory feedback for instrument-playing humanoids

In this paper, we present techniques that enable a humanoid to autonomously play instruments like the metallophone. The core of our approach is a model-based method to estimate the pose of the instrument and the beaters held by the robot using observations from the onboard camera. For accurate playing, we calibrate the kinematic parameters of the robot and find valid configurations of the arms for beating the individual sound bars of the instrument. To determine these, we rely on the estimated pose of the instrument and the beaters and apply inverse kinematics (IK). Hereby, we use precomputed forward kinematics solutions represented by a reachability tree to accelerate the IK computation and compensate for local minima. The robot automatically validates the computed IK configurations based on visual and auditory feedback using its sensors, and adapts its arm configurations if necessary. Our system parses MIDI-files of whole songs, maps the notes to the corresponding arm configurations for beating, and generates trajectories in joint space to hit the sound bars. As we show in the experiments with a Nao humanoid presented in this paper as well as in the accompanying video, our approach allows for clean and in-time playing of complete songs on a metallophone.