A virtual piano-playing environment for rehabilitation based upon ultrasound imaging

In this paper we evaluate ultrasound imaging as a human-machine interface in the context of rehabilitation. Ultrasound imaging can be used to estimate finger forces in real-time with a short and easy calibration procedure. Forces are individually predicted using a transducer fixed on the forearm, which leaves the hand completely free to operate. In this application, a standard ultrasound machine is connected to a virtual-reality environment in which a human operator can play a dynamic harmonium over two octaves, using either finger (including the thumb). The interaction in the virtual environment is managed via a fast collision detection algorithm and a physics engine. Ten human subjects have been engaged in two games of increasing difficulty. Our experimental results, both objective and subjective, clearly show that both tasks could be accomplished to the required degree of precision and that the subjects underwent a typical learning curve. The learning happened uniformly, irrespective of the required finger, force or note. Such a system could be made portable, and has potential applications as rehabilitation device for amputees and muscle impaired, even at home.