Hearing how you touch: Real-time synthesis of contact sounds for multisensory interaction

This paper describes a multisensory interface in which continuous contact sound effects of userpsilas actions on virtual objects are synthesized and rendered with haptic and visual stimuli. The auditory output is generated by convolving the force profile with the impulse response of an object, thus tightly coupled with the haptic output. Since each action, such as striking, scratching, and inhibiting, affects vibration modes distinctively, the nondeterministic nature of userpsilas real-time interactions requires more responsive approach than that for animations. We developed a technique to adjust parameters and drive corresponding modal sound models with low latency (lms). The implemented system produces realistic sound effects and smooth transitions between various actions.