Integrated data processing between image and audio-musical instrument (piano) playing information processing

This paper proposes an integrated data processing system between video and audio. The subject in this paper is the action of “playing the piano”. First the information generated from the action is defined according to the meanings; scale, key, etc. Then depending on the meanings, each piece of information is processed in a media space, where it is likely to be more easily processed. We designed a co-operative musical scale detection system of audio and video data and performed basic experiments. The simulation results show that image processing only gave a detection rate over 90% (white-key) and sound processing could cope with faster key-hit movement and black-key hit detection, and combining the two processes gave a higher detection rate. The proposed method can be applied to audio data compression. With the simple sound description model (musical scale, amplitude and phase), about 40-70% of data reduction was achieved with better quality than processing audio alone