Automated control of sound synthesis in live musical performances using modified online time warping

This paper presents a system for automating sound synthesis parameters during live musical performances. It is a novel practical solution for keyboard players who need to manually control numerous synthesis parameters while playing at the same time. Our idea was to let musicians define how parameters will change through time by inserting automation data into a prerecorded musical piece. During a live performance of the same piece, our system employs a score following algorithm and applies these automation data at the right moments. A referent recording is represented as a MIDI file, whilst a performance is a stream of MIDI events. For the task of score following we implemented the on-line time warping (OLTW) algorithm modified to work with MIDI data. To convert sequential MIDI events into the values at equidistant points in time we employed time framing. As of our knowledge, this is the first usage of OLTW on MIDI representations. We devised a module for applying automation data which includes forward estimation for better time precision. The whole solution was implemented in Java as a real-time system. The evaluation with live performances yielded encouraging results which confirmed the effectiveness of our algorithm and its implementation.