A heterogeneous framework for real-time decoding of bioacoustic signals: Applications to assistive interfaces and prosthesis control

Tongue-movement ear pressure (TMEP) signals provide an unobtrusive, completely non-invasive, wearable and assistive human–machine interface (HMI). The HMI concept is based on monitoring volitional bioacoustic activity generated through prescribed tongue motions. In this paper, a heterogeneous decoding framework is presented, enabling effective real-time polychotomous classification between the various tongue actions and dichotomous discrimination of unintended bioacoustic activity from the volitional signals. Using this customised framework and developed software, the real-time performance was evaluated for both three (six subjects) and four action (four subjects) discrimination, using healthy subjects. Ignoring false negative rejections, the system achieved sensitivities of >90% for three-action discrimination and >80% for four action discrimination, across all tested subjects. The interference rejection (IR) capabilities of the framework were also fully demonstrated, using challenging offline data sets. This included a subset of low frequency interference signals with similar temporal characteristics and frequency distributions as the volitional tongue activity. The IR subsystem achieved an average specificity of 76.2% during three-action discrimination and 79.9% during four-action discrimination. To highlight the potential of the system for substituting or augmenting existing assistive interfaces, a case study is presented demonstrating the utility of TMEP signals for hand prosthesis control. Full tongue control was evaluated against three alternative control strategies, namely natural human-hand manipulation, proportional-based control and a hybrid strategy, when performing an everyday object manipulation task. In all cases, the task was completed with the hybrid strategy performing comparably and even outperforming the proportional-based control strategy.