Learning of object manipulation operations from continuous multimodal input

In this paper we propose an approach for identification of high-level object manipulation operations within a continuous multimodal time-series. We focus on a multimodal approach for robust recognition of action primitive data. Our procedure combines an unsupervised Bayesian multimodal segmentation with a supervised machine learning approach. We briefly outline (1) the unsupervised segmentation and selection of uni- and bi-manual manipulation primitives developed in our previous work. We show (2) an application of the ordered means models to classification of estimated segments. To assess the performance of our approach, we compare the computed labels to the ground truth acquired during the data recording. In our experiments we examined the robustness of the procedure on two different sets of segments: full length (≈ 95% overlap with the ground truth on average), partial length (≈ 10% overlap with ground truth on average). We have achieved a cross validation rate of ≈ 0.95 and recognition accuracy of ≈ 0.97 for full length and ≈ 0.84 for partial length test sets.