A multiview, multimodal fusion framework for classifying small marine animals with an opto-acoustic imaging system

A multiview, multimodal fusion algorithm for classifying marine plankton is described and its performance is evaluated on laboratory data from live animals. The algorithm uses support vector machines with softmax outputs to classify either acoustical or optical features. Outputs from these single-view classifiers are then combined together using a feedback network with confidence weighting. For each view or modality, the initial classification and classifications from all other views and modalities are confidence-weighted and combined to render a final, improved classification. Simple features are computed from acoustic and video data with an aim at noise robustness. The algorithm is tested on acoustic and video data collected in the laboratory from live, untethered copepods and mysids (two dominant crustacean zooplankton). It is shown that the algorithm is able to yield significant (> 50%) reductions in error by combining views together. In addition, it is shown that the algorithm is able boost performance by giving more weight to views or modalities that are more discriminant than others, without any a priori knowledge of which views are more discriminant.