Fine-grained categorization of fish motion patterns in underwater videos

Marine biologists commonly use underwater videos for their research. Their video analysis, however, is typically based on visual inspection. This incurs prohibitively large user costs, and severely limits the scope of biological studies. There is a need for developing vision algorithms that can address specific needs of marine biologists, such as fine-grained categorization of fish motion patterns. This is a difficult problem, because of very small inter-class and large intra-class differences between fish motion patterns. Our approach consists of three steps. First, we apply our new fish detector to identify and localize fish occurrences in each frame, under partial occlusion, and amidst dynamic texture patterns formed by whirls of sand on the sea bed. Then, we conduct tracking-by-detection. Given the similarity between fish detections, defined in terms of fish appearance and motion properties, we formulate fish tracking as transitively linking similar detections between every two consecutive frames, so as to maintain their unique track IDs. Finally, we extract histograms of fish displacements along the estimated tracks. The histograms are classified by the Random Forest technique to recognize distinct classes of fish motion patterns. Evaluation on challenging underwater videos demonstrates that our approach outperforms the state-of-the-art techniques.