Space-time neighborhood based hierarchical descriptor for action recognition

Recent work shows interest-point-based representation is greatly popular in action recognition, due to their simple implementation and good reliability. The neighborhood information of local descriptors usually improves the recognition accuracy. Taking inspiration from this observation, we propose a novel hierarchical neighborhood descriptor for action recognition. At low level, we propose the compound appearance and motion descriptor which describes the feature of neighboring interest points, rather than a single space-time interest point. At high level, another new neighborhood based descriptor is proposed to describe the spatial distribution of neighboring interest points. For classification, we apply multi-channel nonlinear SVM based on the hierarchical vocabulary. Experiments validate that our method achieves the state-of-the-art results on two benchmark datasets.