Incorporating temporal context in Bag-of-Words models

Bag-of-Words (BoW) is a highly popular model for recognition, due to its robustness and simplicity. Its modeling capabilities, however, are somewhat limited since it discards the spatial and temporal order of the codewords. In this paper we propose a new model: Contextual Sequence of Words (CSoW) which incorporates temporal order into the BoW model for video representation. The temporal context is incorporated in three scales that capture different aspects of the variability between different performances of the same action. We show that using CSoW instead of BoW leads to a significant improvement in action recognition rates, on several different setups.