Efficient Action Recognition with MoFREAK

Recent work shows that local binary feature descriptors are effective for increasing the efficiency of object recognition, while retaining comparable performance to other state of the art descriptors. An extension of these approaches to action recognition in videos would facilitate huge gains in efficiency, due to the computational advantage of computing a bag-of-words representation with the Hamming distance rather than the Euclidean distance. We present a new local spatiotemporal descriptor for action recognition that encodes both the appearance and motion in a scene with a short binary string. The first bytes of the descriptor encode the appearance and some implicit motion, through an extension of the recently proposed FREAK descriptor. The remaining bytes strengthen the motion model by building a binary string through local motion patterns. We demonstrate that by exploiting the binary makeup of this descriptor, it is possible to greatly reduce the running time of action recognition while retaining competitive performance with the state of the art.