Residue boundary histograms for action recognition in the compressed domain

Traditional action recognition approaches are too slow for real-time or large-scale applications. This problem has been tackled by replacing optical flow with motion vectors from the compressed domain. Yet further usage of compressed domain information for action recognition is possible. Discrete cosine transform (DCT) coefficients, which correspond to residue data, represent information which the block based motion vectors fail to capture. We propose a set of residue boundary histograms (RBH) features for action recognition, separating each DCT block into four parts to obtain four small residue maps and then encoding each residue map by histogram-based descriptors to obtain local features. Experimental results on three challenging datasets show that proposed RBH features improve upon motion vector based features significantly. While more than 100× faster, the results are highly competitive compared with traditional action recognition approaches.