A generalized data-driven Hamiltonian Monte Carlo for hierarchical activity search

Motion and image analysis are both important for robust solutions to video search of activities; the physics-based, data-driven Hamiltonian Monte Carlo (HMC), a Markov chain Monte Carlo variant that is efficient in searching large dimensional spaces, simultaneously examines the combined motion and image space. In this paper, we generalize the data-driven HMC to no longer depend upon ad hoc Guide Hamiltonians and to no longer require physics-based features from tracks as pre-requisites. Our generalization thus allows it to be used with or without a tracker, overcoming a significant limitation of the physics-based approach, as well as being extensible to utilizing any pre-existing image- or motion-based method. We demonstrate the generalizability of our framework by considering situations when tracking is available and when it is not available. When tracking is available, we utilize Histogram of Oriented Gradients, shapes of trajectories, and Hamiltonian Energy Signatures; when tracking is not available, we use Space-time Interest Points and GIST features. In addition, we show our generalized framework performs better than the physics-based, data-driven HMC, as well as state-of-the-art, by demonstrating the efficacy of our system on real-life video sequences using the well-known Weizmann and YouTube Action datasets.