Learning discriminative features and metrics for measuring action similarity

Measuring the similarity of human actions in videos is a challenging task. Two critical factors that affect the performance include low-level feature representations and similarity metrics. However, finding the right feature representations and metrics is hard. In this paper, we describe a novel approach that jointly learns both of them from the data, while current approaches either only learn one or not learn at all. We propose a generative plus discriminative learning method based on gated auto encoders to simultaneously learn the features and their associated metrics. Our method differs from existing feature or metric learning methods in two ways: 1) while other methods treat feature learning and metric learning as independent tasks, we argue that they should be learned jointly since features and metrics are tightly inter-dependent; 2) our method learns more discriminative features than its purely generative counterparts.