Active scene recognition for programming by demonstration using next-best-view estimates from hierarchical Implicit Shape Models

We present an approach that combines passive scene understanding with object search in order to recognize scenes in indoor environments that cannot be perceived from a single point of view. Passive scene recognition is performed using Implicit Shape Models based on spatial relations between objects. ISMs, a variant of the Generalized Hough Transform, are extended to describe scenes as sets of objects with relations lying between them. Relations are expressed as six-degree-of-freedom (DoF) relative object poses. They are extracted from sensor recordings of human demonstrations of actions usually taking place in the corresponding scene. In a scene ISMs solely represent relations of n objects towards a common reference. Violations of other relations are not detectable. To overcome this limitation, we extend our scene model, using hierarchical agglomerative clustering, to a binary tree consisting of ISMs. Active scene recognition aims to simultaneously detect present scenes and look for objects these scenes consist of. For a pivoting stereo camera rig, we achieve this by performing recognition with ISMs in an object search loop using next-best-view (NBV) estimates. A criterion, on which we greedily choose views the rig shall adopt next, is the confidence to detect objects in them. In each step during the search, confidences on potential positions of objects, not found yet, are calculated based on the best available scene hypothesis. This is done by reversing the principle of ISMs and using spatial relations to predict potential object positions starting from the objects already detected.