Estimation of spatio-temporal object properties for manipulation tasks from observation of humans

We propose a system for vision-based estimation of manipulation-relevant properties of objects in natural scenes based on observation of human actions. The system consists of an a-priori (Atlas) knowledge about known generic objects in the scene and classifies the scene into mission relevant objects and background geometry that is important only for collision avoidance. We present the object-centric structure of our system consisting of an Atlas representation and a Working Memory storing the current knowledge about the scene, the manipulated objects and actions applied to them in the local environment. We present experimental results how the system maintains the information in the database and we show the quality of the results that can be obtained with our system.