Who is IT? Inferring role and intent from agent motion

We present a system which observes humans interacting in the real world and infers their goals and intentions through detecting and analyzing their spatiotemporal relationships. Given the motion trajectories of human agents and inanimate objects within a room, the system attempts to characterize how each agent moves in response to the locations of the others in the room -towards an object, say, and away from the other agent. Each of these calculations leads to an estimate of the agent´s current intentional state. Taken together with the other agents in the room, and paying particular attention to the moments when the various agents´ states change, the system can construct a coherent account of the action similar to the stories humans tell. We illustrate this approach with a robot that watches people playing a game of tag, works out for itself the roles and intentions of the various players, and then attempts to join in the fun. The robot´s interpretation of events agrees with human observers on average 70.8% of the time -nearly as good as the agreement between two humans (78.5%).