Building the databases needed to understand rich, spontaneous human behaviour

One of the motives for studying faces and gestures is the role that they play in spontaneous, socially rich interaction between humans. If computers are to interact with humans in that mode (or to analyse what they are doing in it), methods of interpreting the non-verbal signals that they use are critical. It is becoming clear that developing those methods requires databases whose complexity is of a very different order from those that are standard elsewhere. Samples cannot be generated to order, because acting does not reproduce the way features are distributed in spontaneous action. Data collections need to be very large, because there are extensive situational, individual, and cultural differences. It is a very large task to provide annotations that adequately capture the meaning of facial expressions or gestures. These problems are not insoluble, but sustained and well-directed efforts are needed to solve them.