An applied framework for automatic inference of affective states from speech

Affective states and their behavioral expressions are an important aspect of human reasoning and communication. Their automated recognition can be integrated into a wide variety of applications. Therefore, it is of interest to design systems that can infer the affective states of speakers from their non-verbal expressions in speech. This paper presents a framework of such a system, its design and validation. The framework comprises the definition and extraction of vocal features useful for classification. For each utterance, the designed system infers the level of nine affective-state groups, such as thinking, excited, and interested (multiple-label classification) with overall accuracy of 83%. The validation and generalization of the system comprised characterization of a 570 affective state concepts as combinations of the affective-state groups, and multimodal analysis of affective states that were naturally evoked during sustained human-computer interactions, generalizing to new speakers and to a different language with no additional training.