Towards Improving Visual-Facial Emotion Recognition through Use of Complementary Keyboard-Stroke Pattern Information

In this paper, we investigate the possibility of improving the accuracy of visual-facial emotion recognition through use of additional (complementary) keyboard-stroke information. The investigation is based on two empirical studies that we have conducted involving human subjects and human observers. The studies were concerned with the recognition of emotions from a visual-facial modality and keyboard-stroke information, respectively. They were inspired by the relative shortage of such previous research in empirical work concerning the strengths and weaknesses of each modality so that the extent can be determined to which the keyboard-stroke information complements and improves the emotion recognition accuracy of the visual-facial modality. Specifically, our research focused on the recognition of six basic emotion states, namely happiness, sadness, surprise, anger and disgust as well as the emotionless state which we refer to as neutral. We have found that the visual-facial modality may allow the recognition of certain states, such as neutral and surprise, with sufficient accuracy. However, its accuracy in recognizing anger and happiness can be improved significantly if assisted by keyboard-stroke information.