Using recurrence plots to visualize the temporal dynamics of tutor/student interactions

Non-verbal behavior is a mode of human interaction that is critical to communication, especially in situations where speech is sparse. In this study, we developed novel visualizations (called behavioral recurrence plots) to reveal the non-verbal behavior patterns of a tutor instructing a thirteen year old student in a mathematical problem, and the time-locked responses of the student solving a series of problems. In the first phase of analysis, video recordings were manually annotated to document the participants´ speech, behavior, gesturing, fidgeting, and eye gaze over time, and automatic detection of movement was used to extract gross human motion. In a second phase of analysis, recurrence plots were used to visualize the interaction dynamics between the tutor´s behaviors and between the tutor and student. The recurrence plots reveal the global structure of the tutoring session, points of interest and transitions, as well as interactions between the many modes of non-verbal communication.