An insight- and task-based methodology for evaluating spatiotemporal visual analytics

We present a method for evaluating visualizations using both tasks and exploration, and demonstrate this method in a study of spatiotemporal network designs for a visual analytics system. The method is well suited for studying visual analytics applications in which users perform both targeted data searches and analyses of broader patterns. In such applications, an effective visualization design is one that helps users complete tasks accurately and efficiently, and supports hypothesis generation during open-ended exploration. To evaluate both of these aims in a single study, we developed an approach called layered insight- and task-based evaluation (LITE) that interposes several prompts for observations about the data model between sequences of predefined search tasks. We demonstrate the evaluation method in a user study of four network visualizations for spatiotemporal data in a visual analytics application. Results include findings that might have been difficult to obtain in a single experiment using a different methodology. For example, with one dataset we studied, we found that on average participants were faster on search tasks using a force-directed layout than using our other designs; at the same time, participants found this design least helpful in understanding the data. Our contributions include a novel evaluation method that combines well-defined tasks with exploration and observation, an evaluation of network visualization designs for spatiotemporal visual analytics, and guidelines for using this evaluation method.