Empirical Evaluation of the Dialog-Based Protocol and Think-Aloud Protocol

Protocol analysis is a rigorous methodology for eliciting verbal reports of thought sequences. The think-aloud protocol and the dialog-based protocol are two approaches for protocol analysis, in which the think-aloud protocol uses a single subject and the dialog-based protocol uses two subjects, who are verbally expressing their thoughts while completing the tasks. In this study, we evaluated and made a comparison of the two protocols by conducting case studies with 12 undergraduate students. The students were divided into two groups. One group used the dialog-based protocol, and the other group applied the think-aloud protocol. Both groups implemented a small application, and used free software to record and to capture the screen changes. We observed the case studies and analyzed the media files to study those effects of the two protocols. Our study demonstrates that the dialog-based protocol has a lot of advantages over the think-aloud protocol such as lower placebo effect and less Hawthorne effect, and providing more complete and accurate data.