Crawling Credible Online Medical Sentiments for Social Intelligence

Social intelligence derived from Health 2.0 content has become of significant importance for various applications, including post-marketing drug surveillance, competitive intelligence, and to assess health-related opinions and sentiments. However, the volume, velocity, variety, and quality of online health information present challenges, necessitating enhanced facilitation mechanisms for medical social computing. In this study, we propose a focused crawler for online medical content. The crawler leverages enhanced credibility and context information. An extensive evaluation was performed against several comparison methods, on an online Health 2.0 test bed encompassing millions of pages. The results revealed that the proposed method was able to collect relevant content with considerably higher precision and recall rates than comparison methods, on content associated with medical websites, forums, blogs, and social networking sites. Furthermore, an example was used to illustrate the usefulness of the crawler for accurately representing online drug sentiments. Overall, the results have important implications for social computing, where a high-quality data and information foundation are imperative to the success of any overlying social intelligence initiative.