DIFTSAS: A DIstributed Full Text Search and Analysis System for Big Data

As we enter the big data era, digital world is growing fast and becomes more and more complex in nature, many content-oriented applications require both features of high performance full-text searching and rich functional online data analyzing. In this paper, we exploit the features of both web search and parallel DBMSs to address the challenges to build a scalable, interactive ad-hoc query system for analysis of incrementally accumulated content data, namely DIFTSAS. DIFTSAS is considered as a many-sided system, which is not only capable of handling full-text data as a search engine but also adept at online data analytical processing like a traditional data warehouse. In this paper, we present the design, implementation, and performance evaluation of DIFTSAS.