Extracting new topic contents from hidden Web sites

Many information sources provide their database contents through query interfaces. Hidden Web sites are typical examples. In most cases, their database contents change dynamically, with new documents on emerging topics being appended. In applications like topic detection and trend analysis, we want to discover newly emerging contents in the databases. It is very difficult, however, for ordinary users to detect them through only query interfaces, without support by database contents administrators. In this paper, we propose a novel method to automatically discover such contents. The proposed method generates biased query probes to be issued to a hidden Web site with a keyword-based query interface. The probes focus on extracting documents on newly emerging topics. We evaluate effectiveness with experiments.