A Minimum Cost Based Web Information Gathering Scheme

With the exploration of the Internet, the data gathering from the Web is a challenge task. Both the traditional Client/Server (C/S) scheme and the new Mobile Agent (MA) scheme have their advantages and shortcomings. In this paper, a hybrid data gathering scheme is proposed based on minimum cost principle i.e. the minimum data transferred or the minimum running time. Through the analysis for C/S scheme and MA scheme respectively, we can set a threshold for choosing the appropriate data gathering scheme. The C/S scheme is more efficient if the crawled web site has a few pages. Otherwise, the MA scheme is better. The proposed scheme is tested in VAST (VisuAl & SemanTic image search) system, and the experimental results demonstrate the effectiveness of the proposed approach.