Collaborative Web crawling: information gathering/processing over Internet

The main objective of the IBM Grand Central Station (GCS) project is to gather all types of information in any format (text, data, image, graphics, audio, video) from cyberspace, to process/index/summarize the information, and to push the right information to the right people. Because of the very large scale of cyberspace, parallel processing in both crawling/gathering and information processing is indispensable. We present a scalable method for collaborative Web crawling and information processing. The method includes an automatic cyberspace partitioner which is designed to balance and re-balance the load dynamically among processors. It can be used when all Web crawlers are located on a tightly coupled high-performance system as well as when they are scattered in a distributed environment. We implemented these algorithms in Java.