Improving the Performance of Web Crawler using Parallel Programming Patterns in C#

Web crawler is an important kernel in most search engine. It crawls web and finds new web pages for indexing and retrieving. As the number of web pages increases, retrieving important parts of the web using serial implementation of web crawler is a problem and it is time consuming. Todays, multi-core processors and parallel programming models and tools are used for parallel implementation of computational intensive kernels and applications. C# programming language has some parallel programming patterns and can be used for performance improvement of kernels. In order to improve the performance of web crawler kernel, we use parallel patterns in this language Experimental results show that the speedup of parallel implementation of a web crawler is up to 11x than serial implementation