Title :
PyBot: An Algorithm for Web Crawling
Author :
Leng, Alex Goh Kwang ; Ravi, Kumar P. ; Singh, Ashutosh Kumar ; Dash, Rajendra Kumar
Author_Institution :
Dept. of Electr. & Comput. Eng., Curtin Univ., Miri, Malaysia
Abstract :
PyBot is a Web Crawler developed in Python to crawl the Web using Breadth First Search (BFS). The success of the World Wide Web (WWW), which itself built on the open internet, has changed the way how human share and exchange information and ideas. With the explosive growth of the dynamic Web, users have to spend much time just to retrieve a small portion of the information from the Web. The birth of the search engines have made human lives easier by simply taking them to the resources they want. Web crawler is a program used by search engines to retrieve information from the World Wide Web in an automated manner.
Keywords :
Internet; information retrieval; search engines; PyBot; Python; Web crawler; World Wide Web; breadth first search; information retrieval; open Internet; search engines; Crawlers; Engines; Indexes; Robots; Search engines; Web pages;
Conference_Titel :
Nanoscience, Technology and Societal Implications (NSTSI), 2011 International Conference on
Conference_Location :
Bhubaneswar
Print_ISBN :
978-1-4577-2035-2
DOI :
10.1109/NSTSI.2011.6111993
